In image processing, it is often useful to display overlay graphics on images, e.g. to display a segmented region, but this is not provided for in .NET. I decided to create a custom control that can easily be included into other projects and that provides some basic functionality for images, like zooming, panning and the display and creation of annotations. Together with a library for image processing which I will present later, this should provide the basic building blocks for any application that tackles image processing.
The annotation classes
My first try at this problem as published here was a bit haphazard, and left me not very satisfied. Indeed, the object hierarchy wasn't logical, and the collections weren't type safe. Also, animation wasn't flicker-free. I decided to rework the whole project using a clean object oriented approach, and came up with the following:
Control that holds the bitmap on which the annotations are drawn, and the properties and methods for drawing that bitmap. This is the control that is to be included in other projects.
Class that contains all code for the annotations, and all its collections.
Class used to contain annotations.
Class defining a strongly typed collection of
Abstract class used to derive more specific annotation classes.
Class defining a strongly typed collection of annotations
Class for an annotation consisting of points that form a region, as such.
Abstract class used to derive more specific point-based annotation classes
Annotation properties and methods
Class for an annotation consisting of loose points, i.e. points which do not define a region
Class for an annotation consisting of points that form a closed contour and thus define a region by forming a border.
PointBasedAnnotation properties and methods
Annotations are meant to work together, and they hold references to each other. By instancing an
AnnotatedImage, you get the following hierarchy:
Containers (Type safe collection)
Annotations (Type safe collection)
All the code for drawing the annotations is inside the
Annotations class, except for 2 routines for XOR drawing that are stored inside the
Annotations class intercepts the
Paint event of the
AnnotatedImage control in order to redraw every annotation
Container object, which in turn calls the '
Draw' method on every individual
Annotation object. The
AnnotatedImage control implements double buffering to ensure smooth drawing.
All events pertaining to annotations are handled in the
Annotations class, e.g. interactive drawing, and all those concerned with the image in
AnnotatedImage, e.g. zooming. This means a Windows event like a mouse click, will trigger several event handlers, and these will decide if they should do something with it.
Using the code
The code will compile to a DLL that can be added to the Visual Studio toolbox, and can then be included as a control in any application. Of all the events exposed,
Resize is the only one that must be handled, as it is through this event that the parent control can resize itself in order to accommodate changes in the size of the custom control, e.g. after zooming. The custom control usually has the size of the displayed image, except when it exceeds
MaximumSize for very large images or a large
displayZoom (do not forget to set this property!). In that case, only a portion of the image is displayed, and panning can be activated by dragging the image with the middle mouse button. On top of that, the control responds to the wheel mouse and z and shift-z for zooming, to w and shift-w for setting the averaging window size that can be used to provide feedback to the user about the color in the area around the cursor. This averaging window is also drawn on the control.
Explanations and points of interest
At first, I tried to derive my custom control from the
PictureBox class, but I failed to draw any graphics over the bitmap that can be assigned to the
PictureBox.Bitmap by overriding the
Paint method. Indeed, it seemed that the bitmap was drawn after any custom drawing I performed on the graphics created from the
PictureBox (this was puzzling). Drawing on the bitmap itself was, of course, no option as it corrupts the bitmap. Consequently, I opted to derive from the
control class (the
scrollablecontainer didn't work well either). I tried to include a horizontal and vertical scrollbar, but these wouldn't render on their own, and I found no easy way to paint them manually on the control. So, instead I implemented panning by dragging the middle mouse button, which becomes active when the image cannot fit into the custom control anymore.
Annotations are implemented using 3 extra classes: an
Annotations top-level object, an annotation
Container and an
Annotation object. This last object is actually an abstract class, with the really instantiable classes being
AnnotationRegion. I stayed away from the
GraphicsPath object because it didn't fit my purpose very well, although it is used sometimes internally when computing regions. Each
AnnotatedImage image control has 1
Annotations object, that can hold several
Containers, which each can hold several
Annotation objects. These can be of different subtypes. Note that these objects are not to be confused with vector graphic objects that are drawn on a background bitmap, rather they are conceptual and only represent certain divisions or regions in images. It is for this reason that these annotations have no intrinsic line thickness and are represented by a line of thickness of 1 pixel, regardless of the zoom factor of the image (I therefore did not use the coordinate transformations included in .NET, as they also transform line thicknesses and fonts. This is also one of the reasons why
GraphicsPath was no good)! The reason for using 3 objects instead of 2 is that in doing so, annotations can be grouped together logically. Moreover, it is possible to combine the annotations in a
Container to create a region, based on the
RegionAddMode property of the individual annotations. This makes it easy to create several complex regions, and to use these, e.g. for further image processing. To give an example, I use this at work to compute color differences for clinical images of the skin, with a 'normal skin' and a 'lesion' container. With skin lesions sometimes having a mottled appearance, i.e. with patches of normal skin inside of them, this would not have been possible with a 2 object approach.
An annotation can be added to an image by adding it to the annotation
Container, after this
Container has been added to the top-level object
Dim objAnnSet As Annotations.ContainerCollection = _
Dim objAnnC1 As New Annotations.Container("Normal")
Dim objAnn As New Annotations.AnnotationBorder("Annotation 1", objColor)
objAnn.PointsVisible = False
Dim objrect as new rectangle
In contrast to the first version, all collections are now type safe, and for those who are interested in how to do this, check out the following snippet:
Public Class ContainerCollection
Public Sub New()
Default Public ReadOnly Property Item(ByVal key As String) As Container
Return CType(Me.BaseGet(key), Container)
Default Public ReadOnly Property Item(ByVal index As Integer) As Container
Return CType(Me.BaseGet(index), Container)
Public Sub Add(ByVal value As Container)
If Me.KeyExists(value.Key) Then Me.BaseRemove(value.Key)
Public Function KeyExists(ByVal key As String) As Boolean
If Me.Item(key) Is Nothing Then
Public Overloads Sub Remove(ByVal key As String)
Public Overloads Sub Remove(ByVal index As Integer)
Public Sub Clear()
Public Shadows Function GetEnumerator() As ContainerCollectionEnumerator
Return New ContainerCollectionEnumerator(Me)
Public Class ContainerCollectionEnumerator
Private objContainerCollection As ContainerCollection
Private index As Integer = -1
Public Sub New(ByVal objContainerCollection As ContainerCollection)
Me.objContainerCollection = objContainerCollection
Public ReadOnly Property Current() As Object Implements IEnumerator.Current
Return CType(Me.objContainerCollection.Item(index), Container)
Public Function MoveNext() As Boolean Implements IEnumerator.MoveNext
If index < objContainerCollection.Count - 1 Then
index += 1
Public Sub Reset() Implements IEnumerator.Reset
index = -1
In order to provide smooth redrawing, double buffering was activated on the
AnnotatedImage control by settings some of its style properties:
The important point here is that you must use the graphics object provided by the paint event of the
AnnotatedImage in the drawing routines of the individual annotations. Indeed, with double buffering, this object is actually pointing to an in-memory buffer which is copied selectively to the client area of the control when drawing is finished. This also means it is not appropriate to try to invalidate limited areas of the control when redrawing annotations to speed up redrawing, because we would need to know the bounding box of the annotations to be drawn, before we draw them (and add them to the bounding box of the already drawn annotations). This is tedious and would mean a lot of extra code, with little or no gain as the double buffering already does part of this for us and is pretty fast.
Comments, to do, bugs
This component has been used in a user control specialized in sRGB image viewing, which in its turn is used in an application for measuring skin lesions using calibrated sRGB images. It can be found here. So far this application has been remarkably stable compared to a previous version using VB6 and leadtools 11.
To-do's include mainly serialization of annotations to file, probably using XML, and maybe the ability to lock annotations with passwords (important in medical environments).