Click here to Skip to main content
15,868,016 members
Articles / Programming Languages / Visual Basic
Article

A custom control for images with annotations

Rate me:
Please Sign up or sign in to vote.
4.44/5 (11 votes)
7 Mar 20047 min read 161.3K   946   58   45
An article about a custom control for displaying images with the possibility of adding annotations.

Image 1

Introduction

In image processing, it is often useful to display overlay graphics on images, e.g. to display a segmented region, but this is not provided for in .NET. I decided to create a custom control that can easily be included into other projects and that provides some basic functionality for images, like zooming, panning and the display and creation of annotations. Together with a library for image processing which I will present later, this should provide the basic building blocks for any application that tackles image processing.

The annotation classes

My first try at this problem as published here was a bit haphazard, and left me not very satisfied. Indeed, the object hierarchy wasn't logical, and the collections weren't type safe. Also, animation wasn't flicker-free. I decided to rework the whole project using a clean object oriented approach, and came up with the following:

  • Class AnnotatedImage

    Control that holds the bitmap on which the annotations are drawn, and the properties and methods for drawing that bitmap. This is the control that is to be included in other projects.

    • ImageSize
    • DisplaySize
    • DisplayOffset
    • DisplayZoom
    • Bitmap
    • Annotations
    • MaximumSize
    • PixelAveragingWindowSize
    • DrawErasableRectangle
    • DrawErasableLine
    • SaveImage
    • LoadImage
  • Class Annotations

    Class that contains all code for the annotations, and all its collections.

    • Class Container

      Class used to contain annotations.

    • Class ContainerCollection

      Class defining a strongly typed collection of Containers.

    • Class Annotation

      Abstract class used to derive more specific annotation classes.

      • Key
      • RegionAddMode
      • Color
      • Visible
      • Text
      • Key
      • Points
      • Region
      • BorderRegion (Readonly)
      • AddPoint
      • RemovePoint
      • ClearPoints
    • Class AnnotationCollection

      Class defining a strongly typed collection of annotations

    • Class AnnotationRegion

      Class for an annotation consisting of points that form a region, as such.

    • Class PointBasedAnnotation

      Abstract class used to derive more specific point-based annotation classes

      • Annotation properties and methods
      • LinesVisible
      • PointsVisible
      • MaximumNumberOfPoints
    • Class AnnotationLoose

      Class for an annotation consisting of loose points, i.e. points which do not define a region

    • Class AnnotationBorder

      Class for an annotation consisting of points that form a closed contour and thus define a region by forming a border.

      • PointBasedAnnotation properties and methods
      • AddRectangle

The classes AnnotatedImage and Annotations are meant to work together, and they hold references to each other. By instancing an AnnotatedImage, you get the following hierarchy:

  • AnnotatedImage
    • Annotations
      • Containers (Type safe collection)
        • Container
          • Annotations (Type safe collection)
            • Annotation

All the code for drawing the annotations is inside the Annotations class, except for 2 routines for XOR drawing that are stored inside the AnnotatedImage. The Annotations class intercepts the Paint event of the AnnotatedImage control in order to redraw every annotation Container object, which in turn calls the 'Draw' method on every individual Annotation object. The AnnotatedImage control implements double buffering to ensure smooth drawing.

All events pertaining to annotations are handled in the Annotations class, e.g. interactive drawing, and all those concerned with the image in AnnotatedImage, e.g. zooming. This means a Windows event like a mouse click, will trigger several event handlers, and these will decide if they should do something with it.

Using the code

The code will compile to a DLL that can be added to the Visual Studio toolbox, and can then be included as a control in any application. Of all the events exposed, Resize is the only one that must be handled, as it is through this event that the parent control can resize itself in order to accommodate changes in the size of the custom control, e.g. after zooming. The custom control usually has the size of the displayed image, except when it exceeds MaximumSize for very large images or a large displayZoom (do not forget to set this property!). In that case, only a portion of the image is displayed, and panning can be activated by dragging the image with the middle mouse button. On top of that, the control responds to the wheel mouse and z and shift-z for zooming, to w and shift-w for setting the averaging window size that can be used to provide feedback to the user about the color in the area around the cursor. This averaging window is also drawn on the control.

Explanations and points of interest

At first, I tried to derive my custom control from the PictureBox class, but I failed to draw any graphics over the bitmap that can be assigned to the PictureBox.Bitmap by overriding the Paint method. Indeed, it seemed that the bitmap was drawn after any custom drawing I performed on the graphics created from the PictureBox (this was puzzling). Drawing on the bitmap itself was, of course, no option as it corrupts the bitmap. Consequently, I opted to derive from the control class (the scrollablecontainer didn't work well either). I tried to include a horizontal and vertical scrollbar, but these wouldn't render on their own, and I found no easy way to paint them manually on the control. So, instead I implemented panning by dragging the middle mouse button, which becomes active when the image cannot fit into the custom control anymore.

Annotations are implemented using 3 extra classes: an Annotations top-level object, an annotation Container and an Annotation object. This last object is actually an abstract class, with the really instantiable classes being AnnotationLoose, AnnotationBorder and AnnotationRegion. I stayed away from the GraphicsPath object because it didn't fit my purpose very well, although it is used sometimes internally when computing regions. Each AnnotatedImage image control has 1 Annotations object, that can hold several Containers, which each can hold several Annotation objects. These can be of different subtypes. Note that these objects are not to be confused with vector graphic objects that are drawn on a background bitmap, rather they are conceptual and only represent certain divisions or regions in images. It is for this reason that these annotations have no intrinsic line thickness and are represented by a line of thickness of 1 pixel, regardless of the zoom factor of the image (I therefore did not use the coordinate transformations included in .NET, as they also transform line thicknesses and fonts. This is also one of the reasons why GraphicsPath was no good)! The reason for using 3 objects instead of 2 is that in doing so, annotations can be grouped together logically. Moreover, it is possible to combine the annotations in a Container to create a region, based on the RegionAddMode property of the individual annotations. This makes it easy to create several complex regions, and to use these, e.g. for further image processing. To give an example, I use this at work to compute color differences for clinical images of the skin, with a 'normal skin' and a 'lesion' container. With skin lesions sometimes having a mottled appearance, i.e. with patches of normal skin inside of them, this would not have been possible with a 2 object approach.

An annotation can be added to an image by adding it to the annotation Container, after this Container has been added to the top-level object Annotations.

VB
 Dim objAnnSet As Annotations.ContainerCollection = _
                          AnnotatedImage.Annotations.Containers
 Dim objAnnC1 As New Annotations.Container("Normal")

 objAnnSet.Add(objAnnC1)

 Dim objAnn As New Annotations.AnnotationBorder("Annotation 1", objColor)
 objAnnC1.Annotations.Add(objAnn)
 objAnn.PointsVisible = False

 'Set rectangle
 Dim objrect as new rectangle
 ...
 'Add to annotation
 objAnn.AddRectangle(objrect)

 'Draw it
 AnnotatedImage.Refresh()

In contrast to the first version, all collections are now type safe, and for those who are interested in how to do this, check out the following snippet:

VB
Public Class ContainerCollection
   Inherits NameObjectCollectionBase

   Public Sub New()
   End Sub 'New

   Default Public ReadOnly Property Item(ByVal key As String) As Container
     Get
       Return CType(Me.BaseGet(key), Container)
     End Get
   End Property

   Default Public ReadOnly Property Item(ByVal index As Integer) As Container
     Get
       Return CType(Me.BaseGet(index), Container)
     End Get
   End Property

   Public Sub Add(ByVal value As Container)
     'Key must be unique, so we remove existing entry
     If Me.KeyExists(value.Key) Then Me.BaseRemove(value.Key)
     Me.BaseAdd(value.Key, value)
   End Sub

   Public Function KeyExists(ByVal key As String) As Boolean
     If Me.Item(key) Is Nothing Then
       Return False
     Else
       Return True
     End If
   End Function

   Public Overloads Sub Remove(ByVal key As String)
     Me.BaseRemove(key)
   End Sub

   Public Overloads Sub Remove(ByVal index As Integer)
     Me.BaseRemoveAt(index)
   End Sub

   Public Sub Clear()
     Me.BaseClear()
   End Sub

   Public Shadows Function GetEnumerator() As ContainerCollectionEnumerator
     Return New ContainerCollectionEnumerator(Me)
   End Function

   Public Class ContainerCollectionEnumerator
     ' This overriding of IEnumerator allows for each to be used
     Implements IEnumerator

     Private objContainerCollection As ContainerCollection
     Private index As Integer = -1

     Public Sub New(ByVal objContainerCollection As ContainerCollection)
       Me.objContainerCollection = objContainerCollection
     End Sub

     Public ReadOnly Property Current() As Object Implements IEnumerator.Current
       Get
         Return CType(Me.objContainerCollection.Item(index), Container)
       End Get
     End Property

     Public Function MoveNext() As Boolean Implements IEnumerator.MoveNext
       If index < objContainerCollection.Count - 1 Then
         index += 1
         Return True
       Else
         Return False
       End If
     End Function

     Public Sub Reset() Implements IEnumerator.Reset
       index = -1
     End Sub
   End Class

End Class

In order to provide smooth redrawing, double buffering was activated on the AnnotatedImage control by settings some of its style properties:

VB
setstyle(ControlStyles.UserPaint, True), 
     setstyle(ControlStyles.AllPaintingInWmPaint, 
     True),setstyle(ControlStyles.DoubleBuffer, True)

The important point here is that you must use the graphics object provided by the paint event of the AnnotatedImage in the drawing routines of the individual annotations. Indeed, with double buffering, this object is actually pointing to an in-memory buffer which is copied selectively to the client area of the control when drawing is finished. This also means it is not appropriate to try to invalidate limited areas of the control when redrawing annotations to speed up redrawing, because we would need to know the bounding box of the annotations to be drawn, before we draw them (and add them to the bounding box of the already drawn annotations). This is tedious and would mean a lot of extra code, with little or no gain as the double buffering already does part of this for us and is pretty fast.

Comments, to do, bugs

This component has been used in a user control specialized in sRGB image viewing, which in its turn is used in an application for measuring skin lesions using calibrated sRGB images. It can be found here. So far this application has been remarkably stable compared to a previous version using VB6 and leadtools 11.

To-do's include mainly serialization of annotations to file, probably using XML, and maybe the ability to lock annotations with passwords (important in medical environments).

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here


Written By
Software Developer (Senior)
Belgium Belgium
Physicist, Biomedical Engineer, Phd in engineering. Specific expertise is in medical photography and it's related image processing, and in colourimetry.

Comments and Discussions

 
QuestionCan you Save the image marked by the annotation Pin
coolpjmartin14-Apr-09 4:02
coolpjmartin14-Apr-09 4:02 
AnswerRe: Can you Save the image marked by the annotation Pin
yvdh14-Apr-09 5:12
yvdh14-Apr-09 5:12 
GeneralRe: Can you Save the image marked by the annotation Pin
coolpjmartin15-Apr-09 22:05
coolpjmartin15-Apr-09 22:05 
GeneralRe: Can you Save the image marked by the annotation Pin
yvdh16-Apr-09 0:02
yvdh16-Apr-09 0:02 
GeneralRe: Can you Save the image marked by the annotation Pin
coolpjmartin16-Apr-09 21:54
coolpjmartin16-Apr-09 21:54 
GeneralRe: Can you Save the image marked by the annotation Pin
yvdh16-Apr-09 22:09
yvdh16-Apr-09 22:09 
GeneralRe: Can you Save the image marked by the annotation Pin
coolpjmartin20-Apr-09 23:00
coolpjmartin20-Apr-09 23:00 
GeneralRe: Can you Save the image marked by the annotation Pin
yvdh21-Apr-09 11:44
yvdh21-Apr-09 11:44 
GeneralRe: Can you Save the image marked by the annotation Pin
coolpjmartin23-Apr-09 1:46
coolpjmartin23-Apr-09 1:46 
GeneralRe: Can you Save the image marked by the annotation Pin
yvdh23-Apr-09 2:31
yvdh23-Apr-09 2:31 
GeneralRe: Can you Save the image marked by the annotation Pin
coolpjmartin30-Apr-09 2:04
coolpjmartin30-Apr-09 2:04 
Hi Yvdh, Thanks for all your help, I will play around and see how I get on, if you do manage to upload the latest dll, that would be a great help, could you please let me know the link to download, thanks again.
GeneralRe: Can you Save the image marked by the annotation Pin
coolpjmartin28-Jun-09 21:29
coolpjmartin28-Jun-09 21:29 
GeneralRe: Can you Save the image marked by the annotation Pin
yvdh29-Jun-09 3:24
yvdh29-Jun-09 3:24 
GeneralRe: Can you Save the image marked by the annotation Pin
coolpjmartin4-Jul-09 23:30
coolpjmartin4-Jul-09 23:30 
GeneralFlipping Images Pin
rjlazo_145-Dec-08 12:53
rjlazo_145-Dec-08 12:53 
GeneralRe: Flipping Images Pin
yvdh5-Dec-08 14:09
yvdh5-Dec-08 14:09 
GeneralRe: Flipping Images Pin
rjlazo_145-Dec-08 14:49
rjlazo_145-Dec-08 14:49 
QuestionRe: Flipping Images Pin
rjlazo_148-Dec-08 11:38
rjlazo_148-Dec-08 11:38 
AnswerRe: Flipping Images Pin
yvdh8-Dec-08 13:04
yvdh8-Dec-08 13:04 
GeneralRe: Flipping Images Pin
rjlazo_149-Dec-08 11:50
rjlazo_149-Dec-08 11:50 
GeneralRe: Flipping Images Pin
yvdh9-Dec-08 21:11
yvdh9-Dec-08 21:11 
GeneralGreat Work Pin
richard148014-Nov-08 11:23
richard148014-Nov-08 11:23 
GeneralRe: Great Work Pin
yvdh14-Nov-08 13:32
yvdh14-Nov-08 13:32 
QuestionRe: Great Work Pin
richard148019-Nov-08 7:37
richard148019-Nov-08 7:37 
AnswerRe: Great Work Pin
yvdh19-Nov-08 10:22
yvdh19-Nov-08 10:22 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.