A custom control for images with annotations

• Download source - 27.9 Kb

Introduction

In image processing, it is often useful to display overlay graphics on images, e.g. to display a segmented region, but this is not provided for in .NET. I decided to create a custom control that can easily be included into other projects and that provides some basic functionality for images, like zooming, panning and the display and creation of annotations. Together with a library for image processing which I will present later, this should provide the basic building blocks for any application that tackles image processing.

The annotation classes

My first try at this problem as published here was a bit haphazard, and left me not very satisfied. Indeed, the object hierarchy wasn't logical, and the collections weren't type safe. Also, animation wasn't flicker-free. I decided to rework the whole project using a clean object oriented approach, and came up with the following:

• Class AnnotatedImage

  Control that holds the bitmap on which the annotations are drawn, and the properties and methods for drawing that bitmap. This is the control that is to be included in other projects.

  • ImageSize
  • DisplaySize
  • DisplayOffset
  • DisplayZoom
  • Bitmap
  • Annotations
  • MaximumSize
  • PixelAveragingWindowSize
  • DrawErasableRectangle
  • DrawErasableLine
  • SaveImage
  • LoadImage

• Class Annotations

  Class that contains all code for the annotations, and all its collections.

  • Class Container

    Class used to contain annotations.

  • Class ContainerCollection

    Class defining a strongly typed collection of Containers.

  • Class Annotation

    Abstract class used to derive more specific annotation classes.

    • Key
    • RegionAddMode
    • Color
    • Visible
    • Text
    • Key
    • Points
    • Region
    • BorderRegion (Readonly)
    • AddPoint
    • RemovePoint
    • ClearPoints

  • Class AnnotationCollection

    Class defining a strongly typed collection of annotations

  • Class AnnotationRegion

    Class for an annotation consisting of points that form a region, as such.

  • Class PointBasedAnnotation

    Abstract class used to derive more specific point-based annotation classes

    • Annotation properties and methods
    • LinesVisible
    • PointsVisible
    • MaximumNumberOfPoints

  • Class AnnotationLoose

    Class for an annotation consisting of loose points, i.e. points which do not define a region

  • Class AnnotationBorder

    Class for an annotation consisting of points that form a closed contour and thus define a region by forming a border.

    • PointBasedAnnotation properties and methods
    • AddRectangle

The classes AnnotatedImage and Annotations are meant to work together, and they hold references to each other. By instancing an AnnotatedImage, you get the following hierarchy:

• AnnotatedImage
  • Annotations
    • Containers (Type safe collection)
      • Container
        • Annotations (Type safe collection)
          • Annotation

All the code for drawing the annotations is inside the Annotations class, except for 2 routines for XOR drawing that are stored inside the AnnotatedImage. The Annotations class intercepts the Paint event of the AnnotatedImage control in order to redraw every annotation Container object, which in turn calls the 'Draw' method on every individual Annotation object. The AnnotatedImage control implements double buffering to ensure smooth drawing.

All events pertaining to annotations are handled in the Annotations class, e.g. interactive drawing, and all those concerned with the image in AnnotatedImage, e.g. zooming. This means a Windows event like a mouse click, will trigger several event handlers, and these will decide if they should do something with it.

Using the code

The code will compile to a DLL that can be added to the Visual Studio toolbox, and can then be included as a control in any application. Of all the events exposed, Resize is the only one that must be handled, as it is through this event that the parent control can resize itself in order to accommodate changes in the size of the custom control, e.g. after zooming. The custom control usually has the size of the displayed image, except when it exceeds MaximumSize for very large images or a large displayZoom (do not forget to set this property!). In that case, only a portion of the image is displayed, and panning can be activated by dragging the image with the middle mouse button. On top of that, the control responds to the wheel mouse and z and shift-z for zooming, to w and shift-w for setting the averaging window size that can be used to provide feedback to the user about the color in the area around the cursor. This averaging window is also drawn on the control.

Explanations and points of interest

At first, I tried to derive my custom control from the PictureBox class, but I failed to draw any graphics over the bitmap that can be assigned to the PictureBox.Bitmap by overriding the Paint method. Indeed, it seemed that the bitmap was drawn after any custom drawing I performed on the graphics created from the PictureBox (this was puzzling). Drawing on the bitmap itself was, of course, no option as it corrupts the bitmap. Consequently, I opted to derive from the control class (the scrollablecontainer didn't work well either). I tried to include a horizontal and vertical scrollbar, but these wouldn't render on their own, and I found no easy way to paint them manually on the control. So, instead I implemented panning by dragging the middle mouse button, which becomes active when the image cannot fit into the custom control anymore.

Annotations are implemented using 3 extra classes: an Annotations top-level object, an annotation Container and an Annotation object. This last object is actually an abstract class, with the really instantiable classes being AnnotationLoose, AnnotationBorder and AnnotationRegion. I stayed away from the GraphicsPath object because it didn't fit my purpose very well, although it is used sometimes internally when computing regions. Each AnnotatedImage image control has 1 Annotations object, that can hold several Containers, which each can hold several Annotation objects. These can be of different subtypes. Note that these objects are not to be confused with vector graphic objects that are drawn on a background bitmap, rather they are conceptual and only represent certain divisions or regions in images. It is for this reason that these annotations have no intrinsic line thickness and are represented by a line of thickness of 1 pixel, regardless of the zoom factor of the image (I therefore did not use the coordinate transformations included in .NET, as they also transform line thicknesses and fonts. This is also one of the reasons why GraphicsPath was no good)! The reason for using 3 objects instead of 2 is that in doing so, annotations can be grouped together logically. Moreover, it is possible to combine the annotations in a Container to create a region, based on the RegionAddMode property of the individual annotations. This makes it easy to create several complex regions, and to use these, e.g. for further image processing. To give an example, I use this at work to compute color differences for clinical images of the skin, with a 'normal skin' and a 'lesion' container. With skin lesions sometimes having a mottled appearance, i.e. with patches of normal skin inside of them, this would not have been possible with a 2 object approach.

An annotation can be added to an image by adding it to the annotation Container, after this Container has been added to the top-level object Annotations.

   Dim objAnnSet As Annotations.ContainerCollection = _
                            AnnotatedImage.Annotations.Containers
   Dim objAnnC1 As New Annotations.Container("Normal")

   objAnnSet.Add(objAnnC1)

   Dim objAnn As New Annotations.AnnotationBorder("Annotation 1", objColor)
   objAnnC1.Annotations.Add(objAnn)
   objAnn.PointsVisible = False

   'Set rectangle

   Dim objrect as new rectangle
   ...
   'Add to annotation

   objAnn.AddRectangle(objrect)

   'Draw it

   AnnotatedImage.Refresh()

In contrast to the first version, all collections are now type safe, and for those who are interested in how to do this, check out the following snippet:

Public Class ContainerCollection
   Inherits NameObjectCollectionBase

   Public Sub New()
   End Sub 'New


   Default Public ReadOnly Property Item(ByVal key As String) As Container
     Get
       Return CType(Me.BaseGet(key), Container)
     End Get
   End Property

   Default Public ReadOnly Property Item(ByVal index As Integer) As Container
     Get
       Return CType(Me.BaseGet(index), Container)
     End Get
   End Property

   Public Sub Add(ByVal value As Container)
     'Key must be unique, so we remove existing entry

     If Me.KeyExists(value.Key) Then Me.BaseRemove(value.Key)
     Me.BaseAdd(value.Key, value)
   End Sub

   Public Function KeyExists(ByVal key As String) As Boolean
     If Me.Item(key) Is Nothing Then
       Return False
     Else
       Return True
     End If
   End Function

   Public Overloads Sub Remove(ByVal key As String)
     Me.BaseRemove(key)
   End Sub

   Public Overloads Sub Remove(ByVal index As Integer)
     Me.BaseRemoveAt(index)
   End Sub

   Public Sub Clear()
     Me.BaseClear()
   End Sub

   Public Shadows Function GetEnumerator() As ContainerCollectionEnumerator
     Return New ContainerCollectionEnumerator(Me)
   End Function

   Public Class ContainerCollectionEnumerator
     ' This overriding of IEnumerator allows for each to be used

     Implements IEnumerator

     Private objContainerCollection As ContainerCollection
     Private index As Integer = -1

     Public Sub New(ByVal objContainerCollection As ContainerCollection)
       Me.objContainerCollection = objContainerCollection
     End Sub

     Public ReadOnly Property Current() As Object Implements IEnumerator.Current
       Get
         Return CType(Me.objContainerCollection.Item(index), Container)
       End Get
     End Property

     Public Function MoveNext() As Boolean Implements IEnumerator.MoveNext
       If index < objContainerCollection.Count - 1 Then
         index += 1
         Return True
       Else
         Return False
       End If
     End Function

     Public Sub Reset() Implements IEnumerator.Reset
       index = -1
     End Sub
   End Class

End Class

In order to provide smooth redrawing, double buffering was activated on the AnnotatedImage control by settings some of its style properties:

setstyle(ControlStyles.UserPaint, True), 
     setstyle(ControlStyles.AllPaintingInWmPaint, 
     True),setstyle(ControlStyles.DoubleBuffer, True)

The important point here is that you must use the graphics object provided by the paint event of the AnnotatedImage in the drawing routines of the individual annotations. Indeed, with double buffering, this object is actually pointing to an in-memory buffer which is copied selectively to the client area of the control when drawing is finished. This also means it is not appropriate to try to invalidate limited areas of the control when redrawing annotations to speed up redrawing, because we would need to know the bounding box of the annotations to be drawn, before we draw them (and add them to the bounding box of the already drawn annotations). This is tedious and would mean a lot of extra code, with little or no gain as the double buffering already does part of this for us and is pretty fast.

Comments, to do, bugs

This component has been used in a user control specialized in sRGB image viewing, which in its turn is used in an application for measuring skin lesions using calibrated sRGB images. It can be found here. So far this application has been remarkably stable compared to a previous version using VB6 and leadtools 11.

To-do's include mainly serialization of annotations to file, probably using XML, and maybe the ability to lock annotations with passwords (important in medical environments).