Computer Vision Applications with C# - Part II

By , 25 Apr 2009

Introduction

If you have taken high school Physics, you would know something about Moments. From Physics, it found its way into probability theory, and since computer vision leverages probability theory, it is only natural to use Moments in computer vision. So moving on from Part I, I will discuss image moments and how to use them. Since the theme of this series of articles is Application, I will always emphasis on putting theory to practice. So let's start!

Image Moments

Image moments can simply be described as some function of the image pixel intensity. A pixel intensity is nothing but the pixel colour value. Moments are described with respect to their power as in raised-to-the-power in Mathematics. So, we have Zeroth Moment, First Moment, Second Moment etc. Here is the formula for finding Moments:

Note that from basic algebra, we know anything raised to the power 0 is 1, so that is why some of the terms above have disappeared. Also, Zeroth Moment is also called the Area of an image. So basically, what we need is the pixel intensity or value and the x and y coordinates to find moments. Using moments, we can find the centroid or centre of gravity of an image. There are some more things that can be found out by using moments.

Histogram Backprojection

Histogram backprojection is a technique used for creating a probability image (backprojection image) of the object of interest. For example, suppose we create a histogram of a target in an image. If we backproject this histogram onto the image, it will create an image that shows the probability of a pixel belonging to the target.

Let's dive into it - once we have the target histogram `histT`, we create the histogram of the image `histImage` on which we want to backproject the target histogram. Then, for each pixel value in the image, we get the bin index `idx` from the image histogram. We set the value of the pixel in the backprojection image by using the ratio of the value of the target histogram and the value of the image histogram at the bin index, i.e., `histT[idx] / histImage/[idx]` as shown below.

```Bitmap bpImage = new Bitmap(bmp.Width, bmp.Height, bmp.PixelFormat);
Histogram frameHistogram = Create1DHistogram(bmp, sourceHistogram.BinCount1,
sourceHistogram.BinCount2, sourceHistogram.BinCount3);
:
:
for (int y = 0; y < size.Y; y++)
{
pPixel = fastBitmap[0, y];
pPixelBP = fastBitmapBP[0, y];
for (int x = 0; x < size.X; x++)
{
//get the bin index for the current pixel colour
idx = GetSingleBinIndex(sourceHistogram.BinCount1,
sourceHistogram.BinCount2, sourceHistogram.BinCount3, pPixel);

//find the value for backprojection image and set
if (frameHistogram.Data[idx] > 0) //avoid divide by zero error
val = (byte)(255 * (int)(sourceHistogram.Data[idx] /
frameHistogram.Data[idx]));
else
val = 0;

pPixelBP->red = val;
pPixelBP->green = val;
pPixelBP->blue = val;

//increment the pointer
pPixel++;
pPixelBP++;
}
}```

In the above code, we created a blank backprojection image, created the histogram for the current frame, and iterated through the frame pixels. For each pixel, we find the bin index, and using this bin index, we get the value of the target histogram and the frame histogram. We find the ratio of these values and scale it such that the ratio value ranges from 0 to 255. This value is finally used to set the pixel value in the backprojection image. So, here is an example of a backprojection image:

In the above figure, I have selected the target with the mouse. It is shown as the red rectangle (ignore the yellow rectangle for now). Then, the histogram of the selected target is backprojected (right image) onto the same image frame. We can see that if the bin values of the target and the frame corresponding to a pixel index are the same, the resulting pixel value in the backprojection image will be 255. So, the brighter the pixel value, higher the probability that it belongs to the target. In the backprojection image (ignore the red and blue line for now), we can clearly see the target as well as some noise, i.e., a vertical line and some area in the top left.

The reason to discuss Histogram Backprojection is because:

1. It is a useful technique to have in the bag
3. I am going to implement a simple object tracker using backprojection.

Using backprojection image and moments, we can find the orientation of the target, and the length and the width of the probability distribution. Here are the formulas to use:

The angle is the orientation of the object about the major axis l (the red line above) whereas w is the minor axis (the blue line above).

Centroid Tracker

Now, it is time to do something more exciting - let's track an object in an image sequence. How would we do it? Let's do the following:

1. Select the object in the image and create a histogram of the area inside the target window.
2. Segment the object using backprojection. But, instead of projecting onto the whole frame, we project it onto the search area (yellow box) only to reduce the risk of tracker failure.
3. Find the centroid of this object.
4. Get the next frame, and segment the section under the search window using histogram backprojection.
5. Find the centroid, and centre the target and search windows at the centroid.
6. Repeat 4 till the end of the image sequence.

We should realise that using a search window around the original object is very important. It limits the scope of segmentation to a smaller area. This means the probability of the tracker getting lost because of a similar coloured object in the background is reduced. Furthermore, limited area increases the processing speed, making the tracker very fast.

Using the Code - Moments & Centroid

Step 1: Select a histogram size - the default is 4x4x4.

Step 2: Select an image from the list and click the "Select" button.

Step 3: Select an area in the image using the mouse - right click the mouse and drag. This will create the selected area histogram, backprojected onto the whole image, calculate the moments, and mark the centroid of the selected area.

Using the Code - Centroid Tracking

Step 1: Select a histogram size - the default is 4x4x4

Step 2: Click "Load Sequence" button to see its histogram of the first image.

Step 3: Select an area in the image using mouse - Right click the mouse & drag. This will backproject only onto the search area, calculate moments and mark the centroid of the selected area. The target histogram is shown in red and will not change during the tracking.

Step 4: Click "Track" button to see the tracking in action. The candidate histogram is shown in blue, and will change during tracking as it is the histogram of the area that the tracker "thinks" is the target.

You can drop bitmaps from your own sequence (named 0 to N) in the sequence folder and try the tracker on it.

In this article, we learned about image moments, and then we used our creativity to apply what we learned about moments to create a simple object tracker. Obviously, the tracker is a simple one, but can be very effective given correct circumstances. But in the real world, we do not always have the luxury of correct circumstances. We cannot control the environment where our tracker will be deployed. There will be multiple objects in the scene doing different things, with different colours etc. So, we need a more robust object tracker. This will be the topic of my next article, where I will discuss a very popular tracking algorithm called ....... "Meanshift Algorithm" - Stay tuned.

Acknowledgements

For any third party code or material used, please refer to "Points of Interest" in Part I.

History

• Version 1.0.

 Arif_Khan Australia Member
I have been in the IT industry since April 1996. My main expertise is in Microsoft space.

Coming from engineering background, any application of programming to engineering and related fields easily excites me. I like to use OO and design patterns and find them very useful.

I have been an avid reader of CodeProject. I decided it was time to make a commitment to make my contribution to the community - so here I am.

My Website: http://www.puresolutions-online.com

Votes of 3 or less require a comment

 Search this forum Profile popups    Spacing RelaxedCompactTight   Noise Very HighHighMediumLowVery Low   Layout Open AllThread ViewNo JavascriptPreview   Per page 102550
 First Prev Next
 Thanks Çağrı Daşkın 25 May '12 - 3:45
 Hello Master,   I am a Geomatic Engineering student. I am working on image processing. I was searching mean shift and i found your three articles. These articles really improved my vision. Especially some of your keywords in quotes, like "statistics and probability" are very useful for students and programmers new in image processing.   Thanks... Sign In·View Thread·Permalink
 Re: Thanks Arif_Khan 25 May '12 - 13:01
 Thanks for your message.   I wrote these articles because when I was doing research, I could not find information to clarify my concepts. So when I got my grasp on these topics, I decided to write this 4 part series to help others. I am glad my efforts paid off.   Regards Sign In·View Thread·Permalink
 Re: Thanks priyank_thakkar 30 Jun '12 - 10:42
 Hi Arif,   Thanks for the wonderful insight to the computer vision part. I am looking for the guidance. I am working on the image processing. I am looking for the text detection within an image. I went through the OCR libraries available but they are good at detecting characters not the word as whole. Can you guide me?   Regards, Priyank Thakkar Sign In·View Thread·Permalink
 Hi, I found your article by Google search. It is a clear and simple tutorial which is better than related stuff in wikipedia. I can't find any tutorial better than this one! Thank you very much! By the way, I'm waiting for the next article about meanshift. Why not you give OpenCV library a try? Sign In·View Thread·Permalink
 Thanks! Meanshift is already posted here: Computer Vision Application with C# - Part III[^]   I have used OpenCV but not everyone has C skills. I wanted to have Meanshift implemented from scratch so people can get their hands dirty with coding it - which means they will get a better understanding. Hope you enjoy it! Sign In·View Thread·Permalink
 Thanks for the quick reply. Oh, the Part three was published this month, so, I suggest you can add a hyperlink in Part two. Yes, implemented from scratch would let people know everything behind the scene. I have C/C++ language background, but haven't touch C#. Is there any difficulties on the learn curve? Thanks. Sign In·View Thread·Permalink
 You have to stop thinking in terms of pointers and realise that you can use the objects from C# framework. Although, the image processing code uses pointers for fast processing. Sign In·View Thread·Permalink
 Hi. I just read the OpenCV code about   ```void cvCalcBackProject( IplImage** image, CvArr* back_project, const CvHistogram* hist ); ```   But compare with your explanation about back project, they just copy the histT[idx] value, instead of histT[idx] / histImage/[idx], is that reasonable?   Thanks. Sign In·View Thread·Permalink
 Hello - I haven't looked at the code for the OpenCV function but the method described in this article is correct. Backprojection is a technique proposed by Swain & Ballard in their paper on Colour Indexing. If you google for this paper, you should be able to get it.   It is actually this: min(Mi/Ri, 1). You have to take the minimum of ratio or 1 and scale it to 255. In my code, I am just taking the ratio. Hope it helps! Sign In·View Thread·Permalink
 Thanks for your explanation.It seems there are some difference of the backprojection method. I can confirm that OpenCV code just use the Target histogram( not a ratio histogram). I'm currently discussed with some OpenCV guys. Sign In·View Thread·Permalink
 BackProjectRoi sasasasa 11 May '09 - 22:54
 I checked your code and want to discuss something...   In "CreateBackprojectionImage" your so called "frameHistogram" shouldn't be calculate from so called "bmp"'s ROI? Currently it is beign calculated from all "bmp". Sign In·View Thread·Permalink
 Re: BackProjectRoi Arif_Khan 12 May '09 - 0:48
 It is good to see people trying to understand the code. A target histogram is normally backprojected on to the whole frame to reveal the pixels that have the high probability of belonging to the target. In this method ROI is passed in to avoid scanning the whole frame and only considering the image under the ROI. Since we are only processing the ROI, we could create the histogram of only the ROI region of bmp - but the end result in terms of backprojection will be the same. For proper tracking, I would refer you to Part III of this series. Sign In·View Thread·Permalink
 Interested in next part juergen1969 27 Apr '09 - 21:30
 Hi Arif, I'm very intersted in the next part. I tried something in a very easy way with a webcam before but hadn't enough time to complete it.   Best regards, Juergen Sign In·View Thread·Permalink
 Re: Interested in next part Arif_Khan 28 Apr '09 - 0:44
 simple question Shakeel Mumtaz 27 Apr '09 - 20:01