Click here to Skip to main content
Click here to Skip to main content

Tracking an object from a live video input

, 6 May 2005
Rate this:
Please Sign up or sign in to vote.
Track an object based on its features, using the AVICap window class.

Sample screenshot

Introduction

As part of my research project, I had to implement a feature tracking device that runs entirely on a hardware board. Designing things, especially useful things on a piece of hardware, takes effort and time. To avoid any tedious calibrations of algorithms on board and to ensure the algorithms are all properly designed, I wrote a Windows application to simulate the environment - grabbing frames from a web camera and track. In exactly the same way as I have benefited from open source projects, I would certainly enjoy spending some time contributing to a site such as The Code Project.

AVICap

In this demo application, I have chosen to demonstrate the use of AVICap window class to track objects. AVICap is a window class that provides applications with an extremely convenient programming interface to access video acquisition hardware such as a web camera used in this demo application.

To be able to track objects from a live video input, we obviously need to gain access to individual frames. To gain access to individual frames before they are previewed, use the capSetCallbackOnFrame macro.

BOOL capSetCallbackOnFrame(HWND hwnd, FrameCallback fpProc);
  • HWND hwnd: Handle to the capture window.
  • FrameCallback fpProc: Pointer to the preview callback function. Specify NULL for this parameter to disable a previously installed callback function.
typedef LRESULT (*FrameCallback)(HWND hWnd, LPVIDEOHDR lpVideoHdr);

The LPVIDEOHDR is declared as follows:

typedef struct videohdr_tag {
    LPBYTE lpData;          /* Pointer to buffer. */
    DWORD  dwBufferLength;  /* Length of buffer. */
    DWORD  dwBytesUsed;     /* Bytes actually used. */
    DWORD  dwTimeCaptured;  /* Timefrom start of stream. */
    DWORD  dwUser;
    DWORD  dwFlags;         /* Flags. */
    DWORD  dwReserved[4];
}
#define VHDR_DONE       0x00000001
#define VHDR_PREPARED   0x00000002
#define VHDR_INQUEUE    0x00000004
#define VHDR_KEYFRAME   0x00000008

Once the frame callback procedure is associated to a capture window, we are all set to begin tracking.

Color Space

Before we start processing frames, it is important to understand the different representations for color spaces used in digitized video. There are many color spaces to choose from, and each of them has its own strengths and limitations. Choosing the right color space for a specific application simplifies computation significantly.

The feature that we will be looking at for this demo application is brightness, and we will track objects based on their brightness. A very natural approach is to make sure that the color space that we are dealing with has a brightness component. YUV is one color space that has this very component that we are seeking for. However, YUV is not necessarily one of the input formats that is available from the web camera. Therefore, a conversion is required from the typical RGB24 input format to YUV.

The relationship between RGB and YUV can be expressed simply as the following set of linear equations.

[ Y ]   [  0.257  0.504  0.098  0.063 ][ R ]
[ U ] = [ -0.148 -0.291  0.439  0.500 ][ G ]
[ V ]   [  0.439 -0.368 -0.072  0.500 ][ B ]
[ 1 ]   [  0.000  0.000  0.000  1.000 ][ 1 ]

This matrix results from the concept of change of basis in linear algebra, where in this case, corresponds to the rotation of the color cube such that the new basis has a component with the unique property R = G = B.

Feature Tracking

Now that we have direct access to the brightness of each pixel, a simple algorithm can be used to track a bright object. The algorithm that will be introduced here is a fairly simple one, called the "rectangle algorithm". The rectangle algorithm keeps track of four points in each frame, the top most, left most, right most and bottom most points where the brightness exceeds a certain threshold value.

If you use the following code, make sure you set the input format of your web camera to RGB24.

LRESULT CChildView::FrameCallbackProc(HWND hWnd, LPVIDEOHDR lpVideoHdr)
{
    ...
    ...

    for (int i=0; i<nHeight; ++i) {
        for (int j=0; j<nWidth; ++j) {
            /* Get the appropriate index into the buffer. */
            index = 3*(i*nWidth+j);
            /* Compute the V component. */
            Y = floor(0.299*lpData[index+2] + 0.587*lpData[index+1] + 
                                          0.114*lpData[index] + 0.5);
            /* If brightness exceeds threshold value. */
            if (Y > bThreshold) {
                /* First occurence, initialize points. */
                if (init) {
                    if (pLeft.x > j) {
                        pLeft.x = j;
                        pLeft.y = i;
                    }
                    if (pRight.x < j) {
                        pRight.x = j;
                        pRight.y = i;
                    }
                    pBottom.x = j;
                    pBottom.y = i;
                }
                /* Always keep track of four corners. */
                else {
                    pTop.x = pBottom.x = pLeft.x = pRight.x = j;
                    pTop.y = pBottom.y = pLeft.y = pRight.y = i;
                    init = true;
                }
            }
        }
    }    
    
    ...
    ...

}

A rectangle can be constructed from these points, which tells us where the bright object is. The border of the rectangle is then simply replaced by a predefined color.

if (init) {
    /* Replace border pixels with predefined colour. */
    for (int i=pLeft.x; i<=pRight.x; ++i) {
        index = 3*((pTop.y)*nWidth + i);    /* Top */
        lpData[index]   = 0;   /* B */
        lpData[index+1] = 0;   /* G */
        lpData[index+2] = 255; /* R */
        index = 3*((pBottom.y)*nWidth + i); /* Bottom */
        lpData[index]   = 0;   /* B */
        lpData[index+1] = 0;   /* G */
        lpData[index+2] = 255; /* R */
    }
    for (int i=pTop.y; i<=pBottom.y; ++i) {
        index = 3*((i)*nWidth + pLeft.x);   /* Left */
        lpData[index]   = 0;   /* B */
        lpData[index+1] = 0;   /* G */
        lpData[index+2] = 255; /* R */
        index = 3*((i)*nWidth + pRight.x);  /* Right */
        lpData[index]   = 0;   /* B */
        lpData[index+1] = 0;   /* G */
        lpData[index+2] = 255; /* R */
    }
}

This algorithm obviously has a lot of weaknesses.

  • It only gives the position of the object as a whole on the screen.
  • It does not keep any information about the shape of the object.
  • It does not tell where the middle of the object is.
  • It can never track multiple objects.

An Improved Algorithm

Sample screenshot 2

This algorithm tracks objects by identifying segments that make up the object on the screen. Each segment consists of the head and the length of the segment. The object is constructed by grouping the segments together.

BYTE Y; int index;
/* -- Variables used by the new tracking algorithm. -- */
QSEG segment;
std::list<QSEG> object;

for (int i=0; i<nHeight; ++i) {
    segment.length = 0;

    for (int j=0; j<nWidth; ++j) {
        index = 3*(i*nWidth+j);
        Y = floor(0.299*lpData[index+2] + 0.587*lpData[index+1] +
          0.114*lpData[index] + 0.5);

        if (Y > bThreshold) {
            if (segment.length == 0) {
                segment.head.x = j;
                segment.head.y = i;
            }
            ++segment.length;
        }
    }

    if (segment.length) {
        object.push_back(segment);
    }
}

/* -- Draw the shape of the object with a predefined colour.  -- */
for (std::list<QSEG>::iterator i=object.begin(); i!=object.end(); ++i) {
    index = 3*((*i).head.y*nWidth + (*i).head.x);
    lpData[index]   = 255;
    lpData[index+1] = 0;
    lpData[index+2] = 255;

    index = 3*((*i).head.y*nWidth + (*i).head.x + (*i).length);
    lpData[index]   = 255;
    lpData[index+1] = 0;
    lpData[index+2] = 255;
}

This new tracking algorithm has a few extra advantages.

  • It can track multiple objects.
  • It can track the shape of the objects.
  • The number of pixels that make up the object on the screen can be easily calculated. With this piece of information and proper distance calibration, the position of the object in 3 dimensions can be determined.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

Paul Yi Tung, Ooi
Web Developer
Singapore Singapore
No Biography provided

Comments and Discussions

 
QuestionHow to run the code PinmemberVerghese689129-Mar-13 11:00 
How do you run the code.It has multiple Cpp files.How do I compile it.
QuestionImpressive program. Can you please mail me the VB.NET 2008 source code of the same program? Pinmember# include"sourav"2-Oct-10 12:35 
AnswerRe: Impressive program. Can you please mail me the VB.NET 2008 source code of the same program? Pinmvpthatraja28-Jan-12 6:23 
GeneralGood one Pinmemberuvik20-Sep-09 4:01 
GeneralAssertion Error Pinmembercrockbrol9-Dec-08 0:42 
GeneralSrc files are lacking the resource (res) directory Pinmemberalgorimancer15-May-08 6:27 
Questionplease flowchart or algorithm Pinmemberjoefree18-Apr-08 4:00 
GeneralTracking Brighest Of Certain Colour Pinmemberbeardedbernard22-Nov-07 0:31 
GeneralSign Language Recognitiom Pinmemberosama abu elnasr9-Oct-07 18:43 
Questionhow to get .dsw file?? [modified] Pinmemberabhijit bhopale8-Oct-07 18:18 
GeneralC# code PinmemberAbu Syed Khan24-Mar-07 7:34 
GeneralOnly thanks! Pinmembertumacaco2-Mar-07 1:55 
Generalcontrol movement PinmemberEftekhar Ali25-Feb-07 11:10 
QuestionHow to make it work ! PinmemberJust For You6-Feb-07 20:54 
AnswerRe: How to make it work ! PinmemberEftekhar Ali25-Feb-07 11:11 
GeneralRe: How to make it work ! Pinmemberjung-kreidler28-Feb-07 5:17 
GeneralReally Nice.. PinmemberAnant wakode26-Jul-06 1:45 
Generalfatal error RC1015: cannot open include file 'res\Tracker.rc2'. Pinmemberanishchowdhri27-Jan-06 15:26 
GeneralRe: fatal error RC1015: cannot open include file 'res\Tracker.rc2'. Pinmemberherve3d4-Feb-06 10:48 
GeneralI can only tracking Highlight Objects Pinsussdescartes1-Jul-05 5:25 
GeneralNeed the Tracker.sln file Pinmemberdominic_eales20-Jun-05 22:06 
GeneralRe: Need the Tracker.sln file Pinmembertumacaco2-Mar-07 1:57 
Generalcannot open project .vcproj PinmemberRay0000002-Jun-05 23:56 
GeneralTracking objects based on their color Pinmemberfritz.venter@gmail.com13-May-05 5:09 
GeneralRe: Tracking objects based on their color PinsussAnonymous1-Jun-05 23:55 
QuestionWhat is MFC70.dll????? Pinmembersanjeev j p6-May-05 9:13 
AnswerRe: What is MFC70.dll????? Pinmemberuus999-May-05 16:25 
Generalimproved algortihm PinsussAnonymous26-Apr-05 8:07 
Generalmissing resources PinmemberCurlique8-Mar-05 5:26 
Generalmouse click PinmemberLOUIS Christian26-Feb-05 11:39 
Generalrecording Pinmembermaxpower6315-Feb-05 10:40 
GeneralKernel based object tracking Pinmemberomertion3-Jan-05 12:53 
GeneralMPEG-2 parser Pinmemberhiravi19-Oct-04 12:45 
Generalinterested Pinmembersebsebs15-Oct-04 4:20 
Generalsuggestion Pinmembertom_dx8-Oct-04 14:48 
GeneralRe: suggestion PinmemberPaul Ooi12-Oct-04 0:55 
GeneralJoining of segments PinmemberNelsonKL8-Oct-04 6:22 
GeneralRe: Joining of segments PinmemberPaul Ooi8-Oct-04 7:42 
GeneralGreat article ! PinmemberMarcello7-Oct-04 13:58 
GeneralGood introduction PinmemberJaniOrca7-Oct-04 1:14 
GeneralThe Algorithm PinmemberPaul Ooi30-Sep-04 23:04 
GeneralRe: The Algorithm PinmemberRamsin8-Dec-05 9:52 
GeneralRe: The Algorithm PinmemberPaul Yi Tung, Ooi9-Dec-05 23:20 
GeneralMFC70 missing PinmemberGeorgi Petrov30-Sep-04 20:40 
GeneralRe: MFC70 missing PinmemberPaul Ooi30-Sep-04 23:20 
GeneralRe: MFC70 missing PinmemberKadji7-Oct-04 10:21 
GeneralRe: MFC70 missing PinmemberGeorgi Petrov13-Oct-04 3:41 
Questione-mail? PinmemberBrazilianDudi29-Sep-04 8:35 
Generalgood, but... Pinmembertradnomad27-Sep-04 3:47 
GeneralRe: good, but... PinmemberPaul Ooi27-Sep-04 5:01 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.150224.1 | Last Updated 6 May 2005
Article Copyright 2004 by Paul Yi Tung, Ooi
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid