![]() |
Languages »
C / C++ Language »
General
Advanced
License: The Code Project Open License (CPOL)
TrackEye : Real-Time Tracking Of Human Eyes Using a WebcamBy zafersavasReal-Time Tracking of Human Eyes in video sequences for Human-Computer Interaction using a webcam |
C++ (VC6), Windows, MFC
|
|
Advanced Search Add to IE Search |
|
|
||||||||||||||||||
Eyes are the most important features of the human face. So effective usage of eye movements as a communication technique in user-to-computer interfaces can find place in various application areas.
Eye tracking and the information provided by the eye features have the potential to become an interesting way of communicating with a computer in a human-computer interaction (HCI) system. So with this motivation, designing a real-time eye feature tracking software is the aim of this project.
The purpose of the project is to implement a real-time eye-feature tracker with the following capabilities:
Under TrackEye Menu --> Tracker Settings
ColorSpace type to use during PCA: CV_RGB2GRAY Check “Track eyes in details” and then check “Detect also eye pupils”. Click “Adjust Parameters” button:
Check “Indicate eye boundary using active snakes”. Click “Settings for snake” button:
ColorSpace to use: CV_RGB2GRAY So far there has been a lot of work on eye detection and before the project, the previous methods were carefully studied to determine the implemented method. We can classify studies related to eye into two main categories as listed below:
These type of studies use the necessary equipment which will give a signal of some sort which is proportional to the position of the eye in the orbit. Various methods that are current in use are Electrooculography, Infra-Red Oculography, Scleral search coils. These methods are completely out of our project.
Image based approaches perform eye detections on the images. Most of the image based methods try to detect the eyes using the features of the eyes. Methods used so far are knowledge-based methods, feature-based methods (color, gradient), simple template matching, appearance methods. Another interesting method is “Deformable template matching” which is based on matching a geometrical eye template on an eye image by minimizing the energy of the geometrical model.
The implemented project is on three components:
Two different methods were implemented in the project. They are:
Adaptive Mean Shift algorithm is used for tracking human faces and is based on robust non-parametric technique for climbing density gradients to find the mode (peak) of probability distributions called the mean shift algorithm. As faces are tracked in video sequences, mean shift algorithm is modified to deal with the problem of dynamically changing color probability distributions. The block diagram of the algorithm is given below:
The second face detection algorithm is based on a classifier working with Haar-Like features (namely a cascade of boosted classifiers working with Haar-like features). First of all it is trained with a few hundreds of sample views of a face. After a classifier is trained, it can be applied to a region of interest in an input image. The classifier outputs a "1" if the region is likely to show face and "0" otherwise. To search for the object in the whole image, one can move the search window across the image and check every location using the classifier. The classifier is designed so that it can be easily "resized" in order to be able to find the objects of interest at different sizes, which is more efficient than resizing the image itself.
Two different methods were implemented in the project:
EigenEye Method Template-Matching is a well-known method for object detection. In our template matching method, a standard eye pattern is created manually and given an input image, the correlation values with the standard patterns are computed for the eyes. The existence of an eye is determined based on the correlation values. This approach has the advantage of being simple to implement. However, it may sometimes be inadequate for eye detection since it cannot effectively deal with variation in scale, pose and shape.
Adaptive EigenEye Method is based on the well-known method EigenFaces. However as the method is used for eye detection we named it as “EigenEye Method”. The main idea is to decompose eye images into a small set of characteristics feature images called eigeneyes, which may be thought of as the principal components of the original images. These eigeneyes function as the orthogonal basis vectors of a subspace called eyespace. However we know that the eigenface method is not scale invariant. To provide the scale invariance we can resize the eye-database once with the information gathered by the face detection algorithm (EyeWidth / FaceWidth ? 0.35), we can provide scale-invariant detection using only one database.
OpenCV Library offers a lot of image processing and object tracking & detection libraries. The main function used in these projects and their usage are given below:
void CTrackEyeDlg::HaarFaceDetect( IplImage* img, CvBox2D* faceBox)
{
int scale = 2;
IplImage* temp = cvCreateImage( cvSize(img->width/2,img->height/2), 8, 3 );
CvPoint pt1, pt2;
int i;
cvPyrDown( img, temp, CV_GAUSSIAN_5x5 );
#ifdef WIN32
cvFlip( temp, temp, 0 );
#endif
cvClearMemStorage( storage );
if( hid_cascade )
{
CvSeq* faces = cvHaarDetectObjects( temp, hid_cascade, storage, 1.2, 2,
CV_HAAR_DO_CANNY_PRUNING );
NumOfHaarFaces = faces->total;
if (NumOfHaarFaces > 0)
{
CvRect* r = (CvRect*)cvGetSeqElem( faces, 0, 0 );
pt1.x = r->x*scale;
pt2.x = (r->x+r->width)*scale;
#ifdef WIN32
pt1.y = img->height - r->y*scale;
pt2.y = img->height - (r->y+r->height)*scale;
#else
pt1.y = r->y*scale;
pt2.y = (r->y+r->height)*scale;
#endif
faceBox->center.x = (float)(pt1.x+pt2.x)/2.0;
faceBox->center.y = (float)(pt1.y+pt2.y)/2;
faceBox->size.width = (float)(pt2.x - pt1.x);
faceBox->size.height = (float)(pt1.y - pt2.y);
}
}
cvShowImage( "Tracking", img );
cvReleaseImage( &temp );
}
// Inputs for CamShift algorithm
IplImage* HUE = cvCreateImage(cvGetSize(SampleForHUE), IPL_DEPTH_8U, 1);
extractHUE(SampleForHUE, HUE); // ** Extract HUE information
int hist_size = 20;
float ranges[] = { 0, 180 };
float* pranges[] = {ranges};
hist = cvCreateHist( 1, &hist_size, CV_HIST_ARRAY, pranges, 1 );
cvCalcHist(&HUE, hist); // Calculate histogram of HUE part
hueFrame = cvCreateImage(cvGetSize(CameraFrame), IPL_DEPTH_8U, 1);
backProject = cvCreateImage(cvGetSize(CameraFrame), IPL_DEPTH_8U, 1);
extractHUE(CameraFrame, hueFrame);
while (trackControl != 0)
{
extractHUE( CameraFrame, hueFrame );
cvCalcBackProject( &hueFrame, backProject, hist ); // Probability is formed
//cvShowImage("Tester2", backProject);
cvCamShift( backProject, searchWin, cvTermCriteria( CV_TERMCRIT_EPS |
CV_TERMCRIT_ITER, 15, 0.1 ), &comp, &faceBox );
searchWin = comp.rect;
}
// Template Matching for Eye detection
void Face::findEyes_TM(IplImage* faceImage, TrackingSettings* settings)
{
CvSize faceSize; faceSize = cvGetSize(faceImage);
// Load Template from the eye database
CString fileName;
// Name of the template for left eye
fileName.Format("%s\\eye%d.jpg", settings->params->DBdirectory, 0);
IplImage* eyeImage_Left = cvLoadImage(fileName, -1);
// Name of the template for left eye
fileName.Format("%s\\eye%d.jpg", settings->params->DBdirectory, 1);
IplImage* eyeImage_Right = cvLoadImage(fileName, -1);
IplImage* tempTemplateImg_Left; IplImage* tempTemplateImg_Right;
IplImage* templateImg_Left; IplImage* templateImg_Right;
if (eyeImage_Left == NULL || eyeImage_Right == NULL)
{
MessageBox(NULL, "Templates can not be loaded.\n
Please check your eye database folder", "Error", MB_OK||MB_ICONSTOP);
exit(1);
}
else
{
// Change color space according to the settings entered by the user
tempTemplateImg_Left = cvCreateImage(cvGetSize(eyeImage_Left), IPL_DEPTH_8U, 1);
changeColorSpace(settings, eyeImage_Left, tempTemplateImg_Left);
tempTemplateImg_Right =
cvCreateImage(cvGetSize(eyeImage_Right), IPL_DEPTH_8U, 1);
changeColorSpace(settings, eyeImage_Right, tempTemplateImg_Right);
float idealWidth = faceSize.width * settings->params->ratio;
float conversionRatio = idealWidth/(float)tempTemplateImg_Left->width;
CvSize newSize;
newSize.width = (int)idealWidth;
newSize.height = (int)(tempTemplateImg_Left->height*conversionRatio);
templateImg_Left = cvCreateImage(newSize, IPL_DEPTH_8U, 1);
cvResize(tempTemplateImg_Left, templateImg_Left, CV_INTER_LINEAR); // was NN
cvReleaseImage(&eyeImage_Left);
cvReleaseImage(&tempTemplateImg_Left);
templateImg_Right = cvCreateImage(newSize, IPL_DEPTH_8U, 1);
cvResize(tempTemplateImg_Right, templateImg_Right, CV_INTER_LINEAR); // was NN
cvReleaseImage(&eyeImage_Right);
cvReleaseImage(&tempTemplateImg_Right);
}
// *************************************************************
// ************Search faceImage for eyes************************
// *************************************************************
IplImage* GRAYfaceImage = cvCreateImage(faceSize, IPL_DEPTH_8U, 1);
changeColorSpace(settings, faceImage, GRAYfaceImage);
//cvCvtColor( faceImage, GRAYfaceImage, CV_RGB2GRAY);
//GRAYfaceImage->origin = 1;
// ** Warning at this point image origin is bottom-left corner.
// ** Eye1 search area
int x_left = 0;
int y_left = 0;
int width_left = (int)((float)(faceSize.width/2.0));
int height_left = (int)((float)(faceSize.height));
CvRect rect_Eye1 = cvRect(x_left, y_left, width_left, height_left);
CvMat* Eye1Image = cvCreateMat(width_left, height_left, CV_8UC1);
cvGetSubRect(GRAYfaceImage, Eye1Image, rect_Eye1 );
cvFlip( Eye1Image, Eye1Image, 0);
// ** Eye2 search area
int x_right= (int)((float)(faceSize.width/2.0));
int y_right = 0;
int width_right = (int)((float)(faceSize.width/2.0));
int height_right = (int)((float)(faceSize.height));
CvRect rect_Eye2 = cvRect(x_right, y_right, width_right, height_right);
CvMat* Eye2Image = cvCreateMat(width_right, height_right, CV_8UC1);
cvGetSubRect(GRAYfaceImage, Eye2Image, rect_Eye2 );
cvFlip( Eye2Image, Eye2Image, 0);
// OpenCV says that size of the result must be the following:
CvSize size;
size.height= Eye1Image->height - templateImg_Left->height + 1;
size.width = Eye1Image->width - templateImg_Left->width + 1;
IplImage* result1 = cvCreateImage( size,IPL_DEPTH_32F,1);
IplImage* result2 = cvCreateImage( size,IPL_DEPTH_32F,1);
// Left Eye
cvMatchTemplate( Eye1Image, templateImg_Left, result1, settings->params->tempMatch);
// Right Eye
cvMatchTemplate( Eye2Image, templateImg_Right, result2, settings->params->tempMatch);
// find the best match location - LEFT EYE
double minValue1, maxValue1;
CvPoint minLoc1, maxLoc1;
cvMinMaxLoc( result1, &minValue1, &maxValue1, &minLoc1, &maxLoc1 );
cvCircle( result1, maxLoc1, 5, settings->programColors.colors[2], 1 );
// transform point back to original image
maxLoc1.x += templateImg_Left->width / 2;
maxLoc1.y += templateImg_Left->height / 2;
settings->params->eye1.coords.x = maxLoc1.x;
settings->params->eye1.coords.y = maxLoc1.y;
settings->params->eye1.RectSize.width = templateImg_Left->width;
settings->params->eye1.RectSize.height = templateImg_Left->height;
settings->params->eye1.eyefound = true;
// find the best match location - RIGHT EYE
double minValue2, maxValue2;
CvPoint minLoc2, maxLoc2;
cvMinMaxLoc( result2, &minValue2, &maxValue2, &minLoc2, &maxLoc2 );
cvCircle( result2, maxLoc2, 5, settings->programColors.colors[2], 1 );
// transform point back to original image
maxLoc2.x += templateImg_Left->width / 2;
maxLoc2.y += templateImg_Left->height / 2;
settings->params->eye2.coords.x = maxLoc2.x+(int)faceSize.width/2;
settings->params->eye2.coords.y = maxLoc2.y;
settings->params->eye2.RectSize.width = templateImg_Left->width;
settings->params->eye2.RectSize.height = templateImg_Left->height;
settings->params->eye2.eyefound = true;
cvCircle( Eye1Image, maxLoc1, 5, settings->programColors.colors[2], 1 );
cvCircle( Eye2Image, maxLoc2, 5, settings->programColors.colors[2], 1 );
}
void Face::findEyes(IplImage* faceImage, TrackingSettings* settings)
{
IplImage** images = (IplImage**)malloc(sizeof(IplImage*)*numOfImages);
IplImage** eigens = (IplImage**)malloc(sizeof(IplImage*)*numOfImages);
IplImage* averageImage;
IplImage* projection;
CvSize faceSize; faceSize = cvGetSize(faceImage);
eigenSize newEigenSize;
newEigenSize.width = faceSize.width * settings->params->ratio;
newEigenSize.conversion = ((float)newEigenSize.width) / ((float)database[0]->width);
newEigenSize.height = ((float)database[0]->height) * newEigenSize.conversion;
CvSize newSize;
newSize.width = (int)newEigenSize.width;
newSize.height = (int)newEigenSize.height;
IplImage* tempImg = cvCreateImage( newSize, IPL_DEPTH_8U, 1);
// **********Initializations**********
for (int i=0; i params->nImages; i++)
{
images[i] = cvCreateImage(newSize, IPL_DEPTH_8U, 1);
cvResize(database[i], images[i], CV_INTER_LINEAR); // was NN
}
cvShowImage("Eigen", images[0]);
cvReleaseImage(&tempImg);
// Create space for EigenFaces
for (i=0; i params->nImages; i++)
eigens[i] = cvCreateImage(cvGetSize(images[0]), IPL_DEPTH_32F, 1);
averageImage = cvCreateImage(cvGetSize(images[0]), IPL_DEPTH_32F, 1);
projection = cvCreateImage(cvGetSize(images[0]), IPL_DEPTH_8U, 1);
// *************************************************************
// ************Calculate EigenVectors & EigenValues*************
// *************************************************************
CvTermCriteria criteria;
criteria.type = CV_TERMCRIT_ITER|CV_TERMCRIT_EPS;
criteria.maxIter = 13;
criteria.epsilon = 0.1;
// ** n was present instead of numOfImages
cvCalcEigenObjects( settings->params->nImages, images, eigens,
0, 0, 0, &criteria, averageImage, vals );
// *************************************************************
// ************Search faceImage for eyes************************
// *************************************************************
IplImage* GRAYfaceImage = cvCreateImage(faceSize, IPL_DEPTH_8U, 1);
changeColorSpace(settings,faceImage, GRAYfaceImage);
//cvCvtColor( faceImage, GRAYfaceImage, CV_RGB2GRAY);
// ** Warning at this point image origin is bottom-left corner.
GRAYfaceImage->origin = 1;
int MARGIN = settings->params->MaxError;
double minimum = MARGIN; double distance = MARGIN;
// ** Eye1 search Space
settings->params->eye1.xlimitLeft = 0;
settings->params->eye1.xlimitRight = faceSize.width/2.0 - images[0]->width - 1;
settings->params->eye1.ylimitUp =
(int)( ((float)faceSize.height)*0.75 - images[0]->height - 1);
settings->params->eye1.ylimitDown = faceSize.height/2;
// ** Eye2 search Space
settings->params->eye2.xlimitLeft = faceSize.width/2.0;
settings->params->eye2.xlimitRight = faceSize.width - images[0]->width - 1;
settings->params->eye2.ylimitUp =
(int)( ((float)faceSize.height)*0.75 - images[0]->height - 1);
settings->params->eye2.ylimitDown = faceSize.height/2;
settings->params->eye1.initializeEyeParameters();
settings->params->eye2.initializeEyeParameters();
settings->params->eye1.RectSize.width = images[0]->width;
settings->params->eye1.RectSize.height = images[0]->height;
settings->params->eye2.RectSize.width = images[0]->width;
settings->params->eye2.RectSize.height = images[0]->height;
IplImage* Image2Comp = cvCreateImage(cvGetSize(images[0]), IPL_DEPTH_8U, 1);
int x,y;
// ** Search left eye i.e eye1
for (y=settings->params->eye1.ylimitDown; y params->eye1.ylimitUp; y+=2)
{
for (x=settings->params->eye1.xlimitLeft; x params->eye1.xlimitRight; x+=2)
{
cvSetImageROI(GRAYfaceImage, cvRect
(x, y, images[0]->width, images[0]->height));
if (settings->params->varianceCheck == 1 )
{
if (calculateSTD(GRAYfaceImage) <= (double)(settings->params->variance))
{
cvResetImageROI(GRAYfaceImage);
continue;
}
}
cvFlip( GRAYfaceImage, Image2Comp, 0);
cvResetImageROI(GRAYfaceImage);
// Decide whether it is an eye or not
cvEigenDecomposite( Image2Comp, settings->params->nEigens,
eigens, 0, 0, averageImage, weights );
cvEigenProjection( eigens, settings->params->nEigens,
CV_EIGOBJ_NO_CALLBACK, 0, weights, averageImage, projection );
distance = cvNorm(Image2Comp, projection, CV_L2, 0);
if (distance < minimum && distance > 0)
{
settings->params->eye1.eyefound = true;
minimum = distance;
settings->params->eye1.distance = distance;
settings->params->eye1.coords.x = x;
settings->params->eye1.coords.y = y;
}
}
}
minimum = MARGIN; distance = MARGIN;
// ** Search right eye i.e eye2
for (y=settings->params->eye2.ylimitDown; y params->eye2.ylimitUp; y+=2)
{
for (x=settings->params->eye2.xlimitLeft; x params->eye2.xlimitRight; x+=2)
{
cvSetImageROI(GRAYfaceImage,
cvRect(x, y, images[0]->width, images[0]->height));
if (settings->params->varianceCheck == 1)
{
if (calculateSTD(GRAYfaceImage) <= (double)(settings->params->variance))
{
cvResetImageROI(GRAYfaceImage);
continue;
}
}
cvFlip( GRAYfaceImage, Image2Comp, 0);
cvResetImageROI(GRAYfaceImage);
// ** Decide whether it is an eye or not
cvEigenDecomposite( Image2Comp, settings->params->nEigens,
eigens, 0, 0, averageImage, weights );
cvEigenProjection( eigens, settings->params->nEigens,
0, 0, weights, averageImage, projection );
distance = cvNorm(Image2Comp, projection, CV_L2, 0);
if (distance < minimum && distance > 0)
{
settings->params->eye2.eyefound = true;
minimum = distance;
settings->params->eye2.distance = distance;
settings->params->eye2.coords.x = x;
settings->params->eye2.coords.y = y;
}
}
}
cvReleaseImage(&Image2Comp);
// ** Cleanup
cvReleaseImage(&GRAYfaceImage);
for (i=0; i params->nImages; i++)
cvReleaseImage(&images[i]);
for (i=0; i params->nImages; i++)
cvReleaseImage(&eigens[i]);
cvReleaseImage(&averageImage);
cvReleaseImage(&projection);
free(images);
free(eigens);
}
TrackEyeTrackEye TrackEye v2.0 now supports: * Please note that TrackEye was written with OpenCV Library v3.1, so make sure to use it during rebuild.
General
News
Question
Answer
Joke
Rant
Admin
|
PermaLink |
Privacy |
Terms of Use
Last Updated: 12 Jun 2008 Editor: Deeksha Shenoy |
Copyright 2008 by zafersavas Everything else Copyright © CodeProject, 1999-2009 Web09 | Advertise on the Code Project |