Click here to Skip to main content
14,639,480 members
Articles » Multimedia » Audio and Video » Video
Posted 11 Aug 2011


741 bookmarked

Multiple face detection and recognition in real time

Rate this:
4.89 (323 votes)
Please Sign up or sign in to vote.
4.89 (323 votes)
27 Mar 2015CPOL
Face detection and recognition with support of multiples faces in the same scene and others interesting features using C# and EmguCV



The facial recognition has been a problem worked on around the world for many persons; this problem has emerged in multiple fields and sciences, especially in computer science, others fields that are very interested In this technology are: Mechatronic, Robotic, criminalistics, etc. In this article I work in this interesting topic using EmguCV cross platform .Net wrapper to the Intel OpenCV image processing library and C# .Net, these library’s allow me capture and process image of a capture device in real time. The main goal of this article is show and explains the easiest way how implement a face detector and recognizer in real time for multiple persons using Principal Component Analysis (PCA) with eigenface for implement it in multiple fields.


Facial recognition is a computer application composes for complex algorithms that use mathematical and matricial techniques, these get the image in raster mode(digital format) and then process and compare pixel by pixel using different methods for obtain a faster and reliable results, obviously these results depend of the machine use to process this due to the huge computational power that these algorithms, functions and routines requires, these are the most popular techniques used for solve this modern problem:



Some facial recognition algorithms identify faces by extracting landmarks, or features, from an image of the subject's face. For example, an algorithm may analyze the relative position, size, and/or shape of the eyes, nose, cheekbones, and jaw. These features are then used to search for other images with matching features. Other algorithms normalize a gallery of face images and then compress the face data, only saving the data in the image that is useful for face detection. A probe image is then compared with the face data. One of the earliest successful systems is based on template matching techniques applied to a set of salient facial features, providing a sort of compressed face representation. Recognition algorithms can be divided into two main approaches, geometric, which looks at distinguishing features, or photometric, which is a statistical approach that distill an image into values and comparing the values with templates to eliminate variances. Popular recognition algorithms include Principal Component Analysis with eigenface, Linear Discriminate Analysis, Elastic Bunch Graph Matching fisherface, the Hidden Markov model, and the neuronal motivated dynamic link matching.

text taken from [1]

An example of EigenFaces:

Image 2

image taken from [4]


A newly emerging trend, claimed to achieve previously unseen accuracies, is three-dimensional face recognition. This technique uses 3-D sensors to capture information about the shape of a face. This information is then used to identify distinctive features on the surface of a face, such as the contour of the eye sockets, nose, and chin. One advantage of 3-D facial recognition is that it is not affected by changes in lighting like other techniques. It can also identify a face from a range of viewing angles, including a profile view. Even a perfect 3D matching technique could be sensitive to expressions. For that goal a group at the Technion applied tools from metric geometry to treat expressions as isometries.

Image 3

image taken from [2]

Skin texture analysis

Another emerging trend uses the visual details of the skin, as captured in standard digital or scanned images. This technique, called skin texture analysis, turns the unique lines, patterns, and spots apparent in a person’s skin into a mathematical space Tests have shown that with the addition of skin texture analysis, performance in recognizing faces can increase 20 to 25 percent. It is typically used in security systems and can be compared to other biometrics such as fingerprint or eye iris recognition systems.

text taken from [1]


Image 4

Emgu CV is a cross platform .Net wrapper to the Intel OpenCV image processing library. Allowing OpenCV functions to be called from .NET compatible languages such as C#, VB, VC++, IronPython etc. The wrapper can be compiled in Mono and run on Linux / Mac OS X.

text taken from [3]

In my own words EmguCV is an awesome Wrapper, this let make very interesting things and tasks of computer vision, this library set let do an unlimited amount of wonderful projects in this field, EmguCV have many functions that let us work with CPU and GPU increases the performance dramatically with the latest mentioned.

This Awesome SW project let work and do:

  • Optical Character Recognition(OCR)
  • Face Detection
  • Pedestrian Detection
  • Kinect projects
  • 3D reconstruction
  • SURF feature detector ...between many others interesting tasks.

EmguCV basics: How i start to work?

If you don't had work never with this wrapper, you want see how add refereces to project or solve troubles look this god article/tutorial by C_Johnson:

Creating Your First EMGU Image Processing Project

Other interesting Web/blog whit multiple tutorials to star with emguCV, image processing, and face recognition by mehwish87 is :

EmguCV and basic image processing tutorials

Using the code

First declare all variables an important objects to use:

//Declararation of all variables, vectors and haarcascades
        Image<bgr,> currentFrame;
        Capture grabber;
        HaarCascade face;
        HaarCascade eye;
        MCvFont font = new MCvFont(FONT.CV_FONT_HERSHEY_TRIPLEX, 0.5d, 0.5d);
        Image<gray,> result, TrainedFace = null;
        Image<gray,> gray = null;
        List<image<gray,>> trainingImages = new List<image<gray,>>();
        List<string> labels= new List<string>();
        List<string> NamePersons = new List<string>();
        int ContTrain, NumLabels, t;
        string name, names = null;

Then load the haarcascades for face detection, then I do a little “procedure” to load of previous trained faces and labels for each image stored previously:

//Load haarcascades for face detection
face = new HaarCascade("haarcascade_frontalface_alt_tree.xml");
eye = new HaarCascade("haarcascade_eye.xml");
    //Load of previus trainned faces and labels for each image
    string Labelsinfo = File.ReadAllText(Application.StartupPath + "/TrainedFaces/TrainedLabels.txt");
    string[] Labels = Labelsinfo.Split('%');
    NumLabels = Convert.ToInt16(Labels[0]);
    ContTrain = NumLabels;
    string LoadFaces;

    for (int tf = 1; tf < NumLabels+1; tf++)
        LoadFaces = "face" + tf + ".bmp";
        trainingImages.Add(new Image<gray,>(Application.StartupPath + "/TrainedFaces/" + LoadFaces));
catch(Exception e)
    MessageBox.Show("Nothing in binary database, please add at least a face", "Triained faces load", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);

Initialize the capture device, and FrameGrabber event that performs the detection and procces of images for each frame captured:

grabber = new Capture();
//Initialize the FrameGraber event
Application.Idle += new EventHandler(FrameGrabber);
button1.Enabled = false;

Passing to FrameGrabber event (main part of prototype) we use the most important methods and objects: DetectHaarCascade And EigenObjectRecognizer and perform operations For each face detected in one frame:

MCvAvgComp[][] facesDetected = gray.DetectHaarCascade(
                  new Size(20, 20));

                    //Action for each element detected
                    foreach (MCvAvgComp f in facesDetected[0])
                        t = t + 1;
                        result = currentFrame.Copy(f.rect).Convert<gray,>().Resize(100, 100, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC);
                        //draw the face detected in the 0th (gray) channel with blue color
                        currentFrame.Draw(f.rect, new Bgr(Color.Red), 2);

                        if (trainingImages.ToArray().Length != 0)
                            //TermCriteria for face recognition with numbers of trained images like maxIteration
                        MCvTermCriteria termCrit = new MCvTermCriteria(ContTrain, 0.001);

                        //Eigen face recognizer
                        EigenObjectRecognizer recognizer = new EigenObjectRecognizer(
                           ref termCrit);

                        name = recognizer.Recognize(result);

                            //Draw the label for each face detected and recognized
                        currentFrame.Draw(name, ref font, new Point(f.rect.X - 2, f.rect.Y - 2), new Bgr(Color.LightGreen));



haarObj: Haar classifier cascade in internal representation scaleFactor: The factor by which the search window is scaled between the subsequent scans, for example, 1.1 means increasing window by 10%

minNeighbors: Minimum number (minus 1) of neighbor rectangles that makes up an object. All the groups of a smaller number of rectangles than min_neighbors-1 are rejected. If min_neighbors is 0, the function does not any grouping at all and returns all the detected candidate rectangles, which may be useful if the user wants to apply a customized grouping procedure

flag: Mode of operation. Currently the only flag that may be specified is CV_HAAR_DO_CANNY_PRUNING. If it is set, the function uses Canny edge detector to reject some image regions that contain too few or too much edges and thus cannot contain the searched object. The particular threshold values are tuned for face detection and in this case the pruning speeds up the processing.

minSize: Minimum window size. By default, it is set to the size of samples the classifier has been trained on (~20x20 for face detection)

How to Train the Prototype?

I do this part the most easy possible, the prototype detect faces constantly (Each frame) and you can add this detected face in the image database with one label respectably, the face trained image will show in the imageBoxFrameGrabber and the process will be finished!!

Keep in mind: The face recognition algorithms based in PCA (Principal Component Analysis) do multiple comparisons and matches between a face detected and the trained images stored in binary database for this reason And for improve the accurate of recognition you should add several images of the same person in different angles, positions and luminance conditions, this training do this prototype solid and very accurate.



Code of training button (This perform the adding of training faces and labels for each):

               //Trained face counter
               ContTrain = ContTrain + 1;

               //Get a gray frame from capture device
               gray = grabber.QueryGrayFrame().Resize(320, 240, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC);

               //Face Detector
               MCvAvgComp[][] facesDetected = gray.DetectHaarCascade(
               new Size(20, 20));

               //Action for each element detected
               foreach (MCvAvgComp f in facesDetected[0])
                   TrainedFace = currentFrame.Copy(f.rect).Convert<gray,>();

               //resize face detected image for force to compare the same size with the
               //test image with cubic interpolation type method
               TrainedFace = result.Resize(100, 100, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC);

               //Show face added in gray scale
               imageBox1.Image = TrainedFace;

               //Write the number of triained faces in a file text for further load
               File.WriteAllText(Application.StartupPath + "/TrainedFaces/TrainedLabels.txt", trainingImages.ToArray().Length.ToString() + "%");

               //Write the labels of triained faces in a file text for further load
               for (int i = 1; i < trainingImages.ToArray().Length + 1; i++)
                   trainingImages.ToArray()[i - 1].Save(Application.StartupPath + "/TrainedFaces/face" + i + ".bmp");
                   File.AppendAllText(Application.StartupPath + "/TrainedFaces/TrainedLabels.txt", labels.ToArray()[i - 1] + "%");

               MessageBox.Show(textBox1.Text + "´s face detected and added :)", "Training OK", MessageBoxButtons.OK, MessageBoxIcon.Information);
               MessageBox.Show("Enable the face detection first", "Training Fail", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);

How to improve the recognition?

The default parameters (scale_factor=1.1, min_neighbors=3, flags=0) are tuned for accurate yet slow object detection.

Also you may modify the size for a big value, Modify this in the code:

MCvAvgComp[][] facesDetected = gray.DetectHaarCascade(
new Size(20, 20));

Aditionally modify the 3rd param to 2500 or 3000 instead 5000, this modification do that the EigenObjectRecognizer be more strict/acurate.

//Eigen face recognizer
                       EigenObjectRecognizer recognizer = new EigenObjectRecognizer(
                          ref termCrit);

How to improve the performance for slower CPUs?

All image processing algorithms demand many computational power, in this case the internals process carried on for the CPU with this sw prototype are so hard for slower o monocore CPUS, the Easy way for improve the performance of this Demo is modify the parameters that use the DetectHaarCascade method, these allow decrement the number of iteration, critic sections and Comparisons of the real time image captured for the Webcam improving notoriously the application performance.

Keep in mind: reduce the values of these parameters will affect the efficiency of recognition Algorithms.

First option:

For a faster operation on real video images the settings are: scale_factor=1.2, min_neighbors=2, flags=CV_HAAR_DO_CANNY_PRUNING, min_size=<minimum> (for example, ~1/4 to 1/16 of the image area in case of video conferencing).

Also you may modify the Minsize parameter for a big value.

// DetectHaarCascade Config for optimal performance

MCvAvgComp[][] facesDetected = gray.DetectHaarCascade(
                  new Size(20, 20));

Second option:

Get a “thumbnail”or resize the original image capture for reduce the time of processing In the FrameGrabber method modify the size values for a minor size (originally is 320x240)


//Get the current frame form capture device
                currentFrame = grabber.QueryFrame().Resize(260, 200, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC);

Remember do the same in the Training button:

gray = grabber.QueryGrayFrame().Resize(260, 200, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC);

The prototype does not show the strangers as unknown, Why?

Is a limitation of Eigenface approach, isn't error, use LBPH algorithm instead(have pros and shortcomings) available on EmguCV 2.4.2 and above.

What do you need to run/use this project without errors?

For avoid errors like this:

Image 6

1. First download and decompress it have “external” OS libraries(DLLs) used for some OpenCV functions, after copy these DLLs in C:/Windows/System32 or in bin folder of this Project

2. Then download EmguCV(Wrapper of OpenCV for C#) here:, install it and after go to C:\Emgu\emgucv-windows-x86\bin folder and copy: opencv_calib3d220, opencv_contrib220, opencv_core220, opencv_features2d220, opencv_ffmpeg220, opencv_flann220, opencv_gpu220, opencv_highgui220, opencv_imgproc220, opencv_legacy220, opencv_ml220, opencv_objdetect220 and opencv_video220 in C:/Windows/System32 or in bin folder of this Project

remember this DLLs are OpenCV Libraries and are necessary to run anything project that use EgmuCV

this is the easiest way for run the demo or project, found and Copy These files to folder demo or bin folder(if you want "run" the source code):

opencv_calib3d220.dll, opencv_contrib220.dll, opencv_core220.dll, opencv_features2d220.dll, opencv_ffmpeg220.dll, opencv_flann220.dll, opencv_gpu220.dll, opencv_highgui220.dll, opencv_imgproc220.dll, opencv_legacy220.dll, opencv_ml220.dll, opencv_objdetect220.dll and opencv_video220.dll(Dlls from OpenCV Content in EmguCV Download) and: Emgu.CV.dll Emgu.CV.UI.dll Emgu.Util.dll cv110.dll cvaux110.dll cvextern.dll cxcore110.dll highgui110.dll(Content into Demo Download zip)

Keep in mind: you can copy these files in Windows/System32/ folder and forget your problems of dependencies for this and other project that use Emgu or openCV(Emgu.CV.dll, Emgu.CV.UI.dll, Emgu.Util.dll SHOULD go always in the bin or .exe folder) or Download all ready project(Optimized version) and files here:

Points of Interest

I had many problems with vectors use with EmguCV, for this reason i had learn how use list of components like vectors and it really works for my project.

I learn many of image processing, PCA and EigenFaces and to optimize the code, due to the huge demand of resources for part of artificial vision algorithms.

This project idea emerge after see an Iron man scene…XD

What can you do whit this article and OpenCV?

Something like this....Big Grin | <img src=

Books about it

The official reference book for OpenCV is: "Learning OpenCV: Computer Vision with the OpenCV Library", from O'Reilly (2008)

There are hundreds of other good books for computer vision that are not specific to OpenCV:

  • "Computer Vision: A Modern Approach" by Forsyth and Ponce (2002).
  • "Computer Vision: Algorithms and Applications" by Szeliski (2011).
  • "Digital Image Processing" by Gonzalez and Woods (2001).
  • "The Essential Guide to Image Processing", by Bovik (2009).
  • "Computer Vision and Applications: A Guide for Students and Practitioners", by Jähne and Haußecker (2000).
  • HIPR2 Image Processing Worksheets (simple explanations of many computer vision topics)



[2] "Firms point to biometric future" By Dominic Bailey, BBC News, at Biometrics 2006




This article and prototype is only for academic purposes, not use for commercial or public releases without author permissions.


11-07-2011: Initial release

28-01-2012: Final release(Optimized version)


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

Sergio Andrés Gutiérrez Rojas
Systems Engineer Motion-Soft CO
Colombia Colombia
Systems engineer
Software developer
Networks technologist
Artificial vision, AI and robotic lover
Technology enthusiast

Motion Soft Co-funder

MIA PC Friend Developer:

Comments and Discussions

QuestionRegarding the database Pin
Member 1493836814-Sep-20 18:58
MemberMember 1493836814-Sep-20 18:58 
QuestionAnimals face recognition Pin
Member 1487684123-Jul-20 6:22
MemberMember 1487684123-Jul-20 6:22 
QuestionHow to define eigenobjectrecognizer object Pin
Reza kavian13-Apr-20 21:33
MemberReza kavian13-Apr-20 21:33 
Questionface matching Pin
Member 1462817115-Dec-19 21:38
MemberMember 1462817115-Dec-19 21:38 
QuestionRequire help Pin
sunilgoyani15-Jun-19 2:33
Membersunilgoyani15-Jun-19 2:33 
QuestionFacial Recognition Ethics Pin
BotReject27-May-19 22:24
MemberBotReject27-May-19 22:24 
QuestionGenial articulo. Pin
gfazzola14-Feb-19 6:21
Membergfazzola14-Feb-19 6:21 
QuestionIt doesn't remember the faces after restart the program Pin
Zaher Ha23-Oct-18 8:36
MemberZaher Ha23-Oct-18 8:36 
PraiseGreat Pin
GioviQ27-Jul-18 22:25
MemberGioviQ27-Jul-18 22:25 
QuestionProject is not working, showing only black screen Pin
shubham sharma14-Jun-18 21:22
Membershubham sharma14-Jun-18 21:22 
AnswerRe: Project is not working, showing only black screen Pin
Member 1135798824-Dec-18 5:56
MemberMember 1135798824-Dec-18 5:56 
AnswerRe: Project is not working, showing only black screen Pin
Member 1491463720-Aug-20 4:35
MemberMember 1491463720-Aug-20 4:35 
QuestionSergio please help me. I need your help. I am in a very difficult situation Pin
Member 1379977126-Apr-18 3:48
MemberMember 1379977126-Apr-18 3:48 
QuestionUnable to create capture from camera0 Pin
Member 1378667918-Apr-18 10:15
MemberMember 1378667918-Apr-18 10:15 
Questionface database and im getting errors Pin
Member 1373083116-Mar-18 10:42
MemberMember 1373083116-Mar-18 10:42 
GeneralHow to run this app on Android device Pin
Member 1307445918-Dec-17 3:53
MemberMember 1307445918-Dec-17 3:53 
Questionhow can i Create individual folder in saving faces in this projec? Pin
Member 133863948-Nov-17 14:37
MemberMember 133863948-Nov-17 14:37 
Questioniam getting error Pin
Syed Raza Sherazi10-Sep-17 5:21
MemberSyed Raza Sherazi10-Sep-17 5:21 
QuestionCan this project support all webcams or it only works on a particular webcam if yes then mention the model of it....... Pin
Member 1335491620-Aug-17 20:07
MemberMember 1335491620-Aug-17 20:07 
QuestionConvert to latest EmguCV Pin
djhabana18-Apr-17 22:18
Memberdjhabana18-Apr-17 22:18 
Questionwow Pin
Member 100508097-Mar-17 16:19
MemberMember 100508097-Mar-17 16:19 
QuestionBlack Screen Pin
Member 1296391621-Feb-17 7:00
MemberMember 1296391621-Feb-17 7:00 
QuestionWhat does 'Enable the face detection first' mean? Pin
ganesh199612-Feb-17 6:02
Memberganesh199612-Feb-17 6:02 
AnswerRe: What does 'Enable the face detection first' mean? Pin
Member 1287543724-Feb-17 11:00
MemberMember 1287543724-Feb-17 11:00 
AnswerRe: What does 'Enable the face detection first' mean? Pin
Zombies with Coffee, LLC14-Mar-17 3:43
professionalZombies with Coffee, LLC14-Mar-17 3:43 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.