Click here to Skip to main content
12,558,319 members (63,305 online)
Click here to Skip to main content
Add your own
alternative version

Tagged as


122 bookmarked

Human Emotion Detection from Image

, 20 Sep 2010 CPOL
Rate this:
Please Sign up or sign in to vote.
It can detect human emotion from image


This code can detect human emotion from image. First, it takes an image, then by skin color segmentation, it detects human skin color, then it detect human face. Then it separates the eyes & lip from the face. Then it draws bezier curve for eyes & lips. Then it compares the bezier curve of eyes and lips to the bezier curves of eyes & lips that are stored in the data base. Then it finds the nearest bezier curse from the data base & gives that data base stored bezier curve emotion as this image emotion.

How Does It Work?

Skin Color Segmentation

For skin color segmentation, first we contrast the image. Then we perform skin color segmentation.


Then, we have to find the largest connected region. Then we have to check the probability to become a face of the largest connected region. If the largest connected region has the probability to become a face, then it will open a new form with the largest connected region. If the largest connected regions height & width is larger or equal than 50 and the ratio of height/width is between 1 to 2, then it may be face.


Face Detection

For face detection, first we convert binary image from RGB image. For converting binary image, we calculate the average value of RGB for each pixel and if the average value is below than 110, we replace it by black pixel and otherwise we replace it by white pixel. By this method, we get a binary image from RGB image.


Then, we try to find the forehead from the binary image. We start scan from the middle of the image, then want to find a continuous white pixels after a continuous black pixel. Then we want to find the maximum width of the white pixel by searching vertical both left and right site. Then, if the new width is smaller half of the previous maximum width, then we break the scan because if we reach the eyebrow then this situation will arise. Then we cut the face from the starting position of the forehead and its high will be 1.5 multiply of its width.

pic-11.PNG pic-12.PNG

In the figure, X will be equal to the maximum width of the forehead. Then we will have an image which will contain only eyes, nose and lip. Then we will cut the RGB image according to the binary image.

Eyes Detection

For eyes detection, we convert the RGB face to the binary face. Now, we consider the face width by W. We scan from the W/4 to (W-W/4) to find the middle position of the two eyes. The highest white continuous pixel along the height between the ranges is the middle position of the two eyes.


Then we find the starting high or upper position of the two eyebrows by searching vertical. For left eye, we search w/8 to mid and for right eye we search mid to w – w/8. Here w is the width of the image and mid is the middle position of the two eyes. There may be some white pixels between the eyebrow and the eye. To make the eyebrow and eye connected, we place some continuous black pixels vertically from eyebrow to the eye. For left eye, the vertical black pixel-lines are placed in between mid/2 to mid/4 and for right eye the lines are in between mid+(w-mid)/ 4 to mid+3*(w-mid)/ 4 and height of the black pixel-lines are from the eyebrow starting height to (h- eyebrow starting position)/4. Here w is the width of the image and mid is the middle position of the two eyes and h is the height of the image. Then we find the lower position of the two eyes by searching black pixel vertically. For left eye, we search from the mid/4 to mid - mid/4 width. And for right eye, we search mid + (w-mid)/ 4 to mid+3*(w- mid)/ 4 width from image lower end to starting position of the eyebrow. Then we find the right side of the left eye by searching black pixel horizontally from the mid position to the starting position of black pixels in between the upper position and lower position of the left eye. And left side for right eye we search mid to the starting position of black pixels in between the upper position and lower position of right eye. The left side of the left eye is the starting width of the image and the right side of the right eye is the ending width of the image. Then we cut the upper position, lower position, left side and the right side of the two eyes from the RGB image.


Lip Detection

For lip detection, we determine the lip box. And we consider that lip must be inside the lip box. So, first we determine the distance between the forehead and eyes. Then we add the distance with the lower height of the eye to determine the upper height of the box which will contain the lip. Now, the starting point of the box will be the ¼ position of the left eye box and ending point will be the ¾ position of the right eye box. And the ending height of the box will be the lower end of the face image. So, this box will contain only lip and may some part of the nose. Then we will cut the RGB image according the box.


So, for detection eyes and lip, we only need to convert binary image from RGB image and some searching among the binary image.


Apply Bezier Curve on Lip

In the lip box, there is lip and may be some part of nose. So, around the box there is skin color or the skin. So, we convert the skin pixel to white pixel and other pixel as black. We also find those pixels which are similar to skin pixels and convert them to white pixel. Here, if two pixels RGB values difference is less than or equal 10, then we called them similar pixel. Here, we use histogram for finding the distance between the lower average RGB value and higher average RGB value. If the distance is less than 70, then we use 7 for finding similar pixel and if the distance is getter than or equal 70 then we use 10 for finding similar pixel. So, the value for finding similar pixel depends on the quality of the image. If the image quality is high, we use 7 for finding similar pixel and if the image quality is low, we use 10.


So, in the binary image, there are black regions on lip, nose and may some other little part which have a little different than skin color. Then we apply big connected region for finding the black region which contain lip in binary image. And we are sure that the big connected region is the lip because in the lip box, lip is the largest thing which is different than skin.


Then we have to apply Bezier curve on the binary lip. For apply Bezier curve, we find the starting and ending pixel of the lip in horizontal. Then we draw two tangents on upper lip from the starting and ending pixel and also find two points on the tangent which is not the part of the lip. For the lower lip, we find two point similar process of the upper lip. We use Cubic Bezier curves for draw the Bezier curve of the lip. We draw two Bezier curve for the lip, one for upper lip and one for lower lip.


Apply Bezier Curve on Eye

For apply Bezier curve on eyes, first we have to remove eyebrow from eye. For remove eyebrow, we search 1st continuous black pixel then continuous white pixel and then continuous black pixel from the binary image of the eye box. Then we remove the 1st continuous black pixel from the box and then we get the box which only contains the eye.

pic-20.PNG pic-21.PNG

Now, the eye box which contains only eye, has some skin or skin color around the box. So, we apply similar skin color like the lip for finding the region of eye. Then we apply big connect for finding the highest connected region and this is the eye because in the eye box, eye is the biggest thing which is not similar to the skin color.

pic-22.PNG pic-23.PNG

Then we apply the Bezier curve on the eye box, similar to the lip. Then we get the shape of the eye.


Database and Training

In our database, there are two tables. One table “Person” is for storing the name of people and their index of 4 kinds of emotion which are stored in other table “Position”. In the “Position” table, for each index, there are 6 control points for lip Bezier curve, 6 control points for left eye Bezier curve, 6 control points for right eye Bezier curve, lip height and width, left eye height and width and right eye height and width. So, by this method, the program learns the emotion of the people.


Emotion Detection

For emotion detection of an image, we have to find the Bezier curve of the lip, left eye and right eye. Then we convert each width of the Bezier curve to 100 and height according to its width. If the person’s emotion information is available in the database, then the program will match which emotion’s height is nearest the current height and the program will give the nearest emotion as output.


If the person’s emotion information is not available in the database, then the program calculates the average height for each emotion in the database for all people and then get a decision according to the average height.


Using the Program

We can use it in this way:

  1. Select an image by clicking Browse button.
  2. Then click Skin Color button.
  3. Then click Connected button.
  4. Then click Next button, then it will open a new window.
  5. Then click Binary Image button.
  6. Then click Face button.
  7. Then click eye lip button.
  8. Then click Left_Eye_Next button.
  9. Then click Left_Eye_Next button again.
  10. Then click Left_Eye_Next button again.
  11. Then click Left_Eye_Next button again.
  12. Then click Right_Eye_Next button.
  13. Then click Right_Eye_Next button again.
  14. Then click Right_Eye_Next button again.
  15. Then click Right_Eye_Next button again.
  16. Then click Lip_Next button.
  17. Then click Lip_Next button again.
  18. Then click Lip_Next button again.
  19. Then click Emotion button for show the emotion.
  20. If you want to enter a face result to the database then, write the person's name in the text box & select a emotion from the combo box then click the Entry button.


  • 18 September, 2010: Initial post


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

Software Developer (Senior) TigerIT Bangladesh Ltd.
Bangladesh Bangladesh
I am Committed to professionalism, eagerness to learn and has strong passion in software engineering.

You may also be interested in...

Comments and Discussions

GeneralHelp in Database.... Pin
Member 123884032-Sep-16 20:56
memberMember 123884032-Sep-16 20:56 
Newsregarding the technology Pin
Member 124297842-Aug-16 19:23
memberMember 124297842-Aug-16 19:23 
Questiondoes not retrieve result from database. Pin
Member 1185863810-Apr-16 3:35
memberMember 1185863810-Apr-16 3:35 
SuggestionWhy didn't use Eyebrow for Emotion Detection ? Appreciating this awesome work requesting documentation for details. Pin
Member 1238969419-Mar-16 23:31
memberMember 1238969419-Mar-16 23:31 
kalpesh@jalgaon6-Jan-16 21:33
memberkalpesh@jalgaon6-Jan-16 21:33 
Member 1233165719-Feb-16 0:08
memberMember 1233165719-Feb-16 0:08 
QuestionMessage Removed Pin
Member 1211602328-Nov-15 7:32
memberMember 1211602328-Nov-15 7:32 
QuestionAsking about algorithms Pin
Renas Rekany26-Oct-15 22:26
memberRenas Rekany26-Oct-15 22:26 
QuestionThankyou very much Pin
Member 120348976-Oct-15 21:41
memberMember 120348976-Oct-15 21:41 
QuestionVery Imp!!! Pin
Soneeya19-Jun-15 5:22
memberSoneeya19-Jun-15 5:22 
AnswerRe: Very Imp!!! Pin
shakil03040031-Sep-15 2:19
membershakil03040031-Sep-15 2:19 
QuestionBezier only use to draw Pin
Linh Lee6-Jun-15 20:53
memberLinh Lee6-Jun-15 20:53 
Questionhello Pin
Member 1162904421-Apr-15 0:35
memberMember 1162904421-Apr-15 0:35 
Questiondocumentation Pin
Member 115671111-Apr-15 10:49
memberMember 115671111-Apr-15 10:49 
Questionfor .dll files Pin
Member 1128222630-Mar-15 3:29
memberMember 1128222630-Mar-15 3:29 
QuestionDocumentation please Pin
Hoda Gamal4-Mar-15 10:53
memberHoda Gamal4-Mar-15 10:53 
QuestionDocumentation please Pin
Armand Salcedo4-Mar-15 1:47
memberArmand Salcedo4-Mar-15 1:47 
QuestionProblem with zip Pin
Member 1147563626-Feb-15 1:56
memberMember 1147563626-Feb-15 1:56 
AnswerRe: Problem with zip Pin
Member 120348975-Oct-15 10:46
memberMember 120348975-Oct-15 10:46 
QuestionDocumentation Pin
Member 1145305915-Feb-15 5:45
memberMember 1145305915-Feb-15 5:45 
QuestionGreat Work Guys.. Need Related Docs Pin
maflarich20-Jan-15 1:15
membermaflarich20-Jan-15 1:15 
yuvanathi10-Jan-15 17:37
memberyuvanathi10-Jan-15 17:37 
QuestionNeed Documentation Pin
Member 1131682116-Dec-14 21:37
memberMember 1131682116-Dec-14 21:37 
GeneralSuperb project. Pin
Member 1060361330-Oct-14 4:57
memberMember 1060361330-Oct-14 4:57 
QuestionDocumentation Pin
Member 1105326830-Oct-14 4:15
memberMember 1105326830-Oct-14 4:15 
Questionbase on this article Pin
Member 1109862721-Sep-14 8:01
memberMember 1109862721-Sep-14 8:01 
QuestionSuggest papers u followed Pin
Member 110509611-Sep-14 6:46
memberMember 110509611-Sep-14 6:46 
Questionnice project .please can u send me the documentation Pin
Member 1103858026-Aug-14 15:35
memberMember 1103858026-Aug-14 15:35 
Questionreferences problem Pin
Member 1060361316-Aug-14 7:39
memberMember 1060361316-Aug-14 7:39 
Questiondatabase need Pin
khezranmoon14-Jul-14 2:38
groupkhezranmoon14-Jul-14 2:38 
GeneralEmotion Detection Software Pin
shane007117-Apr-14 7:42
membershane007117-Apr-14 7:42 
QuestionDocumentation Pin
chinmay kulkarnee31-Mar-14 3:50
memberchinmay kulkarnee31-Mar-14 3:50 
Questionplease send the documents Pin
Member 1064995318-Mar-14 23:09
memberMember 1064995318-Mar-14 23:09 
Questiondocumentation Pin
Member 1066303114-Mar-14 1:20
memberMember 1066303114-Mar-14 1:20 
QuestionProject Documentation Pin
murray12345622-Jan-14 8:52
membermurray12345622-Jan-14 8:52 
Questionhelp about database Pin
Member 104529468-Jan-14 5:48
memberMember 104529468-Jan-14 5:48 
QuestionCan I Get Your Project Documentation? Pin
rfitha6-Jan-14 16:23
memberrfitha6-Jan-14 16:23 
Questiondatabase problem Pin
sss230225-Dec-13 22:33
membersss230225-Dec-13 22:33 
QuestionDocumentation Pin
Zeinas29-Oct-13 22:36
memberZeinas29-Oct-13 22:36 
QuestionHi Pin
tnla14-Oct-13 8:47
membertnla14-Oct-13 8:47 
QuestionGood job Pin
tnla14-Oct-13 8:43
membertnla14-Oct-13 8:43 
Questionfor project document Pin
Member 1017885629-Jul-13 20:31
memberMember 1017885629-Jul-13 20:31 
QuestionEmotion detection document Pin
Member 1017097224-Jul-13 4:14
memberMember 1017097224-Jul-13 4:14 
GeneralMy vote of 4 Pin
aya Amer27-Apr-13 2:43
memberaya Amer27-Apr-13 2:43 
QuestionFilters and doc Pin
MiroBeslav Kosanic31-Mar-13 11:52
memberMiroBeslav Kosanic31-Mar-13 11:52 
Questionthere is a lot of references.. Pin
GREG_DORIANcod1-Mar-13 10:11
memberGREG_DORIANcod1-Mar-13 10:11 
QuestionProjectDoc & Database Pin
Mustafa Yalniz14-Jan-13 13:49
memberMustafa Yalniz14-Jan-13 13:49 
QuestionProject Documentation Pin
Member 96983276-Jan-13 17:46
memberMember 96983276-Jan-13 17:46 
Questionerror while running the code Pin
sreevisakh336-Jan-13 3:27
membersreevisakh336-Jan-13 3:27 
the error messsage is like..
the "microsoft.jet.OLEDB.4.0" provider is not registered on the local machine..
plz help me to sort out this problem

GeneralMy vote of 5 Pin
Member 81544248-Dec-12 0:09
memberMember 81544248-Dec-12 0:09 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.161026.1 | Last Updated 20 Sep 2010
Article Copyright 2010 by shakil0304003
Everything else Copyright © CodeProject, 1999-2016
Layout: fixed | fluid