Click here to Skip to main content
11,495,331 members (66,781 online)
Click here to Skip to main content

Mouse Control via Webcam

, , , , 10 Dec 2012 CPOL 76.9K 22.6K 101
Rate this:
Please Sign up or sign in to vote.
Control your mouse using hand gestures.

Introduction 

This application uses emguCV, a .NET wrapper for openCV, to perform image processing, through which we try to recognize hand gestures and control mouse using these gestures. In this app, cursor movement is controlled by the movement of hand and click events are triggered using hand gestures. 

Background

I always wanted to have one such handy app to control mouse. So when, Sir Sajid Hussain, our teacher at Institute of Industrial Electronic Engineering IIEE, asked us to make an application using C#, we (I, Sajjad Idrees and Saad Hafeez) decided it to be the one. There are many applications made before that perform similar task, but they usually use kinect or something else, very few of them used simple webcam for input. Though the application is not so perfect and needs a lot of work to perfect it, but it does work. 

Using the code  

The code uses a well known library for image processing, known as, openCV in the form of emguCV which is a .NET wrapper of it. The code is really simple and easy to understand. 

First of all the application tries to catch a video input device. If it is successful in doing so, it calls the function ProcessFramAndUpdateGUI(); where all the processing is done.

private void Form1_Load(object sender, EventArgs e)
{
	//trying to capture a vedio input device
    try
    {
        CapWebCam = new Capture();
    }
    catch ()
    {                
    }

    Application.Idle += ProcessFramAndUpdateGUI;
}  

The processing code is divided into three main parts:

  1. Image Filtering to obtain biggest skin colored contour.
  2. Finding convexity defects to count number of fingers, for mouse click events. 
  3. Finding center of palm and do necessary noise filtering to move cursor position. 

Filtering to obtain skin colored contour

First of all YCbCr filter is applied, with required thresholds to obtain skin colored parts of the image. 

int Finger_num = 0;
Double Result1 = 0;
Double Result2 = 0;
		
imgOrignal = CapWebCam.QueryFrame();	//querying image

if (imgOrignal == null) return;		// pass only if there is some image
            
//Applying YCrCb filter
Image<Ycc, Byte> currentYCrCbFrame = imgOrignal.Convert<Ycc, byte>();
Image<Gray, byte> skin = new Image<Gray, byte>(imgOrignal.Width, imgOrignal.Height);
            
skin = currentYCrCbFrame.InRange(new Ycc(0, 131, 80), new Ycc(255, 185, 135));
            
StructuringElementEx rect_12 = 
  new StructuringElementEx(10, 10, 5, 5, Emgu.CV.CvEnum.CV_ELEMENT_SHAPE.CV_SHAPE_RECT);

Now eroding and dilation of  the filtered image is done using cv.erode() and cv.dilate() functions. 

StructuringElementEx rect_12 = 
   new StructuringElementEx(10, 10, 5, 5, Emgu.CV.CvEnum.CV_ELEMENT_SHAPE.CV_SHAPE_RECT);
 
//Eroding the source image using the specified structuring element
CvInvoke.cvErode(skin, skin, rect_12, 1);
            
StructuringElementEx rect_6 = 
  new StructuringElementEx(6, 6, 3, 3, Emgu.CV.CvEnum.CV_ELEMENT_SHAPE.CV_SHAPE_RECT);
 
//dilating the source image using the specified structuring element
CvInvoke.cvDilate(skin, skin, rect_6, 2);

Finally smoothing of the image is done using gaussian filter and from the resulting image the biggest contour is extracted.

//smoothing the filterd , eroded and dilated image.
skin = skin.SmoothGaussian(9);                    
          
Contour<Point> contours = skin.FindContours(); //extracting all contours.
Contour<Point> biggestContour = null;
            
//extracting the biggest contour.
while (contours != null)
{
    Result1 = contours.Area;
    if (Result1 >

Counting fingers using convexity defects 

Now to determine the number of fingers, I used a very popular method available known as convexity defects. This method determines all of the defects in our contour, which tells us the number of fingers in the image. The code is as follows:

//applying convexty defect allgoritm to find the count of fingers
if (biggestContour != null)
{
    Finger_num = 0;
    
    biggestContour = biggestContour.ApproxPoly((0.00025));
    imgOrignal.Draw(biggestContour, new Bgr(Color.LimeGreen), 2);
    
    Hull = biggestContour.GetConvexHull(ORIENTATION.CV_CLOCKWISE);
    defects = biggestContour.GetConvexityDefacts(storage, ORIENTATION.CV_CLOCKWISE);
    imgOrignal.DrawPolyline(Hull.ToArray(), true, new Bgr(0, 0, 256), 2);
    
    box = biggestContour.GetMinAreaRect();
    
    defectArray = defects.ToArray();
    
    for (int i = 0; i < defects.Total; i++)
    {
        PointF startPoint = new PointF((float)defectArray[i].StartPoint.X,
                                    (float)defectArray[i].StartPoint.Y);

        PointF depthPoint = new PointF((float)defectArray[i].DepthPoint.X,
                                        (float)defectArray[i].DepthPoint.Y);

        PointF endPoint = new PointF((float)defectArray[i].EndPoint.X,
                                        (float)defectArray[i].EndPoint.Y);

                    
        CircleF startCircle = new CircleF(startPoint, 5f);
        CircleF depthCircle = new CircleF(depthPoint, 5f);
        CircleF endCircle = new CircleF(endPoint, 5f);
        
        
        if (    (startCircle.Center.Y < box.center.Y || depthCircle.Center.Y < box.center.Y) && 
                (startCircle.Center.Y < depthCircle.Center.Y) && 
                (Math.Sqrt(Math.Pow(startCircle.Center.X - depthCircle.Center.X, 2) + 
                           Math.Pow(startCircle.Center.Y - depthCircle.Center.Y, 2)) > 
                           box.size.Height / 6.5)   )
        {
            Finger_num++;
        }

With finger count known i associated mouse click events with it. For the user to click mouse left button, he/she opens his/her palm, as soon as the finger count goes greater than four mouse left button is clicked. 

Finding the center of contour 

Finally, we try to find the center of our contour, using moments of contour. As soon as we were able to get the co ordinates of the center we noticed there was too much fluctuation in the center of contour, as the hand continuously flickered. For this problem we divided the co ordinates of center by 10 to remove unit part of them as there was no fluctuation in the tenths of the co ordinates. Here is the code:

MCvMoments moment = new MCvMoments();               // a new MCvMoments object

moment = biggestContour.GetMoments();           // Moments of biggestContour

CvInvoke.cvMoments(biggestContour, ref moment, 0);
        
double m_00 = CvInvoke.cvGetSpatialMoment(ref moment, 0, 0);
double m_10 = CvInvoke.cvGetSpatialMoment(ref moment, 1, 0);
double m_01 = CvInvoke.cvGetSpatialMoment(ref moment, 0, 1);

int current_X = Convert.ToInt32(m_10 / m_00) / 10;      // X location of centre of contour              
int current_Y = Convert.ToInt32(m_01 / m_00) / 10;      // Y location of center of contour

We also faced another problem, the center shifted when the palm was open and closed. For this problem we placed the conditions that mouse cursor position will only be moved if the palm is closed i.e. finger count is zero, else mouse will remain where it was. The code below also shows mouse click event:

// move cursor to center of contour only if Finger count is 1 or 0
// i.e. palm is closed

if (Finger_num == 0 || Finger_num == 1)
{
    Cursor.Position = new Point(current_X * 20, current_Y * 20);   
}

// Leave the cursor where it was and Do mouse click, if finger count >= 4

if (Finger_num >= 4)
{
    DoMouseClick();                     // function clicks mouse left button
}  

The function definition for DoMouseClick() is as follows, however system32 function mouse_event() is now obsolete but we used it anyway. 

// function for mouse clicks

[DllImport("user32.dll")]
static extern void mouse_event(uint dwFlags, uint dx, uint dy, uint dwData,
   int dwExtraInfo);

private const int MOUSEEVENTF_LEFTDOWN = 0x02;          // mouse left button pressed 
private const int MOUSEEVENTF_LEFTUP = 0x04;            // mouse left button unpressed
private const int MOUSEEVENTF_RIGHTDOWN = 0x08;         // mouse right button pressed
private const int MOUSEEVENTF_RIGHTUP = 0x10;           // mouse right button unpressed

//this function will click the mouse using the parameters assigned to it
public void DoMouseClick()
{
    //Call the imported function with the cursor's current position
    uint X = Convert.ToUInt32(Cursor.Position.X);
    uint Y =Convert.ToUInt32(Cursor.Position.Y);
    mouse_event(MOUSEEVENTF_LEFTDOWN | MOUSEEVENTF_LEFTUP, X, Y, 0, 0);
}     

That's all about using the code, hope it helps you. There are many things yet to be done to perfect this application. We hope suggestions from viewers that may help us improve it.  

Note: We have not included emguCV DLLs in the attached file, one must install emguCV and copy all DLLs to run the project.

Run this application on your PC 

There is a three step process to run this application on your PC:

1. Download and install emguCV library from here.  

2. Copy all files from "C:\Emgu\emgucv-windows-x86 2.4.0.1717\bin\x86\" into your system32 folder.

3. Download and install the setup of application from the following link. 

Mouse Control via Webcam Setup  

Points of Interest 

As Windows 8 is now available in the market, and it provides much of a tablet touch to the PC. We are thinking to improve this application to be able to use hand movements to provide touch effect to a laptop. 

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Authors

sajidhussainbaloch
Instructor / Trainer Institute of Industrial Electronics Engineering
Pakistan Pakistan
No Biography provided

saadhafeez
Software Developer institute of industrial electronics engineering
Pakistan Pakistan
No Biography provided

Anas Hashmi
Student Institute of Industrial Electronic Engineering
Pakistan Pakistan
An undergraduate student at Institute of Industrial Electronic Engineering (IIEE).

Sajjad Idrees
Student
Pakistan Pakistan
No Biography provided

Comments and Discussions

 
GeneralMessage Removed Pin
Member 1172448327-May-15 22:10
memberMember 1172448327-May-15 22:10 
QuestionWorking, but Pin
Member 116549322-May-15 5:42
memberMember 116549322-May-15 5:42 
AnswerRe: Working, but Pin
Anas Hashmi7-May-15 22:44
memberAnas Hashmi7-May-15 22:44 
Questiondouble click on button using finger movement Pin
sureshsharma12312-Jan-15 20:32
membersureshsharma12312-Jan-15 20:32 
AnswerRe: double click on button using finger movement Pin
saadhafeez30-Mar-15 21:21
membersaadhafeez30-Mar-15 21:21 
QuestionDoes this program not work for windows 8? Pin
Member 1108676316-Nov-14 18:45
memberMember 1108676316-Nov-14 18:45 
AnswerRe: Does this program not work for windows 8? Pin
Anas Hashmi7-May-15 22:42
memberAnas Hashmi7-May-15 22:42 
GeneralMy vote of 3 Pin
Member 996931720-Aug-14 13:37
memberMember 996931720-Aug-14 13:37 
GeneralAll of you are Awsome! Pin
Zain Ul Abidin24-May-14 8:38
memberZain Ul Abidin24-May-14 8:38 
QuestionImpressive Work! Pin
Member 1058130527-Mar-14 5:50
memberMember 1058130527-Mar-14 5:50 
Questionobjecttracking_openCV has stopped working Pin
adambharathi1326-Apr-13 0:37
memberadambharathi1326-Apr-13 0:37 
AnswerRe: objecttracking_openCV has stopped working Pin
payjo20-Sep-13 19:56
memberpayjo20-Sep-13 19:56 
GeneralRe: objecttracking_openCV has stopped working Pin
Member 1085265229-May-14 8:30
memberMember 1085265229-May-14 8:30 
GeneralMy vote of 5 Pin
ibrahim_ragab11-Apr-13 14:26
memberibrahim_ragab11-Apr-13 14:26 
BugTypeInitializationException was unhandled in Form1 (at MemStorage() constructor call)? Pin
supernorb26-Mar-13 3:39
membersupernorb26-Mar-13 3:39 
GeneralRe: TypeInitializationException was unhandled in Form1 (at MemStorage() constructor call)? Pin
Anas Hashmi26-Mar-13 4:43
memberAnas Hashmi26-Mar-13 4:43 
Questionregarding installation Pin
devandran1923-Mar-13 6:45
memberdevandran1923-Mar-13 6:45 
QuestionWill this run on windows xp pro? Pin
dnicolson3-Mar-13 1:06
memberdnicolson3-Mar-13 1:06 
GeneralMy vote of 5 Pin
Phat (Phillip) H. VU8-Jan-13 21:50
memberPhat (Phillip) H. VU8-Jan-13 21:50 
GeneralMy vote of 4 Pin
maq_rohit8-Jan-13 17:37
membermaq_rohit8-Jan-13 17:37 
GeneralMy vote of 5 Pin
Hammad Raza24-Dec-12 22:53
memberHammad Raza24-Dec-12 22:53 
GeneralMy vote of 5 Pin
hzq0124-Dec-12 16:13
memberhzq0124-Dec-12 16:13 
GeneralMy vote of 5 Pin
Ahsan Murshed19-Dec-12 17:39
memberAhsan Murshed19-Dec-12 17:39 
QuestionSuper Application! Pin
Jean Paul V.A18-Dec-12 5:59
memberJean Paul V.A18-Dec-12 5:59 
GeneralMy vote of 5 Pin
soulprovidergr18-Dec-12 3:31
membersoulprovidergr18-Dec-12 3:31 
GeneralMy vote of 5 Pin
wsc091816-Dec-12 19:43
memberwsc091816-Dec-12 19:43 
GeneralMy vote of 5 Pin
Mihai MOGA14-Dec-12 6:09
memberMihai MOGA14-Dec-12 6:09 
GeneralMy vote of 5 Pin
Farhan Ghumra11-Dec-12 2:41
memberFarhan Ghumra11-Dec-12 2:41 
GeneralMy vote of 5 Pin
Savalia Manoj M10-Dec-12 21:54
memberSavalia Manoj M10-Dec-12 21:54 
GeneralMy vote of 5 Pin
bubifengyun9-Dec-12 21:36
memberbubifengyun9-Dec-12 21:36 
QuestionCould you add your opencv.dll in your project to download Pin
bubifengyun9-Dec-12 21:35
memberbubifengyun9-Dec-12 21:35 
AnswerRe: Could you add your opencv.dll in your project to download Pin
csharpbd18-Aug-14 23:49
membercsharpbd18-Aug-14 23:49 
GeneralMy vote of 5 Pin
Md. Marufuzzaman9-Dec-12 2:08
mentorMd. Marufuzzaman9-Dec-12 2:08 
QuestionMaybe I didn't read it carefully... Pin
Cristian Amarie8-Dec-12 6:45
memberCristian Amarie8-Dec-12 6:45 
QuestionMy vote of 5 Pin
Muhammad Shahid Farooq28-Nov-12 22:36
memberMuhammad Shahid Farooq28-Nov-12 22:36 
GeneralMy vote of 5 Pin
Marco Bertschi27-Nov-12 5:22
memberMarco Bertschi27-Nov-12 5:22 
Questionregarding openCV dlls Pin
kumarjammula24-Nov-12 5:38
memberkumarjammula24-Nov-12 5:38 
AnswerRe: regarding openCV dlls Pin
Anas Hashmi24-Nov-12 5:48
memberAnas Hashmi24-Nov-12 5:48 
GeneralMessage Removed Pin
kumarjammula24-Nov-12 6:07
memberkumarjammula24-Nov-12 6:07 
GeneralRe: regarding openCV dlls Pin
Anas Hashmi24-Nov-12 6:30
memberAnas Hashmi24-Nov-12 6:30 
GeneralRe: regarding openCV dlls Pin
kumarjammula24-Nov-12 7:21
memberkumarjammula24-Nov-12 7:21 
QuestionemguCV Pin
gil6524-Nov-12 4:50
membergil6524-Nov-12 4:50 
AnswerRe: emguCV Pin
Anas Hashmi24-Nov-12 5:39
memberAnas Hashmi24-Nov-12 5:39 
GeneralMy vote of 4 Pin
Muhammad Zaid Pirwani24-Nov-12 4:28
memberMuhammad Zaid Pirwani24-Nov-12 4:28 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.150520.1 | Last Updated 10 Dec 2012
Article Copyright 2012 by sajidhussainbaloch, saadhafeez, Anas Hashmi, Sajjad Idrees
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid