Click here to Skip to main content
15,867,756 members
Articles / Programming Languages / XML

Mouse Control via Webcam

, , ,
Rate me:
Please Sign up or sign in to vote.
4.82/5 (62 votes)
10 Dec 2012CPOL4 min read 202.1K   36.9K   121   57
Control your mouse using hand gestures

Introduction

This application uses emguCV, a .NET wrapper for openCV, to perform image processing, through which we try to recognize hand gestures and control mouse using these gestures. In this app, cursor movement is controlled by the movement of hand and click events are triggered using hand gestures.

Image 1

Background

I always wanted to have one such handy app to control mouse. So when, Sir Sajid Hussain, our teacher at Institute of Industrial Electronic Engineering IIEE, asked us to make an application using C#, we (I, Sajjad Idrees and Saad Hafeez) decided it to be the one. There are many applications made before that perform a similar task, but they usually use kinect or something else, very few of them used simple webcam for input. Though the application is not so perfect and needs a lot of work to perfect it, it does work.

Using the Code

The code uses a well known library for image processing, known as, openCV in the form of emguCV which is a .NET wrapper of it. The code is really simple and easy to understand.

First of all, the application tries to catch a video input device. If it is successful in doing so, it calls the function ProcessFramAndUpdateGUI(); where all the processing is done.

C#
private void Form1_Load(object sender, EventArgs e)
{
	//trying to capture a vedio input device
    try
    {
        CapWebCam = new Capture();
    }
    catch ()
    {                
    }

    Application.Idle += ProcessFramAndUpdateGUI;
}  

The processing code is divided into three main parts:

  1. Image Filtering to obtain biggest skin colored contour
  2. Finding convexity defects to count number of fingers, for mouse click events
  3. Finding center of palm and do necessary noise filtering to move cursor position

Filtering to Obtain Skin Colored Contour

First of all, YCbCr filter is applied, with required thresholds to obtain skin colored parts of the image.

C#
int Finger_num = 0;
Double Result1 = 0;
Double Result2 = 0;
		
imgOrignal = CapWebCam.QueryFrame();	//querying image

if (imgOrignal == null) return;		// pass only if there is some image
            
//Applying YCrCb filter
Image<Ycc, Byte> currentYCrCbFrame = imgOrignal.Convert<Ycc, byte>();
Image<Gray, byte> skin = new Image<Gray, byte>(imgOrignal.Width, imgOrignal.Height);
            
skin = currentYCrCbFrame.InRange(new Ycc(0, 131, 80), new Ycc(255, 185, 135));
            
StructuringElementEx rect_12 = 
  new StructuringElementEx(10, 10, 5, 5, Emgu.CV.CvEnum.CV_ELEMENT_SHAPE.CV_SHAPE_RECT);

Now eroding and dilation of the filtered image is done using cv.erode() and cv.dilate() functions.

C#
StructuringElementEx rect_12 = 
   new StructuringElementEx(10, 10, 5, 5, Emgu.CV.CvEnum.CV_ELEMENT_SHAPE.CV_SHAPE_RECT);
 
//Eroding the source image using the specified structuring element
CvInvoke.cvErode(skin, skin, rect_12, 1);
            
StructuringElementEx rect_6 = 
  new StructuringElementEx(6, 6, 3, 3, Emgu.CV.CvEnum.CV_ELEMENT_SHAPE.CV_SHAPE_RECT);
 
//dilating the source image using the specified structuring element
CvInvoke.cvDilate(skin, skin, rect_6, 2);

Finally, smoothing of the image is done using Gaussian filter and from the resulting image, the biggest contour is extracted.

C#
//smoothing the filterd , eroded and dilated image.
skin = skin.SmoothGaussian(9);                    
          
Contour<Point> contours = skin.FindContours(); //extracting all contours.
Contour<Point> biggestContour = null;
            
//extracting the biggest contour.
while (contours != null)
{
    Result1 = contours.Area;
    if (Result1 >

Counting Fingers using Convexity Defects

Now to determine the number of fingers, I used a very popular method available known as convexity defects. This method determines all of the defects in our contour, which tells us the number of fingers in the image. The code is as follows:

C#
//applying convexty defect allgoritm to find the count of fingers
if (biggestContour != null)
{
    Finger_num = 0;
    
    biggestContour = biggestContour.ApproxPoly((0.00025));
    imgOrignal.Draw(biggestContour, new Bgr(Color.LimeGreen), 2);
    
    Hull = biggestContour.GetConvexHull(ORIENTATION.CV_CLOCKWISE);
    defects = biggestContour.GetConvexityDefacts(storage, ORIENTATION.CV_CLOCKWISE);
    imgOrignal.DrawPolyline(Hull.ToArray(), true, new Bgr(0, 0, 256), 2);
    
    box = biggestContour.GetMinAreaRect();
    
    defectArray = defects.ToArray();
    
    for (int i = 0; i < defects.Total; i++)
    {
        PointF startPoint = new PointF((float)defectArray[i].StartPoint.X,
                                    (float)defectArray[i].StartPoint.Y);

        PointF depthPoint = new PointF((float)defectArray[i].DepthPoint.X,
                                        (float)defectArray[i].DepthPoint.Y);

        PointF endPoint = new PointF((float)defectArray[i].EndPoint.X,
                                        (float)defectArray[i].EndPoint.Y);

                    
        CircleF startCircle = new CircleF(startPoint, 5f);
        CircleF depthCircle = new CircleF(depthPoint, 5f);
        CircleF endCircle = new CircleF(endPoint, 5f);
        
        
        if (    (startCircle.Center.Y < box.center.Y || depthCircle.Center.Y < box.center.Y) && 
                (startCircle.Center.Y < depthCircle.Center.Y) && 
                (Math.Sqrt(Math.Pow(startCircle.Center.X - depthCircle.Center.X, 2) + 
                           Math.Pow(startCircle.Center.Y - depthCircle.Center.Y, 2)) > 
                           box.size.Height / 6.5)   )
        {
            Finger_num++;
        }

With finger count known, I associated mouse click events with it. For the user to click mouse left button, he/she opens his/her palm, as soon as the finger count goes greater than four, mouse left button is clicked.

Finding the Center of Contour

Finally, we try to find the center of our contour, using moments of contour. As soon as we were able to get the coordinates of the center, we noticed there was too much fluctuation in the center of contour, as the hand continuously flickered. For this problem, we divided the coordinates of center by 10 to remove unit part of them as there was no fluctuation in the tenths of the coordinates. Here is the code:

C#
MCvMoments moment = new MCvMoments();                    // a new MCvMoments object

moment = biggestContour.GetMoments();                    // Moments of biggestContour

CvInvoke.cvMoments(biggestContour, ref moment, 0);
        
double m_00 = CvInvoke.cvGetSpatialMoment(ref moment, 0, 0);
double m_10 = CvInvoke.cvGetSpatialMoment(ref moment, 1, 0);
double m_01 = CvInvoke.cvGetSpatialMoment(ref moment, 0, 1);

int current_X = Convert.ToInt32(m_10 / m_00) / 10;       // X location of centre of contour              
int current_Y = Convert.ToInt32(m_01 / m_00) / 10;       // Y location of center of contour

We also faced another problem, the center shifted when the palm was open and closed. For this problem, we placed the conditions that mouse cursor position will only be moved if the palm is closed, i.e., finger count is zero, else mouse will remain where it was. The code below also shows mouse click event:

C#
// move cursor to center of contour only if Finger count is 1 or 0
// i.e. palm is closed

if (Finger_num == 0 || Finger_num == 1)
{
    Cursor.Position = new Point(current_X * 20, current_Y * 20);   
}

// Leave the cursor where it was and Do mouse click, if finger count >= 4

if (Finger_num >= 4)
{
    DoMouseClick();                     // function clicks mouse left button
}  

The function definition for DoMouseClick() is as follows, however system32 function mouse_event() is now obsolete but we used it anyway.

C#
// function for mouse clicks

[DllImport("user32.dll")]
static extern void mouse_event(uint dwFlags, uint dx, uint dy, uint dwData,
   int dwExtraInfo);

private const int MOUSEEVENTF_LEFTDOWN = 0x02;          // mouse left button pressed 
private const int MOUSEEVENTF_LEFTUP = 0x04;            // mouse left button unpressed
private const int MOUSEEVENTF_RIGHTDOWN = 0x08;         // mouse right button pressed
private const int MOUSEEVENTF_RIGHTUP = 0x10;           // mouse right button unpressed

//this function will click the mouse using the parameters assigned to it
public void DoMouseClick()
{
    //Call the imported function with the cursor's current position
    uint X = Convert.ToUInt32(Cursor.Position.X);
    uint Y =Convert.ToUInt32(Cursor.Position.Y);
    mouse_event(MOUSEEVENTF_LEFTDOWN | MOUSEEVENTF_LEFTUP, X, Y, 0, 0);
}     

That's all about using the code, hope it helps you. There are many things yet to be done to perfect this application. We hope suggestions from viewers may help us improve it.

Note: We have not included emguCV DLLs in the attached file, one must install emguCV and copy all DLLs to run the project.

Run this Application on Your PC

There is a three step process to run this application on your PC:

  1. Download and install emguCV library from here.
  2. Copy all files from "C:\Emgu\emgucv-windows-x86 2.4.0.1717\bin\x86\" into your system32 folder.
  3. Download and install the setup of application from the following link:

Points of Interest

As Windows 8 is now available in the market, and it provides much of a tablet touch to the PC, we are thinking of improving this application to be able to use hand movements to provide touch effect to a laptop.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Instructor / Trainer Institute of Industrial Electronics Engineering
Pakistan Pakistan
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Written By
Student Institute of Industrial Electronic Engineering
Pakistan Pakistan
An undergraduate student at Institute of Industrial Electronic Engineering (IIEE).

Written By
Software Developer institute of industrial electronics engineering
Pakistan Pakistan
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Written By
Student
Pakistan Pakistan
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
QuestionPlease share the libraries Pin
Sunny Boyka13-Jan-19 17:33
professionalSunny Boyka13-Jan-19 17:33 
Questionobjecttracking_openCV has stopped working Pin
Member 1311322126-Jul-17 17:15
Member 1311322126-Jul-17 17:15 
Questionlink not working Pin
Member 129334493-Jan-17 19:05
Member 129334493-Jan-17 19:05 
GeneralApplication Set Up link Pin
Member 115916333-Jan-17 18:59
Member 115916333-Jan-17 18:59 
Kindly provide a valid application set up link !

Link you provided is not working.
QuestionApplication Set Up link not Valid Pin
Member 115916333-Jan-17 19:33
Member 115916333-Jan-17 19:33 
QuestionMouse Control via Webcam giving me exception Pin
amirgee00728-Apr-16 16:08
amirgee00728-Apr-16 16:08 
QuestionProblems with emgu 3.0.0 Pin
Member 1205808414-Jan-16 3:51
Member 1205808414-Jan-16 3:51 
Questioni cant access this ... Pin
Member 1190876313-Aug-15 23:46
Member 1190876313-Aug-15 23:46 
QuestionWorking, but Pin
Member 116549322-May-15 4:42
Member 116549322-May-15 4:42 
AnswerRe: Working, but Pin
Anas Hashmi7-May-15 21:44
Anas Hashmi7-May-15 21:44 
Questiondouble click on button using finger movement Pin
sureshsharma12312-Jan-15 19:32
sureshsharma12312-Jan-15 19:32 
AnswerRe: double click on button using finger movement Pin
saadhafeez30-Mar-15 20:21
saadhafeez30-Mar-15 20:21 
QuestionDoes this program not work for windows 8? Pin
Member 1108676316-Nov-14 17:45
Member 1108676316-Nov-14 17:45 
AnswerRe: Does this program not work for windows 8? Pin
Anas Hashmi7-May-15 21:42
Anas Hashmi7-May-15 21:42 
GeneralMy vote of 3 Pin
Member 996931720-Aug-14 12:37
Member 996931720-Aug-14 12:37 
GeneralAll of you are Awsome! Pin
Zain Ul Abidin24-May-14 7:38
Zain Ul Abidin24-May-14 7:38 
QuestionImpressive Work! Pin
Member 1058130527-Mar-14 4:50
Member 1058130527-Mar-14 4:50 
Questionobjecttracking_openCV has stopped working Pin
adambharathi1325-Apr-13 23:37
adambharathi1325-Apr-13 23:37 
AnswerRe: objecttracking_openCV has stopped working Pin
payjo20-Sep-13 18:56
payjo20-Sep-13 18:56 
GeneralRe: objecttracking_openCV has stopped working Pin
Member 1085265229-May-14 7:30
Member 1085265229-May-14 7:30 
AnswerRe: objecttracking_openCV has stopped working Pin
Member 1311322126-Jul-17 17:17
Member 1311322126-Jul-17 17:17 
GeneralMy vote of 5 Pin
ibrahim_ragab11-Apr-13 13:26
professionalibrahim_ragab11-Apr-13 13:26 
BugTypeInitializationException was unhandled in Form1 (at MemStorage() constructor call)? Pin
supernorb26-Mar-13 2:39
supernorb26-Mar-13 2:39 
GeneralRe: TypeInitializationException was unhandled in Form1 (at MemStorage() constructor call)? Pin
Anas Hashmi26-Mar-13 3:43
Anas Hashmi26-Mar-13 3:43 
GeneralRe: TypeInitializationException was unhandled in Form1 (at MemStorage() constructor call)? Pin
amirgee00728-Apr-16 16:09
amirgee00728-Apr-16 16:09 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.