Click here to Skip to main content
15,867,308 members
Articles / Programming Languages / XML

Mouse Control via Webcam

, , ,
Rate me:
Please Sign up or sign in to vote.
4.82/5 (62 votes)
10 Dec 2012CPOL4 min read 202K   36.9K   121   57
Control your mouse using hand gestures

Introduction

This application uses emguCV, a .NET wrapper for openCV, to perform image processing, through which we try to recognize hand gestures and control mouse using these gestures. In this app, cursor movement is controlled by the movement of hand and click events are triggered using hand gestures.

Image 1

Background

I always wanted to have one such handy app to control mouse. So when, Sir Sajid Hussain, our teacher at Institute of Industrial Electronic Engineering IIEE, asked us to make an application using C#, we (I, Sajjad Idrees and Saad Hafeez) decided it to be the one. There are many applications made before that perform a similar task, but they usually use kinect or something else, very few of them used simple webcam for input. Though the application is not so perfect and needs a lot of work to perfect it, it does work.

Using the Code

The code uses a well known library for image processing, known as, openCV in the form of emguCV which is a .NET wrapper of it. The code is really simple and easy to understand.

First of all, the application tries to catch a video input device. If it is successful in doing so, it calls the function ProcessFramAndUpdateGUI(); where all the processing is done.

C#
private void Form1_Load(object sender, EventArgs e)
{
	//trying to capture a vedio input device
    try
    {
        CapWebCam = new Capture();
    }
    catch ()
    {                
    }

    Application.Idle += ProcessFramAndUpdateGUI;
}  

The processing code is divided into three main parts:

  1. Image Filtering to obtain biggest skin colored contour
  2. Finding convexity defects to count number of fingers, for mouse click events
  3. Finding center of palm and do necessary noise filtering to move cursor position

Filtering to Obtain Skin Colored Contour

First of all, YCbCr filter is applied, with required thresholds to obtain skin colored parts of the image.

C#
int Finger_num = 0;
Double Result1 = 0;
Double Result2 = 0;
		
imgOrignal = CapWebCam.QueryFrame();	//querying image

if (imgOrignal == null) return;		// pass only if there is some image
            
//Applying YCrCb filter
Image<Ycc, Byte> currentYCrCbFrame = imgOrignal.Convert<Ycc, byte>();
Image<Gray, byte> skin = new Image<Gray, byte>(imgOrignal.Width, imgOrignal.Height);
            
skin = currentYCrCbFrame.InRange(new Ycc(0, 131, 80), new Ycc(255, 185, 135));
            
StructuringElementEx rect_12 = 
  new StructuringElementEx(10, 10, 5, 5, Emgu.CV.CvEnum.CV_ELEMENT_SHAPE.CV_SHAPE_RECT);

Now eroding and dilation of the filtered image is done using cv.erode() and cv.dilate() functions.

C#
StructuringElementEx rect_12 = 
   new StructuringElementEx(10, 10, 5, 5, Emgu.CV.CvEnum.CV_ELEMENT_SHAPE.CV_SHAPE_RECT);
 
//Eroding the source image using the specified structuring element
CvInvoke.cvErode(skin, skin, rect_12, 1);
            
StructuringElementEx rect_6 = 
  new StructuringElementEx(6, 6, 3, 3, Emgu.CV.CvEnum.CV_ELEMENT_SHAPE.CV_SHAPE_RECT);
 
//dilating the source image using the specified structuring element
CvInvoke.cvDilate(skin, skin, rect_6, 2);

Finally, smoothing of the image is done using Gaussian filter and from the resulting image, the biggest contour is extracted.

C#
//smoothing the filterd , eroded and dilated image.
skin = skin.SmoothGaussian(9);                    
          
Contour<Point> contours = skin.FindContours(); //extracting all contours.
Contour<Point> biggestContour = null;
            
//extracting the biggest contour.
while (contours != null)
{
    Result1 = contours.Area;
    if (Result1 >

Counting Fingers using Convexity Defects

Now to determine the number of fingers, I used a very popular method available known as convexity defects. This method determines all of the defects in our contour, which tells us the number of fingers in the image. The code is as follows:

C#
//applying convexty defect allgoritm to find the count of fingers
if (biggestContour != null)
{
    Finger_num = 0;
    
    biggestContour = biggestContour.ApproxPoly((0.00025));
    imgOrignal.Draw(biggestContour, new Bgr(Color.LimeGreen), 2);
    
    Hull = biggestContour.GetConvexHull(ORIENTATION.CV_CLOCKWISE);
    defects = biggestContour.GetConvexityDefacts(storage, ORIENTATION.CV_CLOCKWISE);
    imgOrignal.DrawPolyline(Hull.ToArray(), true, new Bgr(0, 0, 256), 2);
    
    box = biggestContour.GetMinAreaRect();
    
    defectArray = defects.ToArray();
    
    for (int i = 0; i < defects.Total; i++)
    {
        PointF startPoint = new PointF((float)defectArray[i].StartPoint.X,
                                    (float)defectArray[i].StartPoint.Y);

        PointF depthPoint = new PointF((float)defectArray[i].DepthPoint.X,
                                        (float)defectArray[i].DepthPoint.Y);

        PointF endPoint = new PointF((float)defectArray[i].EndPoint.X,
                                        (float)defectArray[i].EndPoint.Y);

                    
        CircleF startCircle = new CircleF(startPoint, 5f);
        CircleF depthCircle = new CircleF(depthPoint, 5f);
        CircleF endCircle = new CircleF(endPoint, 5f);
        
        
        if (    (startCircle.Center.Y < box.center.Y || depthCircle.Center.Y < box.center.Y) && 
                (startCircle.Center.Y < depthCircle.Center.Y) && 
                (Math.Sqrt(Math.Pow(startCircle.Center.X - depthCircle.Center.X, 2) + 
                           Math.Pow(startCircle.Center.Y - depthCircle.Center.Y, 2)) > 
                           box.size.Height / 6.5)   )
        {
            Finger_num++;
        }

With finger count known, I associated mouse click events with it. For the user to click mouse left button, he/she opens his/her palm, as soon as the finger count goes greater than four, mouse left button is clicked.

Finding the Center of Contour

Finally, we try to find the center of our contour, using moments of contour. As soon as we were able to get the coordinates of the center, we noticed there was too much fluctuation in the center of contour, as the hand continuously flickered. For this problem, we divided the coordinates of center by 10 to remove unit part of them as there was no fluctuation in the tenths of the coordinates. Here is the code:

C#
MCvMoments moment = new MCvMoments();                    // a new MCvMoments object

moment = biggestContour.GetMoments();                    // Moments of biggestContour

CvInvoke.cvMoments(biggestContour, ref moment, 0);
        
double m_00 = CvInvoke.cvGetSpatialMoment(ref moment, 0, 0);
double m_10 = CvInvoke.cvGetSpatialMoment(ref moment, 1, 0);
double m_01 = CvInvoke.cvGetSpatialMoment(ref moment, 0, 1);

int current_X = Convert.ToInt32(m_10 / m_00) / 10;       // X location of centre of contour              
int current_Y = Convert.ToInt32(m_01 / m_00) / 10;       // Y location of center of contour

We also faced another problem, the center shifted when the palm was open and closed. For this problem, we placed the conditions that mouse cursor position will only be moved if the palm is closed, i.e., finger count is zero, else mouse will remain where it was. The code below also shows mouse click event:

C#
// move cursor to center of contour only if Finger count is 1 or 0
// i.e. palm is closed

if (Finger_num == 0 || Finger_num == 1)
{
    Cursor.Position = new Point(current_X * 20, current_Y * 20);   
}

// Leave the cursor where it was and Do mouse click, if finger count >= 4

if (Finger_num >= 4)
{
    DoMouseClick();                     // function clicks mouse left button
}  

The function definition for DoMouseClick() is as follows, however system32 function mouse_event() is now obsolete but we used it anyway.

C#
// function for mouse clicks

[DllImport("user32.dll")]
static extern void mouse_event(uint dwFlags, uint dx, uint dy, uint dwData,
   int dwExtraInfo);

private const int MOUSEEVENTF_LEFTDOWN = 0x02;          // mouse left button pressed 
private const int MOUSEEVENTF_LEFTUP = 0x04;            // mouse left button unpressed
private const int MOUSEEVENTF_RIGHTDOWN = 0x08;         // mouse right button pressed
private const int MOUSEEVENTF_RIGHTUP = 0x10;           // mouse right button unpressed

//this function will click the mouse using the parameters assigned to it
public void DoMouseClick()
{
    //Call the imported function with the cursor's current position
    uint X = Convert.ToUInt32(Cursor.Position.X);
    uint Y =Convert.ToUInt32(Cursor.Position.Y);
    mouse_event(MOUSEEVENTF_LEFTDOWN | MOUSEEVENTF_LEFTUP, X, Y, 0, 0);
}     

That's all about using the code, hope it helps you. There are many things yet to be done to perfect this application. We hope suggestions from viewers may help us improve it.

Note: We have not included emguCV DLLs in the attached file, one must install emguCV and copy all DLLs to run the project.

Run this Application on Your PC

There is a three step process to run this application on your PC:

  1. Download and install emguCV library from here.
  2. Copy all files from "C:\Emgu\emgucv-windows-x86 2.4.0.1717\bin\x86\" into your system32 folder.
  3. Download and install the setup of application from the following link:

Points of Interest

As Windows 8 is now available in the market, and it provides much of a tablet touch to the PC, we are thinking of improving this application to be able to use hand movements to provide touch effect to a laptop.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Instructor / Trainer Institute of Industrial Electronics Engineering
Pakistan Pakistan
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Written By
Student Institute of Industrial Electronic Engineering
Pakistan Pakistan
An undergraduate student at Institute of Industrial Electronic Engineering (IIEE).

Written By
Software Developer institute of industrial electronics engineering
Pakistan Pakistan
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Written By
Student
Pakistan Pakistan
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
GeneralMy vote of 4 Pin
maq_rohit8-Jan-13 16:37
professionalmaq_rohit8-Jan-13 16:37 
GeneralMy vote of 5 Pin
Hammad Raza24-Dec-12 21:53
professionalHammad Raza24-Dec-12 21:53 
GeneralMy vote of 5 Pin
hzq0124-Dec-12 15:13
hzq0124-Dec-12 15:13 
GeneralMy vote of 5 Pin
Ahsan Murshed19-Dec-12 16:39
Ahsan Murshed19-Dec-12 16:39 
QuestionSuper Application! Pin
Jean Paul V.A18-Dec-12 4:59
Jean Paul V.A18-Dec-12 4:59 
GeneralMy vote of 5 Pin
soulprovidergr18-Dec-12 2:31
soulprovidergr18-Dec-12 2:31 
GeneralMy vote of 5 Pin
wsc091816-Dec-12 18:43
wsc091816-Dec-12 18:43 
GeneralMy vote of 5 Pin
Ștefan-Mihai MOGA14-Dec-12 5:09
professionalȘtefan-Mihai MOGA14-Dec-12 5:09 
This is a great inspiring article. I am pretty much pleased with your good work. You put really very helpful information. Keep it up once again.
GeneralMy vote of 5 Pin
Farhan Ghumra11-Dec-12 1:41
professionalFarhan Ghumra11-Dec-12 1:41 
GeneralMy vote of 5 Pin
Savalia Manoj M10-Dec-12 20:54
Savalia Manoj M10-Dec-12 20:54 
GeneralMy vote of 5 Pin
bubifengyun9-Dec-12 20:36
bubifengyun9-Dec-12 20:36 
QuestionCould you add your opencv.dll in your project to download Pin
bubifengyun9-Dec-12 20:35
bubifengyun9-Dec-12 20:35 
AnswerRe: Could you add your opencv.dll in your project to download Pin
csharpbd18-Aug-14 22:49
professionalcsharpbd18-Aug-14 22:49 
GeneralMy vote of 5 Pin
Md. Marufuzzaman9-Dec-12 1:08
professionalMd. Marufuzzaman9-Dec-12 1:08 
QuestionMaybe I didn't read it carefully... Pin
Cristian Amarie8-Dec-12 5:45
Cristian Amarie8-Dec-12 5:45 
QuestionMy vote of 5 Pin
Muhammad Shahid Farooq28-Nov-12 21:36
professionalMuhammad Shahid Farooq28-Nov-12 21:36 
GeneralMy vote of 5 Pin
Marco Bertschi27-Nov-12 4:22
protectorMarco Bertschi27-Nov-12 4:22 
Questionregarding openCV dlls Pin
kumarjammula24-Nov-12 4:38
kumarjammula24-Nov-12 4:38 
AnswerRe: regarding openCV dlls Pin
Anas Hashmi24-Nov-12 4:48
Anas Hashmi24-Nov-12 4:48 
GeneralRe: regarding openCV dlls Pin
Anas Hashmi24-Nov-12 5:30
Anas Hashmi24-Nov-12 5:30 
GeneralRe: regarding openCV dlls Pin
kumarjammula24-Nov-12 6:21
kumarjammula24-Nov-12 6:21 
QuestionemguCV Pin
gil6524-Nov-12 3:50
gil6524-Nov-12 3:50 
AnswerRe: emguCV Pin
Anas Hashmi24-Nov-12 4:39
Anas Hashmi24-Nov-12 4:39 
GeneralMy vote of 4 Pin
Zaid Pirwani24-Nov-12 3:28
Zaid Pirwani24-Nov-12 3:28 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.