Click here to Skip to main content
15,867,568 members
Articles / Programming Languages / C#

Computer Vision: Virtual Buttons

Rate me:
Please Sign up or sign in to vote.
4.91/5 (50 votes)
2 Mar 2010CPOL3 min read 78.3K   6K   133   31
In this article, we will put together code to control three virtual buttons in a webcam view.

Image 1

Introduction

Wouldn't it be great if you could control your computer with your hands? In this article, I will show you a small application I created to control the Windows Media Player with my hand motion and an ordinary web-cam. You can see the picture above; this application creates three "hot-spots" in the web-cam view, and you can assume that these hot-spots are virtual buttons, and they get activated when you make a click movement in them. I've programmed each of the hot-spots to control a particular button in Windows Media Player; the one at the top (blue) is for play/pause, and the other two are for next/previous songs. Check out this video to see my application in action.

Creating click-able virtual buttons

The criteria for clicking of a "hot-spot" is the increase in motion to a certain threshold; when it crosses its limit, then a click event is raised for a particular "hot-spot", which could be programmed to do further actions (like controlling the Media Player). Basically, I process each frame of my web-cam to check for motion levels in each of "hot-spots", and whenever it sees a motion is continuously increasing its threshold value in 10 to 20 frames, then it raises an event. Pretty simple!

To detect motion, I used a fairly simple algorithm of subtracting two images. Every image is made of three layers: red, blue, and green. Thus, each of the pixels in an image holds three values corresponding to its RGB. But, what about gray-scale images? Do they have three layers too? Well, yes, but the value of R, G, and B are the same for a particular pixel in a gray-scaled image. Imagine that there is an image of 2X2 pixels:

{1R 2G 3B, 4R 5G 6B} 
{7R 8G 9B, 2R 3G 4B}

The above 2X2 matrix is for a bitmap image where each pixel consists of RGB. Now, to make a gray-scale of the above image, we will take the average of the RGB values of each pixel and assign this average value to the RGB of the pixel. So, our gray-scaled image would be:

{(1+2+3)/3R (1+2+3)/3G (1+2+3)/3B, (4+5+5)/3R (4+5+5)/3G (4+5+5)/3B}
{(7+8+9)/3R (7+8+9)/3G (7+8+9)/3B, (2+3+4)/3R (2+3+4)/3G (2+3+4)/3B}

We can see that the gray-scaled images have the same RGB value for each pixel. So, I preferred to convert the web-cam images to gray-scale and then simply subtract it from the previous images using matrix algebra.

C#
public ImageMatrix(Bitmap image)//constructor
{
    this.width = image.Width;
    this.height = image.Height;

    byte[,] function = new byte[width, height];
    //define bounds of function

    //unsafe bitmap to iterate through the image
    unSafeBitmap usb = new unSafeBitmap(image);
    usb.LockImage();

    for (int i = 0; i < width; i++)
    {
        for (int j = 0; j < height; j++)
        {
            /* Reading average (gray scale) 
               value for each of the pixel
             * and storing it in function*/
            this.function[i, j] = usb.GetAverage(i, j);
        }
    }
}

The above code snippet shows that it is easy to get a matrix of form byte[,] from a bitmap image. Once you have two of such matrices from consecutive web-cam frames, you could subtract them to find the matrix that shows the motion in each of the pixels. Further, you can make an image from this byte[,] using the code snippet shown below:

C#
public Bitmap ToImage()//to convert imagematrix to 
{
    Bitmap image = new Bitmap(width, height);

    unSafeBitmap usb = new unSafeBitmap(image);
    //making object for unsafe bitmapping
    usb.LockImage();
    
    //making a grayscale image according to function array
    for(int i=0;i<width;i++)
    {
        for(int j=0;j<height;j++)
        {
            //writign each pixel on image bitmap.
            byte Value = function[i, j];
            Color c = Color.FromArgb(Value, Value, Value);
            //gray scale image

            usb.SetPixel(i, j, c);
        }
    }

    //unlock image before returning 
    usb.UnlockImage();

    return image;
}

Once you get the motion levels in each of the "hot-spots", add code to raise an event if it increases beyond a certain value, and that's it!

Fast image processing

Even in my earlier articles, I've used a class for faster image processing in C#. If you are not aware of it, then you should check my previous article, "Computer Vision - Decoding a Morse Code Flashing LED".

Using the program

The program was designed and tested on my web-cam; you can make it work perfectly fine on your computer by adjusting a few settings. You can see the settings in the screenshot below:

Image 2

In the sample program I've provided with this article, there are three buttons:

  • Top: Play/Pause
  • Left: Previous song
  • Right: Next song

You can also easily modify the code to control whatever you wish, or you could even add more virtual buttons! If you have questions about the program's functioning, then leave your comments here.

You can visit my blog for more information about this program.

Have fun!

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Canada Canada
cat me.txt

Comments and Discussions

 
Buggetting invalid operation exception Pin
Member 1148510828-Feb-15 0:40
Member 1148510828-Feb-15 0:40 
GeneralRe: getting invalid operation exception Pin
Deepak Swamy15-Mar-16 20:33
Deepak Swamy15-Mar-16 20:33 
QuestionHi. Great work here! Pin
three_xtreme26-Feb-13 0:25
three_xtreme26-Feb-13 0:25 
QuestionOnly one at a time Pin
sasa618-Dec-12 21:55
sasa618-Dec-12 21:55 
Question(Interaction with Computer using Webcam) Pin
alshabi12-Jun-12 6:01
alshabi12-Jun-12 6:01 
GeneralMy vote of 5 Pin
marekbar2186-Dec-11 12:14
marekbar2186-Dec-11 12:14 
GeneralSource of project Pin
saavedranet23-Mar-11 9:20
saavedranet23-Mar-11 9:20 
Questionreading error & adding a new button?? Pin
Saket Vyas26-Jan-11 12:32
Saket Vyas26-Jan-11 12:32 
QuestionMotion reading error?? Pin
George_micom14-Oct-10 23:51
George_micom14-Oct-10 23:51 
GeneralHi Pin
angel_100359-Apr-10 5:19
angel_100359-Apr-10 5:19 
GeneralRe: Hi Pin
Shivamkalra9-Apr-10 9:23
Shivamkalra9-Apr-10 9:23 
GeneralHi Shivam kalra Pin
angel_100358-Apr-10 7:12
angel_100358-Apr-10 7:12 
GeneralRe: Hi Shivam kalra Pin
Shivamkalra8-Apr-10 13:49
Shivamkalra8-Apr-10 13:49 
GeneralRe: Hi Shivam kalra Pin
Khaniya6-Oct-10 0:04
professionalKhaniya6-Oct-10 0:04 
Generalversion of windows media player Pin
saavedranet11-Mar-10 5:33
saavedranet11-Mar-10 5:33 
GeneralRe: version of windows media player Pin
shivamkalra11-Mar-10 13:16
shivamkalra11-Mar-10 13:16 
GeneralRe: version of windows media player Pin
saavedranet12-Mar-10 8:55
saavedranet12-Mar-10 8:55 
GeneralRe: version of windows media player Pin
Adrabi Abderrahim26-Sep-10 20:24
Adrabi Abderrahim26-Sep-10 20:24 
GeneralHelp.. Regarding COLOR recognition. Pin
shirishyc4-Mar-10 6:17
shirishyc4-Mar-10 6:17 
GeneralRe: Help.. Regarding COLOR recognition. Pin
Paulo Zemek4-Mar-10 7:14
mvaPaulo Zemek4-Mar-10 7:14 
GeneralRe: Help.. Regarding COLOR recognition. Pin
shirishyc4-Mar-10 7:17
shirishyc4-Mar-10 7:17 
GeneralMinor error... [modified] Pin
Paulo Zemek3-Mar-10 8:10
mvaPaulo Zemek3-Mar-10 8:10 
GeneralRe: Minor error... Pin
shivamkalra3-Mar-10 8:57
shivamkalra3-Mar-10 8:57 
GeneralRe: Minor error... Pin
shivamkalra3-Mar-10 20:22
shivamkalra3-Mar-10 20:22 
GeneralRe: Minor error... Pin
Paulo Zemek4-Mar-10 2:21
mvaPaulo Zemek4-Mar-10 2:21 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.