Webcamera, Multithreading and VFW

• Download demo project - 10.7 KB
• Download source files - 18.7 KB

Introduction

There are several ways to grab and process webcamera images: WIA, DirectShow, VFW... There are lots of C# VFW examples on the Internet and most of them use .NET clipboard to transfer each frame's data from buffer to Bitmap-recognizable object. Unfortunately, this makes multithreading unavailable and reduces FPS (frames per second). The native Win32 clipboard and multithreading solve the speed problem, but I thought that it wasn't the most elegant solution and there should be another way to get frames from Avicap. I have referred to MSDN (see VFW link above) and discovered that function callback was available. This article explains, step-by-step, how to capture frames using avicap32.dll (VFW) in a multi-thread environment.

The Idea

This is an approach that you can find in lots of examples from the Web: Create a capture window. Connect the capture window to the device. Set the video format (height and width in pixels). Capture the frame to temporary unreachable buffer. Copy the contents of the video frame buffer and associated palette to the clipboard. Get the frame from the clipboard, converting data to RGB Bitmap. Process Bitmap. Go to Step 4.

This is an approach I worked with: Create a capture window. Connect the capture window to the device. Set the video format (height and width in pixels, bits per frame). Capture the frame to temporary unreachable buffer. Make a callback (make buffer data available in VIDEOHDR structure). Get the frame data, converting data to RGB Bitmap. Process Bitmap. Go to Step 4.

API

[DllImport("avicap32.dll", EntryPoint = "capCreateCaptureWindow")]
static extern int capCreateCaptureWindow(string lpszWindowName,
    int dwStyle, int X, int Y,
    int nWidth, int nHeight, int hwndParent, int nID);

The capCreateCaptureWindow function creates a new window for video stream capturing and returns its handle. This function is called in step 1.

[DllImport("user32", EntryPoint = "SendMessage")]
static extern bool SendMessage(int hWnd, uint wMsg, int wParam, int lParam);

The SendMessage function sends the specified message to a window or windows. It calls the window procedure for the specified window and does not return until the window procedure has processed the message. Note that both wParam and lParam specify additional message-specific information, i.e. numbers, pointers to structures, buffers. SendMessage is overloaded in this project with different lParam types. It is called in steps 2, 3, 4 and 5.

[DllImport("avicap32.dll")]
static extern bool capGetDriverDescription(intwDriverIndex,
    [MarshalAs(UnmanagedType.VBByRefStr)] ref String lpszName, int cbName,
    [MarshalAs(UnmanagedType.VBByRefStr)] ref String lpszVer, int cbVer);

The capGetDriverDescription function retrieves the version description of the capture driver. The wDriverIndex parameter specifies the index of the capture driver. The index can range from 0 through 9.

The structures are: BITMAPINFO, BITMAPINFOHEADER and VIDEOHDR. The VIDEOHDR structure is used by the callback function. It contains the buffered frame data. The BITMAPINFO structure defines the dimensions and color information of a Windows-based, device-independent Bitmap (DIB). The BITMAPINFOHEADER structure contains information about the dimensions and color format of a DIB. In our case, it defines the video format (frame size and bits per frame).

Inside the Code

The main part of the solution is the WebCamera class library, which consists of three classes:

• WebCameraDevice
• WebCameraEventArgs
• WebCameraDeviceManager

WebCameraDevice

public WebCameraDevice(int frameWidth, int frameHeight, int preferredFPS,
    int camID, int parentHwnd)
{
    /*...*/
}

This initializes a new instance of the WebCameraDevice object. Focus of preferredFPS parameter: generally, Web cameras support a maximum of 30 FPS. The maximum FPS I could get on my A4Tech webcam was 20. Also, FPS depends on driver details. For example, enabling flicker slightly reduces FPS. Use the WebCamDeviceManager class to get all available devices and their indices. The camID parameter represents the selected device's index.

public void Start()
{
    /*...*/
    camHwnd = capCreateCaptureWindow("WebCam", 0, 0, 0, frameWidth,
        frameHeight, parentHwnd, camID); // Step 2

    //Try to connect a capture window to a capture driver

    if (SendMessage(camHwnd, WM_CAP_DRIVER_CONNECT, 0, 0))
    {
        //Step 3, fill bitmap structure (see source)
        /*...*/
        // Enables preview mode
        SendMessage(camHwnd, WM_CAP_SET_PREVIEW, 1, 0);
        // Sets the frame display rate in preview mode. 34 ms ~ 29FPS
        SendMessage(camHwnd, WM_CAP_SET_PREVIEWRATE, 34, 0);
        // Sets the format of captured video data.
        SendBitmapMessage(camHwnd, WM_CAP_SET_VIDEOFORMAT,
            Marshal.SizeOf(bInfo), ref bInfo);

        // Multithreading begins here
        frameThread = new Thread(new ThreadStart(this.FrameGrabber));
        bStart = true;       // Flag variable

        frameThread.Priority = ThreadPriority.Lowest;
        frameThread.Start();
    }
        /*...*/
}

The multithreading mechanism in WebCameraDevice consists of the AutoResetEvent object and frame-grabbing worker thread. Setting preferredFPS to 0 allows the user to control the frame capturing process manually. The worker thread waits (WaitOne() is called) until the user calls the AutoResetEvent object's Set() method (WaitHandle receives a signal). Otherwise, WaitOne(..) with the preferredFPSms (1000 / preferredFPS) parameter is called to wait for a defined amount of milliseconds.

After calling the Start() method, the worker thread starts capturing frames to the buffer. The WebCameraDevice object raises the OnCameraFrame event that contains frame data in Bitmap form.

private void FrameGrabber()
{
    while (bStart) // if worker active thread is still required
    {
        /*...*/
        // get the next frame. This is the SLOWEST part of the program
        SendMessage(camHwnd, WM_CAP_GRAB_FRAME_NOSTOP, 0, 0);
        //Make a function callback
        SendHeaderMessage(camHwnd, WM_CAP_SET_CALLBACK_FRAME, 0,
            delegateFrameCallBack);
        /*...*/
    }
}

What happens in this block of code? The bStart variable is a flag that turns to false when the Stop() method is called. While bStart remains true, the WM_CAP_GRAB_FRAME_NOSTOP message fills the frame buffer with a single uncompressed frame from the capture device. Then a callback is made. We use the delegateFrameCallBack variable instead of a direct callback function's name to avoid GC errors. Try replacing delegateFrameCallBack with FrameCallBack (callback function's name) and see what happens. The callback function looks like this:

private void "code-string" name="<span">"FrameCallBack">FrameCallBack(IntPtr hwnd, ref VIDEOHEADER hdr)
{
    if (OnCameraFrame != null)
    {
        Bitmap bmp = new Bitmap(frameWidth, frameHeight, 3 *
            frameWidth, System.Drawing.Imaging.PixelFormat.Format24bppRgb,
            hdr.lpData);
        OnCameraFrame(this, new WebCameraEventArgs(bmp));
    }

    if (preferredFPSms == 0)
    {
        // blocks thread until WaitHandle receives a signal
        autoEvent.WaitOne();
    }
    else
    {
        // blocks thread for preferred milliseconds
        autoEvent.WaitOne(preferredFPSms, false);
    }
}

As you can see, the function contains all Bitmap converting, event raising and WaitHandler operating stuff. That's it! The remaining methods are:

public void Set()
{
    //Send a signal to the current WainHandle and allow blocked worker
    //(FrameGrabber) thread to proceed
    autoEvent.Set();
}

public void Stop()
{
    try
    {
        bStart = false;
        Set();
        SendMessage(camHwnd, WM_CAP_DRIVER_DISCONNECT, 0, 0);
    }
    catch { }
}

public void ShowVideoDialog()
{
    SendMessage(camHwnd, WM_CAP_DLG_VIDEODISPLAY, 0, 0);
}

How Does It Work?

We have a class library with all the necessary Web camera image capturing classes. First of all, we have to get the available VFW devices and display them to the user:

public FormMain()
{
    InitializeComponent();
    WebCameraDeviceManager camManager = new WebCameraDeviceManager();
    // fill combo box with available devices' names
    cmbDevices.Items.AddRange(camManager.Devices);
    // First available video device.
    // I always receive "Microsoft WDM Image Capture (Win32)"
    cmbDevices.SelectedIndex = 0;
}

The start button and the OnCameraFrame event handler's code:

private void btnStart_Click(object sender, EventArgs e)
{
    /*...*/
    camDevice = new WebCameraDevice
        (320, 200, 0, cmbDevices.SelectedIndex, this.Handle.ToInt32());
    // Register for event notification
    camDevice.OnCameraFrame +=
        new WebCameraFrameDelegate(camDevice_OnCameraFrame);
    camDevice.Start();
    /*...*/
}

void camDevice_OnCameraFrame(object sender, WebCameraEventArgs e)
{
    /*...*/
    ImageProcessing.Filters.Flip(e.Frame, false, true); // Explained below
    pictureBox.Image = e.Frame;
    camDevice.Set(); // comment this if prefferedFPS != 0
    /*...*/
}

Have you noticed the prefferedFPS parameter's 0 value in WebCameraDevice's constructor? That's why the Set() method is called in the camDevice_OnCameraFrame event handler. Do you remember what happens inside the camDevice object? If not, check FrameCallBack above.

Unexpected Image Flip

There was an unexpected vertical image flip. I haven't discovered why this happens yet. It happens only in the case of BITMAPINFOHEADER's buffer conversion. Maybe there is a bug in the Bitmap class. To avoid flipping, I've referred to a great Image Processing for Dummies with C# and GDI+ article by Christian Graus. A fast grayscale filter was found on Bob Powell's site.

History

• Release - 12 September, 2007

P.S.

I'd like to ask you to be lenient with the article because it's my first article on The Code Project. Please let me know if you have liked/disliked it or have any questions about it.