Click here to Skip to main content
Click here to Skip to main content
Go to top

Capturing Live-video from Web-camera on Windows 7 and Windows 8

, 23 May 2014
Rate this:
Please Sign up or sign in to vote.
Simple lib for capturing live-video from web-camera by using Media Foundation

Introduction

About one year ago, I wrote a short article about using Media Foundation API for capturing live-video from web-camera - old article. Many developers paid attention to the article and gave some advice about it. I was surprised that this code had been started to use in project OpenCV. It had inspired me to review the code and I found that it has poor quality of coding. As a result, I had decided to rewrite the project, but after some time I made a decision to write a new article, because the new code is different from the old one. I think that it would be better to have two different articles with two different solutions of the problem.

Background

The previous article about using Media Foundation for capturing live-video from web-camera was developed in target to keep interface of the other library - videoInput. However, now I think that it would be better to use another interface for more flexible solution. I have decided to include into the solution options for setting image format of the uncompressed image data, interface for processing of removing video device by object way, setting for synchronised and asynchronised reading mode, encapsulation requests to the library into object instances and result-code.

For development of the most stable code, I have decided to use Test-Driven Development strategy. As a result, I developed 45 tests for testing of classes and code. Many objects of the Media Foundation are tested many times in different ways. It allows to develop more clear and stable code, which can be easily verified, and it includes into the project united code-result enumeration for checking result of execution of the code which has 51 items. Also I removed the thread which was used in the previous article and simplified the write-read conveyor, and it made code more stable.

The new project is very different from the previous one and I think it will help to resolve the problems of other developers.

Using the Code

The library videoInput is written on Visual Studio 2012 - videoInputVS2012-static.zip . It can be used like a static library and it is enough to include into the new project only videoInput.lib and videoInput.h. Also code of this project can be downloaded - videoInputVS2012-Source and it can be included into any project.

The project includes 15 classes and interfaces:

  • videoInput - is class-singleton. This class is made as a singleton which makes managing of resources easy.
  • MediaFoundation - is a class-singleton which manages the allocation and releasing of resources of Media Foundation. Almost all calls of Media Foundation functions are made from this class and results of these calls are controlled in this class.
  • VideoCaptureDeviceManager - is class-singleton which manages the allocation, access and releasing of the video devices.
  • VideoCaptureDevice - is a class which is inherited from IUnknow interface. It allows to use smart pointer - CComPtr, for control live-time of this class. This class allows control of the selected video capture device.
  • VideoCaptureSession - is a class which is inherited from IMFAsyncCallback. This class is used for processing events which are generated by MediaEventGenerator - IMFMediaSession. This class processes event of the video capture device and starts capture of video.
  • VideoCaptureSink - is a class which is inherited from IMFSampleGrabberSinkCallback. This class is used for getting raw data and writing it into the buffer.
  • IWrite - is a class-interface for using different types of buffers by the only one way in VideoCaptureSink for writing data into the buffers.
  • IRead - is a class-interface for using different types of buffers by the only one way in VideoCaptureDevice for reading raw data from buffers.
  • IReadWriteBuffer - is a class-interface which is inherited from IRead, IWrite and IUnknown. This class is the base class for different types of buffers and can be used with a smart pointer CComPtr.
  • ReadWriteBufferRegularAsync - is a class which is inherited from IReadWriteBuffer. This class uses critical section for blocking access to writing-reading of data by user thread and Media Foundation inner thread. However, the user thread checks flag readyToRead before entering into the critical section, if it is not ready, the user thread goes out buffer without reading data. As a result, the user thread is not blocked by writing Media Foundation inner thread.
  • ReadWriteBufferRegularSync - is a class which is inherited from public IReadWriteBuffer. This class uses critical section for blocking access to writing-reading of data by user thread and Media Foundation inner thread. However, the user thread is blocked by WaitForSingleObject before entering into critical section. As a result, the user thread waits the object event about 1 second from Media Foundation inner thread and when it comes the user thread starts reading from buffer, but if event does not come the user thread leaves the buffer with code state - READINGPIXELS_REJECTED_TIMEOUT.
  • ReadWriteBufferFactory - is class-singleton which produces ReadWriteBufferRegularAsync or ReadWriteBufferRegularSync buffers.
  • FormatReader - is a class for reading data of MediaType format.
  • DebugPrintOut - is the class for printing text into console.
  • CComMassivPtr - is a template class for working with massive of objects with IUnknow interface. This class allows to encapsulate calling function SafeRelease() and control releasing of these objects.
It is enough to use the file videoInput.h as the interface of the library. Listing of it is presented below:
//

#include <string>
#include <vector>
#include <guiddef.h>

using namespace std;

// Structure of info MediaType 
struct MediaType
{
    unsigned int MF_MT_FRAME_SIZE;

    unsigned int height;

    unsigned int width;
            
    unsigned int MF_MT_YUV_MATRIX;
    
    unsigned int MF_MT_VIDEO_LIGHTING;
    
    unsigned int MF_MT_DEFAULT_STRIDE;
    
    unsigned int MF_MT_VIDEO_CHROMA_SITING;
    
    GUID MF_MT_AM_FORMAT_TYPE;
    
    wstring MF_MT_AM_FORMAT_TYPEName;
    
    unsigned int MF_MT_FIXED_SIZE_SAMPLES;
    
    unsigned int MF_MT_VIDEO_NOMINAL_RANGE;

    float MF_MT_FRAME_RATE_RANGE_MAX;

    float MF_MT_FRAME_RATE;

    float MF_MT_FRAME_RATE_RANGE_MIN;

    float MF_MT_PIXEL_ASPECT_RATIO;
    
    unsigned int MF_MT_ALL_SAMPLES_INDEPENDENT;
    
    unsigned int MF_MT_SAMPLE_SIZE;
    
    unsigned int MF_MT_VIDEO_PRIMARIES;
    
    unsigned int MF_MT_INTERLACE_MODE;

    GUID MF_MT_MAJOR_TYPE;
    
    wstring MF_MT_MAJOR_TYPEName;
    
    GUID MF_MT_SUBTYPE;
    
    wstring MF_MT_SUBTYPEName;    
};

// Stream structure
struct Stream
{
    std::vector<MediaType> listMediaType;
};

// Device info structure
struct Device
{
    wstring friendlyName;

    wstring symbolicName;

    std::vector<Stream> listStream;
};

// Structure for collecting info about one parametr of current video device
struct Parametr
{
    long CurrentValue;

    long Min;
    
    long Max;
    
    long Step;
    
    long Default; 
    
    long Flag;

    Parametr();
};

// Structure for collecting info about 17 parametrs of current video device
struct CamParametrs
{
        Parametr Brightness;
        Parametr Contrast;
        Parametr Hue;
        Parametr Saturation;
        Parametr Sharpness;
        Parametr Gamma;
        Parametr ColorEnable;
        Parametr WhiteBalance;
        Parametr BacklightCompensation;
        Parametr Gain;


        Parametr Pan;
        Parametr Tilt;
        Parametr Roll;
        Parametr Zoom;
        Parametr Exposure;
        Parametr Iris;
        Parametr Focus;
};


// Structure for defining color format for out data
struct CaptureVideoFormat
{
    enum VideoFormat
    {
        RGB24 = 0,
        RGB32 = 1,
        AYUV = 2
    };
};

// structure of defining of the type callback event
struct StopCallbackEvent
{
    enum CallbackEvent
    {
        STOP = 0,
        CAPTUREDEVICEREMOVED = 1
    };
};

// Interface for processing callback of the emergency stop - removing video device.
class IStopCallback
{
public:
    virtual void Invoke(StopCallbackEvent::CallbackEvent callbackEvent) = 0;
};

// Structure for defining mode of reading pixels
struct ReadMode
{
    enum Read
    {
        ASYNC = 0,
        SYNC = 1
    };
};

// Settings for setting up of video device.
struct DeviceSettings
{
    wstring symbolicLink;

    unsigned int indexStream;

    unsigned int indexMediaType;
    
};

// Settings for setting up mode of capturing raw pixels
struct CaptureSettings
{
    CaptureVideoFormat::VideoFormat videoFormat;

    IStopCallback *pIStopCallback;

    ReadMode::Read readMode;
};

// Structure for controlling reading raw pixels.
struct ReadSetting
{
    std::wstring symbolicLink;

    unsigned char *pPixels;
};

// Structure for controlling parameters of video device
struct CamParametrsSetting
{
    std::wstring symbolicLink;

    CamParametrs settings;
};

// Structure for defining result code of the working program
struct ResultCode
{
    enum Result
    {
        OK = 0,
        UNKNOWN_ERROR = 1,
        MEDIA_FOUNDATION_INITIALIZECOM_ERROR = 2,
        MEDIA_FOUNDATION_INITIALIZEMF_ERROR = 3,
        MEDIA_FOUNDATION_SHUTDOWN_ERROR = 4,
        MEDIA_FOUNDATION_ENUMDEVICES_ERROR = 5,
        MEDIA_FOUNDATION_CREATEATTRIBUTE_ERROR = 6,
        MEDIA_FOUNDATION_READFRIENDLYNAME_ERROR = 7,
        MEDIA_FOUNDATION_READSYMBOLICLINK_ERROR = 8,
        MEDIA_FOUNDATION_GETDEVICE_ERROR = 9,
        MEDIA_FOUNDATION_createPresentationDescriptor_ERROR = 10,
        MEDIA_FOUNDATION_GETTHEAMOUNTOFSTREAMS_ERROR = 11,
        MEDIA_FOUNDATION_GETSTREAMDESCRIPTORBYINDEX_ERROR = 12,
        MEDIA_FOUNDATION_ENUMMEDIATYPE_ERROR = 13,
        VIDEOCAPTUREDEVICEMANAGER_GETLISTOFDEVICES_ERROR = 14,
        MEDIA_FOUNDATION_SETSYMBOLICLINK_ERROR = 15,
        MEDIA_FOUNDATION_SETCURRENTMEDIATYPE_ERROR = 16,
        MEDIA_FOUNDATION_GETCURRENTMEDIATYPE_ERROR = 17,
        MEDIA_FOUNDATION_SELECTSTREAM_ERROR = 18,
        MEDIA_FOUNDATION_CREATESESSION_ERROR = 19,
        MEDIA_FOUNDATION_CREATEMEDIATYPE_ERROR = 20,
        MEDIA_FOUNDATION_SETGUID_ERROR = 21,
        MEDIA_FOUNDATION_SETUINT32_ERROR = 22,
        MEDIA_FOUNDATION_CREATESAMPLERGRABBERSINKACTIVE_ERROR = 23,
        MEDIA_FOUNDATION_CREATETOPOLOGY_ERROR = 24,
        MEDIA_FOUNDATION_CREATETOPOLOGYNODE_ERROR = 25,
        MEDIA_FOUNDATION_SETUNKNOWN_ERROR = 26,
        MEDIA_FOUNDATION_SETOBJECT_ERROR = 27,
        MEDIA_FOUNDATION_ADDNODE_ERROR = 28,
        MEDIA_FOUNDATION_CONNECTOUTPUTNODE_ERROR = 29,
        MEDIA_FOUNDATION_SETTOPOLOGY_ERROR = 30,
        MEDIA_FOUNDATION_BEGINGETEVENT_ERROR = 31,
        VIDEOCAPTUREDEVICEMANAGER_DEVICEISSETUPED = 32,
        VIDEOCAPTUREDEVICEMANAGER_DEVICEISNOTSETUPED = 33,
        VIDEOCAPTUREDEVICEMANAGER_DEVICESTART_ERROR = 34,
        VIDEOCAPTUREDEVICE_DEVICESTART_ERROR = 35,
        VIDEOCAPTUREDEVICEMANAGER_DEVICEISNOTSTARTED = 36,
        VIDEOCAPTUREDEVICE_DEVICESTOP_ERROR = 37,
        VIDEOCAPTURESESSION_INIT_ERROR = 38,
        VIDEOCAPTUREDEVICE_DEVICESTOP_WAIT_TIMEOUT = 39,
        VIDEOCAPTUREDEVICE_DEVICESTART_WAIT_TIMEOUT = 40,
        READINGPIXELS_DONE = 41,
        READINGPIXELS_REJECTED = 42,
        READINGPIXELS_MEMORY_ISNOT_ALLOCATED = 43,
        READINGPIXELS_REJECTED_TIMEOUT = 44,
        VIDEOCAPTUREDEVICE_GETPARAMETRS_ERROR = 45,
        VIDEOCAPTUREDEVICE_SETPARAMETRS_ERROR = 46,
        VIDEOCAPTUREDEVICE_GETPARAMETRS_GETVIDEOPROCESSOR_ERROR = 47,
        VIDEOCAPTUREDEVICE_GETPARAMETRS_GETVIDEOCONTROL_ERROR = 48,
        VIDEOCAPTUREDEVICE_SETPARAMETRS_SETVIDEOCONTROL_ERROR = 49,
        VIDEOCAPTUREDEVICE_SETPARAMETRS_SETVIDEOPROCESSOR_ERROR = 50
    };

};

class videoInput
{
public:

    // get static instance of the singleton
    static videoInput& getInstance();

    // filling list of the video devices. 
    ResultCode::Result getListOfDevices(vector<Device> &listOfDevices);

    // setting up selected video device and capture settings.
    ResultCode::Result setupDevice(DeviceSettings deviceSettings, 
                                   CaptureSettings captureSettings);

    // closing selected video device.
    ResultCode::Result closeDevice(DeviceSettings deviceSettings);

    // closing all setup video devices
    ResultCode::Result closeAllDevices();

    // reading raw data of pixels
    ResultCode::Result readPixels(ReadSetting readSetting);

    // getting parametrs of video device
    ResultCode::Result getParametrs(CamParametrsSetting &parametrs);

    // setting parametrs of video device
    ResultCode::Result setParametrs(CamParametrsSetting parametrs);

    // Setting of the state of outprinting info on consol
    ResultCode::Result setVerbose(bool state);

private:
    videoInput(void);
    ~videoInput(void);
    videoInput(const videoInput&);
    videoInput& operator=(const videoInput&);
}; 

The interface of library has become simple and some work with raw data has become the duty of the developer. Methods of the videoInput have the next purpose:

  • getListofDevices - Method for filling of the list of the active video devices. The fact is that in the old project, this list was filled at the time of initialisation and did not change. Now, calling this method will generate a new list of the active video devices and can be changed. Each device includes strings friendlyName, symbolicLink and list of streams. String friendlyName is used for presenting "readable" name of the device. String symbolicLink is the unique name for management of the device.
  • setupDevice - Method for setting up and starting capture from device. It has two arguments: deviceSettings and captureSettings. The first is used for setting device into the chosen mode by symbolicLink, indexStream and indexMediaType. The second is used for setting capturing mode. It includes videoFormat for choosing the format of the raw image data - RGB24, RGB32 and AYUV; pIStopCallback is a pointer on the interface of callback class that is executed in the case of the removing video capture device; readMode is enumeration of the type buffer with synchronised and asynchronised reading raw data.
  • closeDevice - Method is used for stopping and closing the selected device. This device is defined by symbolicName.
  • closeAllDevices - Method for closing all setup devices.
  • readPixels - Method for reading data from buffer into the pointer. The argument readSetting has symbolicLink for access to the device and pPixels for keeping pointer on the used raw image data.
  • getParametrs - Method for getting parameters of video capture device.
  • setParametrs - Method for setting parameters of video capture device.
  • setVerbose - Method for setting mode of the printing information on console.

All these methods return item from ResultCode::Result enumeration. It allows to control the result execution of each method and get information about almost all processes in the code.

I would like to mark three special features of this library:

  1. The resolution of the captured image is defined by selecting appropriate MediaType of the device. In the old article, the resolution is set by manually. However, it leads to a mistake that it is possible to set any resolution. I have decided that the new solution should present more clearly the limits of the video capture devices.
  2. It is possible to chose one of three colour formats for captured image - RGB24, RGB32 and AYUV. It allows to select the appropriate format. Colour formats RGB24 and RGB32 have the same appearance, but the second is more suitable for processing data on the modern processors with aligning(16).
  3. Reading from buffer can be done in one of two ways - synchrony and asynchrony. This choice is made by setting mode readMode. In the mode ReadMode::ASYNC reading from the buffer is not blocked and if the data is not ready, the method readPixels() returns result-code ResultCode::READINGPIXELS_REJECTED. In the mode ReadMode::SYNC reading from the buffer is blocked and if the data is not ready, the method readPixels() blocks the user thread for 1 second and after returns result-code ResultCode::READINGPIXELS_REJECTED_TIMEOUT.

The next listing of code presents using this library with OpenCV for capturing live-video from web-camera.

#include "stdafx.h"
#include "../videoInput/videoInput.h"
#include "include\opencv2\highgui\highgui_c.h"
#include "include\opencv2\imgproc\imgproc_c.h"

#pragma comment(lib, "../Debug/videoInput.lib")
#pragma comment(lib, "lib/opencv_highgui248d.lib")
#pragma comment(lib, "lib/opencv_core248d.lib")

int _tmain(int argc, _TCHAR* argv[])
{
    using namespace std;        
    
    vector<Device> listOfDevices;

    ResultCode::Result result = videoInput::getInstance().getListOfDevices(listOfDevices);
  
    DeviceSettings deviceSettings;

    deviceSettings.symbolicLink = listOfDevices[0].symbolicName;

    deviceSettings.indexStream = 0;

    deviceSettings.indexMediaType = 0;            

    CaptureSettings captureSettings;
    
    captureSettings.pIStopCallback = 0;

    captureSettings.readMode = ReadMode::SYNC;

    captureSettings.videoFormat = CaptureVideoFormat::RGB32;


    MediaType MT = listOfDevices[0].listStream[0].listMediaType[0];

    cvNamedWindow ("VideoTest", CV_WINDOW_AUTOSIZE);
                
    CvSize size = cvSize(MT.width, MT.height);
 

    IplImage* frame;
 
    frame = cvCreateImage(size, 8,4);

    
    
    ReadSetting readSetting;

    readSetting.symbolicLink = deviceSettings.symbolicLink;

    readSetting.pPixels = (unsigned char *)frame->imageData;

    result = videoInput::getInstance().setupDevice(deviceSettings, captureSettings);

    while(1)
    {
        ResultCode::Result readState = videoInput::getInstance().readPixels(readSetting);

        if(readState == ResultCode::READINGPIXELS_DONE)
        {            
            cvShowImage("VideoTest", frame);
        }
        else
            break;
                    
        char c = cvWaitKey(33);
 
        if(c == 27) 
            break;
    }

    result = videoInput::getInstance().closeDevice(deviceSettings);
    

    return 0;
}

At the end, I would like to say that in this project, I removed the code for reordering colour BGR to RGB and vertical flipping. The fact is that these specific features of the Windows platform are compensated in many different ways - for example OpenCV library process in BGR colour order by default, DirectX and OpenGL support texturing BGR and RGB, and vertical flipping. I have decided that the developer can write the code for reordering colour and vertical flipping for own purpose.

I published the code of the project on the Git repository of this site and anyone can get a clone of it by using link videoInput.

Points of Interest

I developed this project on the base of the TDD strategy. I think this will help other developers to resolve some problems with this code.

History

This project is situated on Git Repos CodeProject: videoInput.

This article is based on the old article.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Evgeny Pereguda
Software Developer
Australia Australia
No Biography provided

Comments and Discussions

 
QuestionAccess violation exception PinmemberMember 1081789621-Jul-14 4:26 
AnswerRe: Access violation exception PinmemberEvgeny Pereguda23-Jul-14 22:53 
GeneralRe: Access violation exception PinmemberMember 1081789624-Jul-14 0:28 
GeneralRe: Access violation exception PinmemberEvgeny Pereguda24-Jul-14 1:51 
GeneralRe: Access violation exception PinmemberMember 1081789627-Jul-14 11:17 
Questionwhy not see network issue? note that it is web-camera [modified] PinmemberMember 109269395-Jul-14 19:04 
AnswerRe: why not see network issue? note that it is web-camera PinmemberEvgeny Pereguda5-Jul-14 23:51 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.140916.1 | Last Updated 23 May 2014
Article Copyright 2014 by Evgeny Pereguda
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid