Click here to Skip to main content
Click here to Skip to main content
Go to top

Real-time video image processing / frame grabber using a minimalistic approach

, 6 Apr 2010
Rate this:
Please Sign up or sign in to vote.
Probably the simplest possible sample for video image grabber on Windows. Supports AVI, MPG, QTV, Webcam,TVTuner, etc.

Introduction

All my life, I have been possessed with anything that has something to do with robots and computer vision. So lately, I have been looking for some simple code to read images from movies in any format and from the webcam. Here, I want to share my experiences in this API jungle with you. Also, please consider voting using the bar bellow so I get an idea if this information was useful to anyone.

Background

As hard as I was looking, all I found were samples for the VFW interface, which is a practically dead API with a lot of issues. First of all, it's only for AVI files, and most of the codecs (like divx) just don't work in it anyway (those who have tried it knows), and also in some resolutions, on most webcams, it just crashes. There are a few samples for the newer DirectShow API but all the important code is just buried in the zillion unimportant wrappers/support classes/files. Not to mention that the informational value of the samples themselves is lost in often redundant, error handling jungle. And finally, they are based on using/compiling additional MS DirectShow filter samples, like SampleGrabber or its bug fixed version, GrabberSample, making the whole approach non-portable between the SDK versions. They also seem just disappeared from the last SDK and the Web itself. So here is what I think is a much simpler/versatile alternative to those.


New method not requiring DirectShow SDK

Lately I added to article this new method because a lot of people had problems compiling DirectShow SDK related stuff. Most of them just need to get the RGB data and forget the whole DirectShow thing so I got an idea. If we know how data flows inside directshow graph why don't we just redirect particular method to grab data ?
Now in Directshow data can flow either via:

Push method:    Where FILTER1->out pin pushes data to in<-FILTER2 via Receive on his IMemInputPin interface (our Renderer case).
Pull method:      Where second filter in<-FILTER2 asks first source FILTER1->out for more data via SyncReadAlligned method of his IAsyncReader interface.

In our case (last filter renderer) we capture data of Push method. you can find  sample capturing Pull mode in my other article Fun with Google TTS.
I tested Following code on VisualStudio 2008 and 2010 and it compiled without DirectShow SDK just fine.  Also remember that you will probably need to convert received data to RGB if you wish but it's trivial.

The code

#include <windows.h>
#include <dshow.h>

#pragma comment(lib,"Strmiids.lib")

#define DsHook(a,b,c) if (!c##_) { INT_PTR* p=b+*(INT_PTR**)a;   VirtualProtect(&c##_,4,PAGE_EXECUTE_READWRITE,&no);\
                                          *(INT_PTR*)&c##_=*p;   VirtualProtect(p,    4,PAGE_EXECUTE_READWRITE,&no);   *p=(INT_PTR)c; }


// Here you get image video data in buf / len. Process it before calling Receive_ because renderer dealocates it.
HRESULT ( __stdcall * Receive_ ) ( void* inst, IMediaSample *smp ) ; 
HRESULT   __stdcall   Receive    ( void* inst, IMediaSample *smp ) {     
    BYTE*     buf;    smp->GetPointer(&buf); DWORD len = smp->GetActualDataLength();
    HRESULT   ret  =  Receive_   ( inst, smp );   
    return    ret; 
}

int WINAPI WinMain(HINSTANCE inst,HINSTANCE prev,LPSTR cmd,int show){
    HRESULT hr = CoInitialize(0); MSG msg={0}; DWORD no;

    IGraphBuilder*  graph= 0;  hr = CoCreateInstance( CLSID_FilterGraph, 0, CLSCTX_INPROC,IID_IGraphBuilder, (void **)&graph );
    IMediaControl*  ctrl = 0;  hr = graph->QueryInterface( IID_IMediaControl, (void **)&ctrl );

    ICreateDevEnum* devs = 0;  hr = CoCreateInstance (CLSID_SystemDeviceEnum, 0, CLSCTX_INPROC, IID_ICreateDevEnum, (void **) &devs);
    IEnumMoniker*   cams = 0;  hr = devs?devs->CreateClassEnumerator (CLSID_VideoInputDeviceCategory, &cams, 0):0;  
    IMoniker*       mon  = 0;  hr = cams->Next (1,&mon,0);  // get first found capture device (webcam?)    
    IBaseFilter*    cam  = 0;  hr = mon->BindToObject(0,0,IID_IBaseFilter, (void**)&cam);
                               hr = graph->AddFilter(cam, L"Capture Source"); // add web cam to graph as source
    IEnumPins*      pins = 0;  hr = cam?cam->EnumPins(&pins):0;   // we need output pin to autogenerate rest of the graph
    IPin*           pin  = 0;  hr = pins?pins->Next(1,&pin, 0):0; // via graph->Render
                               hr = graph->Render(pin); // graph builder now builds whole filter chain including MJPG decompression on some webcams
    IEnumFilters*   fil  = 0;  hr = graph->EnumFilters(&fil); // from all newly added filters
    IBaseFilter*    rnd  = 0;  hr = fil->Next(1,&rnd,0); // we find last one (renderer)
                               hr = rnd->EnumPins(&pins);  // because data we are intersted in are pumped to renderers input pin 
                               hr = pins->Next(1,&pin, 0); // via Receive member of IMemInputPin interface
    IMemInputPin*   mem  = 0;  hr = pin->QueryInterface(IID_IMemInputPin,(void**)&mem);

    DsHook(mem,6,Receive); // so we redirect it to our own proc to grab image data

    hr = ctrl->Run();   

    while ( GetMessage(   &msg, 0, 0, 0 ) ) {  
        TranslateMessage( &msg );   
        DispatchMessage(  &msg ); 
    }
};

Changing WebCam resolution

Optionally you can add following code to set resolution right after you get capture device output pin. [thx to LightWing in forum below for finding it Wink | ;) ]

   ...
        IPin*           pin  = 0;  hr = pins?pins->Next(1,&pin, 0):0; // via graph->Render
   
	IAMStreamConfig* cfg = 0;  hr = pin->QueryInterface( IID_IAMStreamConfig, (void **)&cfg);  // (Those are optional steps to set better resolution)
	int       sz,max_res = 0;  hr = cfg->GetNumberOfCapabilities(&max_res, &sz); VIDEO_STREAM_CONFIG_CAPS cap[2]; // find last =
	AM_MEDIA_TYPE*  fmt  = 0;  hr = cfg->GetStreamCaps(max_res-1, &fmt, (BYTE*)cap); 	// max supported resolution (cap contains res x and y sizes)
                                   hr = cfg->SetFormat(fmt); // and set it to device before capture starts
   ...


Previous method requiring DirectShow SDK:

Starting with DirectShow

Most users who for the first time try to compile any DirectShow sample from web will spend 90% of time searching for the SDK itself (as for now it has been moved to the Platform SDK from the DirectX SDK, so be sure to get latest) and then spend the rest of the day solving the undefined variables (add YOUR_PLATFORMSDK_DIR\Samples\Multimedia\DirectShow\BaseClasses to the compiler include paths) and unresolved externals (add strmiids.lib, winmm.lib, and strmbasd.lib (strmbase.lib for the release build) to the linker input). But the latter two libs are not redistributed by Microsoft and must usually be built manually (just and run vcvars32 and nmake (nmake nodebug=1 for release lib) in the include directory mentioned above and copy there the libs from the subdirectories created by nmake).

Using the code

So here is the sample. Just create an empty Windows C++ application in whatever development environment you have (Disable UNICODE in VC++ or you get linking errors), and copy/paste it there. I stripped it to a bare minimum to run, so please forgive its compressed formatting for the unimportant code parts. And as usual, you can find information on any used API simply by searching for its name in Google and using the first found (usually MSDN) page.

#include <windows.h>
#include <dshow.h>
#include <streams.h>

int w,h; HWND hwnd; MSG msg = {0}; BITMAPINFOHEADER bmih={0}; 

struct __declspec(  uuid("{71771540-2017-11cf-ae26-0020afd79767}")  ) CLSID_Sampler;

struct Sampler : public CBaseVideoRenderer {
    Sampler( IUnknown* unk, HRESULT *hr ) : CBaseVideoRenderer(__uuidof(CLSID_Sampler), NAME("Frame Sampler"), unk, hr) {};
    HRESULT CheckMediaType(const CMediaType *media ) {    
        VIDEOINFO* vi; if(!IsEqualGUID( *media->Subtype(), MEDIASUBTYPE_RGB24) || !(vi=(VIDEOINFO *)media->Format()) ) return E_FAIL;
        bmih=vi->bmiHeader;	SetWindowPos(hwnd,0,0,0,20+(w=vi->bmiHeader.biWidth),60+(h=vi->bmiHeader.biHeight),SWP_NOZORDER|SWP_NOMOVE);
        return  S_OK;
    }
    HRESULT DoRenderSample(IMediaSample *sample){
        BYTE* data; sample->GetPointer( &data ); 
        // Process RGB Frame data* here. For Example: ZeroMemory(data+w*h,w*h);
        BITMAPINFO bmi={0}; bmi.bmiHeader=bmih; RECT r; GetClientRect( hwnd, &r );
        HDC dc=GetDC(hwnd);
        StretchDIBits(dc,0,16,r.right,r.bottom-16,0,0,w,h,data,&bmi,DIB_RGB_COLORS,SRCCOPY);
        ReleaseDC(dc);
        return  S_OK;
    }
    HRESULT ShouldDrawSampleNow(IMediaSample *sample, REFERENCE_TIME *start, REFERENCE_TIME *stop) {
        return S_OK; // disable droping of frames
    }
};

int WINAPI WinMain(HINSTANCE inst,HINSTANCE prev,LPSTR cmd,int show){
    HRESULT hr = CoInitialize(0);
    hwnd=CreateWindowEx(0,"LISTBOX",0,WS_SIZEBOX,0,0,0,0,0,0,0,0); 
           
    IGraphBuilder*  graph= 0; hr = CoCreateInstance( CLSID_FilterGraph, 0, CLSCTX_INPROC,IID_IGraphBuilder, (void **)&graph );
    IMediaControl*  ctrl = 0; hr = graph->QueryInterface( IID_IMediaControl, (void **)&ctrl );

    Sampler*        sampler      = new Sampler(0,&hr); 
    IPin*           rnd  = 0; hr = sampler->FindPin(L"In", &rnd);
                              hr = graph->AddFilter((IBaseFilter*)sampler, L"Sampler");

    ICreateDevEnum* devs = 0; hr = CoCreateInstance (CLSID_SystemDeviceEnum, 0, CLSCTX_INPROC, IID_ICreateDevEnum, (void **) &devs);
    IEnumMoniker*   cams = 0; hr = devs?devs->CreateClassEnumerator (CLSID_VideoInputDeviceCategory, &cams, 0):0;  
    IMoniker*       mon  = 0; hr = cams?cams->Next (1, &mon, 0):0;
    IBaseFilter*    cam  = 0; hr = mon?mon->BindToObject(0,0,IID_IBaseFilter, (void**)&cam):0;
    IEnumPins*      pins = 0; hr = cam?cam->EnumPins(&pins):0; 
    IPin*           cap  = 0; hr = pins?pins->Next(1,&cap, 0):0;
                              hr = graph->AddFilter(cam, L"Capture Source"); 

    IBaseFilter*    vid  = 0; hr = graph->AddSourceFilter (L"c:\\Windows\\clock.avi", L"File Source", &vid);
    IPin*           avi  = 0; hr = vid?vid->FindPin(L"Output", &avi):0;

    hr = graph->Connect(cap?cap:avi,rnd);
    hr = graph->Render( cap?cap:avi );
    hr = ctrl->Run();
    SendMessage(hwnd, LB_ADDSTRING, 0, (long)(cap?"Capture source ...":"File source ..."));
    ShowWindow(hwnd,SW_SHOW);

    while ( msg.message != WM_QUIT) {  
        if( PeekMessage(      &msg, 0, 0, 0, PM_REMOVE ) ) {
            TranslateMessage( &msg );   
            DispatchMessage(  &msg ); 
            if(msg.message == WM_KEYDOWN && msg.wParam==VK_ESCAPE ) break;
        }   Sleep(30);
    }
};

Points of interest

The most important lines are in DoRenderSample since there you get your image in RGB format in every frame. The sample itself tries to find and connects to the first found pin on the first found video capturing device. And if none is found, then it tries to open and run a video file from the provided path. So in fact, these are two tutorials in one. The cool side-feature is that you can use the Alt+PrintScreen key to grab a screenshot from (any?) movie/ webcam and paste/save it to your favorite image editor.

Place for enhancements 

Consider using some form of smart pointers like CComPtr<IGRAPHBUILDER> instead of raw pointers to interfaces like IGraphBuilder*. I removed them to keep the sample buildable by people who don't have the ATL libs. Example code to have more control over the video playback (auto-rewind in this case).

...
IMediaControl*  ctrl  = 0;  hr = graph->QueryInterface( IID_IMediaControl, (void **)&ctrl );
IMediaSeeking*  seek  = 0;  hr = graph->QueryInterface( IID_IMediaSeeking, (void **)&seek );     
IMediaEvent*    event = 0;  hr = graph->QueryInterface( IID_IMediaEventEx, (void **)&event );
...
    while ( msg.message != WM_QUIT) {  
        long code, a, b; event->GetEvent(&code, &a, &b, 0); 
        if( code == EC_COMPLETE ) {	
              LONGLONG pos=0;
              hr=seek->SetPositions(&pos,AM_SEEKING_AbsolutePositioning,0,0);
              hr=ctrl->Run();
        }
...

You may add this code to inspect (connect to a remote graph) the whole render graph, graphically, in graphedit (to debug the capture input pins, for example):

WCHAR wsz[256]; (void)StringCchPrintfW(wsz, NUMELMS(wsz),L"FilterGraph %08x  pid %08x\0", (DWORD_PTR) 0, GetCurrentProcessId());
...
IGraphBuilder*  graph      = 0; hr = CoCreateInstance( CLSID_FilterGraph, 0, CLSCTX_INPROC,IID_IGraphBuilder, (void **)&graph );
IMoniker*       moniker    = 0; hr = CreateItemMoniker(L"!", wsz, &moniker); DWORD regno = 0xfedcba98;
IRunningObjectTable* gedit = 0; hr = GetRunningObjectTable(0, &gedit);
                                hr = gedit->Register(ROTFLAGS_REGISTRATIONKEEPSALIVE, graph, moniker, &regno); 

Some valuable tips from the Forum users.

Unresolved externals: Make sure you did build libs mentioned in article.
VS6: Get Platform SDK and don't forget to set standard SDK include / lib paths
VS2005: unresolved externals
Go to "Configuration Properties" -> "General" -> "Character Set"
By default, it is set to "Use Unicode Character Set". Change it to "Not Set" and the project should link.

So enjoy it, and don't forget that most of the advanced error handling etc. is up to you to implement, and this is just very basic sample so you can focus on how it works.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Ladislav Nevery
Software Developer (Senior)
Slovakia Slovakia
Past Projects:
[Siemens.sk]Mobile network software: HLR-Inovation for telering.at (Corba)
Medical software: CorRea module for CT scanner
[cauldron.sk]Computer Games:XboxLive/net code for Conan, Knights of the temple II, GeneTroopers, CivilWar, Soldier of fortune II
[www.elveon.com]Computer Games:XboxLive/net code for Elveon game based on Unreal Engine 3
ESET Reasearch.
Looking for job

Comments and Discussions

 
QuestionNeed only 1 frame Pinmemberguju4-Jun-14 5:06 
QuestionMe PinmemberMember 1026182226-Dec-13 6:38 
QuestionThanks for excellent intro to this stuff PinmemberMember 1039745210-Dec-13 19:49 
QuestionExcellent article PinmemberMember 1039745210-Dec-13 19:46 
GeneralMy vote of 5 PinmemberMember 101328458-Jul-13 5:12 
QuestionHow can I save the video recording in a file? Pinmembershahzebrahat29-Apr-13 11:31 
QuestionThanks PinmemberEnsamblador5-Sep-12 21:42 
GeneralNice PinmvpEspen Harlinn4-Sep-12 10:56 
Questiongreate article, no source download. PinmemberSpoolrd1-Aug-12 22:26 
Questioncapturing avi problem Pinmemberhoseinhero25-Jul-12 9:25 
hey Ladislav. I have been tried VFW capture, but it dosen't support my webcam driver. and in directshow I have trouble when i want to save webcam data to an avi file. I tried your first sample but I can't save data to disk. Would you please show me how can I save it to disk.
AnswerRe: capturing avi problem [modified] Pinmembermerano8-Apr-13 13:48 
Questioncan't use the code Pinmembertedysoegiantoarsenal3-Jul-12 19:32 
Questionwhere Receive() and Receive_() arguments come from ? [modified] Pinmemberdsant25-Apr-12 2:22 
GeneralMy vote of 5 Pinmemberdsant25-Apr-12 2:07 
GeneralMy vote of 5 Pinmembercoronafire12-Dec-11 14:21 
GeneralMy vote of 5 PinmemberD Greene9-Dec-11 15:51 
Question2 displays PinmemberMember 83193911-Dec-11 4:04 
GeneralMy vote of 5 Pinmemberliftmike23-Jul-11 1:41 
GeneralMy vote of 5 Pinmembermerano7-Mar-11 7:30 
GeneralMy vote of 5 Pinmembermr colorbooks23-Feb-11 14:09 
GeneralMy vote of 5 Pinmemberforgi00713-Jan-11 4:36 
GeneralIt got slow when get buffer data Pinmemberstonechao28-Dec-10 20:49 
GeneralRe: It got slow when get buffer data Pinmemberodedelyada19-Feb-11 6:57 
GeneralDownload code and example program Pinmembervanwoudenberg21-Dec-10 3:02 
GeneralReal time video PinmemberMiriV10-Nov-10 6:22 
QuestioncapAVI and clipboard? PinmemberVaclav_Sal5-Oct-10 6:14 
GeneralFormat of data Pinmemberragnor129-Aug-10 3:59 
GeneralMe too Pinmemberdsant25-Apr-12 23:38 
QuestionHow to set the frame rate? Pinmembercb714916-Aug-10 12:21 
QuestionSmall question: how to hide, close, or not to create preview window? Pinmemberpvagin8-May-10 11:14 
AnswerRe: Small question: how to hide, close, or not to create preview window? PinmemberLadislav Nevery11-May-10 13:22 
GeneralExcellent ! But a little problem: WinMain doesn't exit normally PinmemberEric Chi3-May-10 0:09 
GeneralRe: Excellent ! But a little problem: WinMain doesn't exit normally PinmemberLadislav Nevery11-May-10 13:36 
GeneralHere's how i got it to compile. PinmemberFocusedWolf3-Oct-09 15:51 
GeneralAdvise needed for video processing art installation PinmemberPat Wickline13-Apr-09 10:14 
Generalunresolved symbol linker fix Pinmemberptest1017-Jan-09 15:03 
GeneralGetting Error: PinmemberMember 360679228-Jul-08 20:52 
GeneralRe: Getting Error: [modified] Pinmembermerano24-Nov-08 11:20 
GeneralSuccessfully compiled but do nothing in runtime Pinmemberhmcen3-Jul-08 23:40 
GeneralThank you for showing Microsoft how it should be done Pinmemberjohnramunas20-Jun-08 22:58 
Generalrenqual.cpp Pinmemberirene24-Feb-08 1:10 
QuestionCleaning up Graph ? PinsussChristoph Fuenfzig16-Dec-07 14:20 
GeneralBeautiful Pinmemberalargespeaker14-Nov-07 8:24 
GeneralRe: Beautiful Pinmember83564783420-Nov-07 13:06 
QuestionReal-time video image processing - Selected Area PinmemberTechnocrack11-Nov-07 5:04 
GeneralReal-time video image processing PinmemberTechnocrack11-Nov-07 5:03 
GeneralCompiler error PinmemberReve1017-May-07 20:05 
GeneralLinking Error Pinmembervishaljogi111-Apr-07 6:28 
Generalcrashed on running this code Pinmembervishaljogi110-Apr-07 22:51 
GeneralHad some problems, but solved them Pinmembersukatoss5-Apr-07 17:35 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.140905.1 | Last Updated 6 Apr 2010
Article Copyright 2006 by Ladislav Nevery
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid