Prototyping DirectShow filters in C#

Sample Image

Introduction

DirectShow filters are the basic building blocks of multimedia applications on the Windows platform. Normally, they are written in C++. In this article, I'll show how to implement a couple of DirectShow transformation filters in C#. Since Microsoft doesn't recommend using managed code for filters, I'll call them prototypes but they perform quite well in most circumstances. We'll develop Sobel transformation as a DirectShow filter. Sobel transformations are often used for video segmentation which is the process of identifying objects in a scene.

Background

The strategy we use to implement a DirectShow filter in C# relies on having a C++ class that contains a pointer to a managed object which implements our filter. The C++ class just forwards all the processing to the managed object. The C++ class is generated using Cutler's DirectShow Filters Wizard. We just clean up the wizard-generated code to include a pointer to our C# object and implement all the filter processing as methods of the managed object.

I've written this Sobel filter to have an idea of the kind of performance we'd get with C#. If you want to "take your CPU for a spin", this is a good test. As written (i.e., with no optimization effort), there are around 20 integer multiplications per pixel. Usually, you'd transform an image to black and white before applying Sobel transformation to the image. So I've also written a transformation filter that converts a RGB24 bitmap to black and white.

Using the code

If you just want to have a filter that implements Sobel transformations, you can download and install the managed DLL and Ax files (from the download link). Then after registering the Ax file and installing the managed DLL in the Global Assembly Cache (GAC), you can start the GraphEdit utility and add the "CsSobelV2" filter (found in the DirectShow filters category of GraphEdit). Then render a media file to see a black and white "sketch" of the frames in the video. (Note that the open-source DirectShowLib must also be in the GAC.) More information can be found in the "readme.txt" file that is included in the download.

If you want to implement your own filter using these filters as a starting point, you can modify the Transform and ProcessFrame methods of the MySobelV2 class. This is sufficient if you are satisfied with a single media type for the video frame, here Rgb24, and you can live with the default memory allocator of the C++ DirectShow base classes (this covers a lot of situations).

Sobel transformation

The Sobel transformation uses the following two matrices:

  // define the horizontal Sobel mat
  int [,] hx = new  int [,] { {-1, 0, 1}, 
                    {-2, 0, 2}, {-1, 0, 1} };

  // define the vertical Sobel mat
  int[,] hy = new int [,]  { { 1,  2,  1}, 
                  { 0,  0,  0}, {-1, -2, -1} };

And consists of the following code, found in ProcessFrame, which is repeated for every pixel in every frame of a video:

        gradX = 0;
        gradY = 0;
        for (int row = -1; row <= 1; row++)
        {
          for (int col = -1; col <= 1; col++)
          {
            position = GetPosition( x + col, y + row ); 
            intensity = pwSource[position];
            gradX += (intensity * hx[col + 1,row + 1]);
            gradY += (intensity * hy[col + 1,row + 1]); 
          }
        }
         
        // compute the square of the gradient        
        gradMag = gradX * gradX + gradY * gradY;

        // threshold default at 128
        gradMag = (gradMag < threshold*threshold) ? 255 : 0;
               
        // assign the pixel to the destination
        position = GetPosition( x, y );
        pwTarget[position] = pwTarget[position+1] = 
              pwTarget[position+2] = (byte)gradMag;

The intuition behind the transformation is relatively simple. Since the contour of objects consists of a region in a picture where the color changes rapidly, the Sobel transformation amplifies these differences and simply sets to white those pixels whose changes are too small (and black, the other ones). The horizontal Sobel transformation looks at the left and right neighboring pixels and multiplies these values by 2 or -2, it also looks at the upper right and left, and lower right and left pixels. After summing all these values, you compute the square of the Euclidian distance and use an arbitrary threshold to divide the values.

Points of Interest

I've also included the code for a black and white transformation filter for uncompressed Rgb24 video frames. As mentioned, you'd normally perform this transformation before feeding the result to the Sobel filter. So you can test the performances of a filter graph with two filters written in managed code.

Property pages for filters are handy when used in a utility like GraphEdit. But being able to access custom filter properties from C# (easily) is even more useful. So I've modified Cutler's Wizard-generated C++ code to include a property that can be changed in C#. Similar changes could be done to implement different properties.

Limitations and known issues

The media type accepted by the filters is uncompressed Rgb24 video. Since this is one of the most "natural" media types, it's not a big limitation but the filters were not designed to be very flexible. They are just prototypes to experiment with.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here