Click here to Skip to main content
12,821,494 members (26,178 online)
Click here to Skip to main content
Add your own
alternative version

Tagged as


24 bookmarked
Posted 7 Jun 2011

Video Shadering with Direct3D

, 7 Jun 2011 LGPL3
Rate this:
Please Sign up or sign in to vote.
This library allows rendering YUV and RGB pixel formats using Direct3D 9 runtime. You can also apply shaders and add text, bitmap and line overlays.


This project started as a testing application for another project I am developing and since I needed some code for rendering YUV420 pixel data and was curious about how Direct3D could be used for 2D graphics - I decided to give it a try. But before I jump into the video implementation details, I will briefly describe some background of 2D graphics with a low-level 3D API. I assume you have a basic knowledge of Direct3D 9 API and HLSL programming.

2D Graphics with Direct3D API

Direct3D is around since 1995 and for a long time was considered a gaming playground. With the rise of the GPU processing power, more and more applications started to take advantage of Direct3D API to facilitate the capabilities of parallel processing and floating point calculations - areas where the GPU outperforms CPU. Image and video processing is one of these areas where GPU may dramatically improve performance and user experience.

Direct3D is a low level API which allows you to design your application by exact needs of your business - therefore, as a developer, you have to write more code and take care of all the rendering details which are usually not trivial to understand. By the way, with the release of Windows 7, Microsoft introduced a new 2D API called, not surprisingly, Direct2D. It is also a GPU accelerated API and much easier to use, however it does not include (for now) support for YUV surfaces and is therefore less suitable for video rendering. Nevertheless, I recently wrote a video rendering implementation with Direct2D which can be found here.

Pixel Formats

Video decoders usually output frames in YUV pixel format which is more suitable for video processing since it divides each video frame into luminance - the black and white data, and into chrominance - the colored data. The most used formats are I420 (also called IYUV) and YV12, both belong to YUV420 type which means each frame has W(width) x H(height) number of luma bytes followed by W x H / 2 chroma bytes. Direct3D allows you to convert those frames into RGB format using graphics device, thus saving CPU power and boosting performance.

This library was tested with the following pixel formats:

  • YV12 – 12 bits per pixel planar format with Y plane followed by V and U planes
  • NV12 – 12 bits per pixel planar format with Y plane and interleaved UV plane
  • YUY2 – 16 bits per pixel packed YUYV array
  • UYVY - 16 bits per pixel packed UYVY array
  • RGB555 – 16 bits per pixel with 1 bit unused and 5 bits for each RGB channel
  • RGB565 – 16 bits per pixel with 5 bits Red, 6 bits Green, and 5 bits Blue
  • RGB32 – 32 bits per pixel with 8 bits unused and 8 bits for each RGB channel
  • ARGB32 - 32 bits per pixel with 8 bits Alpha and 8 bits for each RGB channel

Support for pixel formats may vary between different video cards, therefore you should check if your card supports given format (see using the code section) before creating video surface.

2D Vertices and Matrices

Vertex is the most common primitive of Direct3D API - it has at least 3 points, X, Y and Z which describe a location in a 3D space and optionally diffuse color and a texture coordinate (discussed later). When working with 2D graphics, the Z value is unnecessary and always will be zero. Direct3D lets you construct vertex constructs of your flavor and you also need to tell the API what each field of the Vertex structure represents. In this case, I am using the following structure:

struct VERTEX
          D3DXVECTOR3 pos;        // vertex untransformed position
          D3DCOLOR    color;      // diffuse color
          D3DXVECTOR2 texPos;     // texture relative coordinates

The D3DFVF_CUSTOMVERTEX macro tells the API that the first parameter is 3D position vector, the second one is vertex color and the third is texture position. After the vertex format is defined, we can move forward and create triangles. Our goal is to create a rectangle on which the video will be rendered, so we need 2 triangles. Since these triangles are adjacent, 4 vertices will do the work.

After our rectangle is ready, we need to define matrices which transform 3D model into 2D screen space. Since our model itself is a 2D model, we only need to define Projection matrix, and set View and World matrices to identity. Direct3D uses left-handed Cartesian coordinates which means that the Z axis is pointing into the screen and Direct3DX provides D3DXMatrixOrthoOffCenterLH API for setting orthographic projection matrix according to the width and height of the render window:

HRESULT D3D9RenderImpl::SetupMatrices(int width, int height)
    D3DXMATRIX matOrtho; 
    D3DXMATRIX matIdentity;
    D3DXMatrixOrthoOffCenterLH(&matOrtho, 0, width, height, 0, 0.0f, 1.0f);
    HR(m_pDevice->SetTransform(D3DTS_PROJECTION, &matOrtho));
    HR(m_pDevice->SetTransform(D3DTS_WORLD, &matIdentity));
    return m_pDevice->SetTransform(D3DTS_VIEW, &matIdentity);

Surfaces and Textures

In order to perform YUV -> RGB conversion and scale the video from its original size to the video window size, we need 2 surfaces. One will store YUV planar data and have dimensions of the video frames and the other will have the same size as the video window and store ARGB pixel format same as display adapter format. We can set the second surface as a render target, however using texture as render target improves performance and allows to use different effects like blending with overlays. Texture itself is a special kind of surface which can have several embedded surfaces. The render target texture is created automatically as soon as you pass the window handle of the display window:

HRESULT D3D9RenderImpl::CreateRenderTarget(int width, int height)
    HR(m_pDevice->CreateTexture(width, height, 1, D3DUSAGE_RENDERTARGET, 
    m_displayMode.Format, D3DPOOL_DEFAULT, &m_pTexture, NULL));
    HR(m_pTexture->GetSurfaceLevel(0, &m_pTextureSurface));
    HR(m_pDevice->CreateVertexBuffer(sizeof(VERTEX) * 4, 
    D3DPOOL_DEFAULT, &m_pVertexBuffer, NULL));
    VERTEX vertexArray[] =
        { D3DXVECTOR3(0, 0, 0),          
        D3DCOLOR_ARGB(255, 255, 255, 255), D3DXVECTOR2(0, 0) },  // top left
        { D3DXVECTOR3(width, 0, 0),      
        D3DCOLOR_ARGB(255, 255, 255, 255), D3DXVECTOR2(1, 0) },  // top right
        { D3DXVECTOR3(width, height, 0), 
        D3DCOLOR_ARGB(255, 255, 255, 255), D3DXVECTOR2(1, 1) },  // bottom right
        { D3DXVECTOR3(0, height, 0),     
        D3DCOLOR_ARGB(255, 255, 255, 255), D3DXVECTOR2(0, 1) },  // bottom left
    VERTEX *vertices;
    HR(m_pVertexBuffer->Lock(0, 0, (void**)&vertices, D3DLOCK_DISCARD));
    memcpy(vertices, vertexArray, sizeof(vertexArray));
    return m_pVertexBuffer->Unlock();

The width and height belong to display window and after creating the texture and getting its first level surface, I set the vertex buffer with the rectangle on which the texture will be applied. Regarding the source surface - it must be created by you by calling CreateVideoSurface method and passing the video width, height and pixel format:

HRESULT D3D9RenderImpl::CreateVideoSurface(int width, int height, D3DFORMAT format)
    m_videoWidth = width;
    m_videoHeight = height;
    m_format = format;
    HR(m_pDevice->CreateOffscreenPlainSurface(width, height, format, 
			D3DPOOL_DEFAULT, &m_pOffsceenSurface, NULL));
    return m_pDevice->ColorFill(m_pOffsceenSurface, NULL, D3DCOLOR_ARGB(0xFF, 0, 0, 0));

Please note that the C++ implementation uses D3DFORMAT from DirectX SDK and the managed C++/CLI version uses PixelFormat enumeration which is also converted to D3DFORMAT in the interop layer.

The StretchRect Magic

After successfully filling the video surface with YUV pixel data, we have to convert this data to format supported by the display adapter - usually it will be ARGB32. If you performed this conversion on CPU, you probably noticed the overhead it has on both time of conversion and CPU load. Using the DIrect3D 9 SDK, it is done in a fast and efficient way through the StretchRect API. This method takes the source surface which in this case is the video surface and blits its contents to the target surface - in this case, the surface of the texture render target. In addition, it performs scaling to fit the video frame size to the size of the display window and also can be used for changing the aspect ratio of the video frames.

When the frame is ready for display, you will call the Display method passing 3 pointers pointing to Y, U(Cb) and V(Cr) pixel planes. Data from these pointers is copied to the video surface and then stretched to the render target surface and finally displayed to the user.


Shaders are programs written using HLSL (High Level Shadering Language) and executed on GPU. Before jumping into HLSL programming, make sure to refresh your knowledge in linear algebra since HLSL is all about scalars, vectors, matrices and operations between them. The language itself is C like without direct memory access, i.e., no pointers to video memory. Shaders were introduced in DirectX 8 and since then gained a lot of popularity in graphics programming. Direct3D 9 provides 2 shader types available in its graphics pipeline: Vertex shader and Pixel Shader. Vertex shaders are responsible for transforming vertices in 3D scene and less used in 2D graphics since the number of vertices usually will be small. Nevertheless, I use a vertex shader to perform video transformations like horizontal or vertical flipping and rotations. Pixel shader is a more interesting shader type since it is responsible for setting pixel color for each pixel visible on scene. The shader program you write in HLSL is executed once for every pixel and since pixels' colors are independent from each other - this process runs in parallel on tens or even hundreds of GPU cores with great efficiency. You benefit from both offloading CPU from intensive floating point computations and provide great user experience even in HD video with 30 fps. The ability to apply pixel shader on every frame of the video stream also introduces great flexibility as you can write custom effects and perform, a so called, video shadering. It is worth mentioning, the library can render video in any mode, i.e., with both shaders enabled, both disabled or either one of them enabled.

HLSL primer is beyond the scope of this article, however I must notice a great tool for basic shader authoring called Shazzam. It contains a lot of samples at both introduction and advanced levels. You can write the shader code in its text editor, compile it and test it on one of the images before applying it in video rendering.

Video Overlays

As described earlier, when using texture as render target, you benefit from better performance and it also allows you to add overlay geometries and blend them with the video frames. Such overlays may be lines, polygons, text and bitmaps. Each of these elements have an opacity key which determines how the underlying texture which holds the video frame's pixel are blended with the overlay.

Using the Code

This library was developed using C++ for best performance and flexibility. Using C++/CLI, it is also exposed to the .NET Framework. The most basic usage in C# would be:

D3D9Renderer render = new D3D9Renderer(panel1.Handle);
if (render.CheckFormatConversion(PixelFormat.YV12))
   render.CreateVideoSurface(video.Width, video.Height, PixelFormat.YV12);
render.Display(planes[0], planes[1], planes[2], false);

To add overlays, you can call one of the Add-xxx-Overlay functions. Overlays are stored in a hash table and are protected by critical section so you can freely add and remove overlays during rendering. The following code adds bitmap overlay (must be a 32 bit bitmap) at point 20,20 and half transparent. After that, we add blue text with Ariel font and size 16. The overlays later may be removed by their keys 3 or 12.

Image img = Image.FromFile(@"C:\Logo.png");
render.AddBitmapOverlay(12, new System.Drawing.Point(20, 20), new Bitmap(img), 135);
render.AddTextOverlay(3, "Roman", 
	new Rectangle(20, 20, 200, 200), 16, Color.Blue, "Ariel", 100);

Setting pixel shader is also an easy task. After you successfully compiled your shader code and tested it, you can set pixel or vertex shader, or both. The applied shader will be effective on the frame data only and not on the overlays, since before rendering overlays, shaders are disabled. The following code is a pretty simple shader which inverts color for each pixel.

sampler2D TexSampler : register(S0);

float4 main(float2 uv : TEXCOORD) : COLOR
   float4 color = tex2D(TexSampler, uv );
   return float4(1 - color.rgb, color.a);
renders.SetPixelShader(@"Shaders\effect.fx", "main", "ps_2_0");

You can also use more advanced shaders with input scalars, vectors and matrices.


I had a lot of fun developing this library and writing this tutorial. This is probably one of the most complex and time consuming projects I have ever worked on and I learned a lot from it. Thanks to all the tutorials mentioned below - without them, I would have hardly succeeded. I hope you will find this library useful. You are welcome to submit your thoughts and comments regarding future directions.



  • 6th June, 2011: Initial post


This article, along with any associated source code and files, is licensed under The GNU Lesser General Public License (LGPLv3)


About the Author

Roman Ginzburg
Software Developer (Senior)
Israel Israel
No Biography provided

You may also be interested in...

Comments and Discussions

GeneralHow to use from C++? Pin
yafan8-Jun-11 8:00
memberyafan8-Jun-11 8:00 
GeneralRe: How to use from C++? Pin
roman_gin8-Jun-11 9:41
memberroman_gin8-Jun-11 9:41 
GeneralRe: How to use from C++? Pin
yafan8-Jun-11 12:27
memberyafan8-Jun-11 12:27 
GeneralFinalizer Pin
yafan9-Jun-11 6:47
memberyafan9-Jun-11 6:47 
GeneralRe: Finalizer Pin
roman_gin10-Jun-11 0:00
memberroman_gin10-Jun-11 0:00 
GeneralRe: How to use from C++? Pin
roman_gin9-Jun-11 23:58
memberroman_gin9-Jun-11 23:58 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.170308.1 | Last Updated 7 Jun 2011
Article Copyright 2011 by Roman Ginzburg
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid