Click here to Skip to main content
15,867,568 members
Articles / Programming Languages / C++
Article

Video Shadering with Direct3D

Rate me:
Please Sign up or sign in to vote.
4.65/5 (13 votes)
7 Jun 2011LGPL39 min read 92.9K   4.4K   28   30
This library allows rendering YUV and RGB pixel formats using Direct3D 9 runtime. You can also apply shaders and add text, bitmap and line overlays.

Introduction

This project started as a testing application for another project I am developing and since I needed some code for rendering YUV420 pixel data and was curious about how Direct3D could be used for 2D graphics - I decided to give it a try. But before I jump into the video implementation details, I will briefly describe some background of 2D graphics with a low-level 3D API. I assume you have a basic knowledge of Direct3D 9 API and HLSL programming.

2D Graphics with Direct3D API

Direct3D is around since 1995 and for a long time was considered a gaming playground. With the rise of the GPU processing power, more and more applications started to take advantage of Direct3D API to facilitate the capabilities of parallel processing and floating point calculations - areas where the GPU outperforms CPU. Image and video processing is one of these areas where GPU may dramatically improve performance and user experience.

Direct3D is a low level API which allows you to design your application by exact needs of your business - therefore, as a developer, you have to write more code and take care of all the rendering details which are usually not trivial to understand. By the way, with the release of Windows 7, Microsoft introduced a new 2D API called, not surprisingly, Direct2D. It is also a GPU accelerated API and much easier to use, however it does not include (for now) support for YUV surfaces and is therefore less suitable for video rendering. Nevertheless, I recently wrote a video rendering implementation with Direct2D which can be found here.

Pixel Formats

Video decoders usually output frames in YUV pixel format which is more suitable for video processing since it divides each video frame into luminance - the black and white data, and into chrominance - the colored data. The most used formats are I420 (also called IYUV) and YV12, both belong to YUV420 type which means each frame has W(width) x H(height) number of luma bytes followed by W x H / 2 chroma bytes. Direct3D allows you to convert those frames into RGB format using graphics device, thus saving CPU power and boosting performance.

This library was tested with the following pixel formats:

  • YV12 – 12 bits per pixel planar format with Y plane followed by V and U planes
  • NV12 – 12 bits per pixel planar format with Y plane and interleaved UV plane
  • YUY2 – 16 bits per pixel packed YUYV array
  • UYVY - 16 bits per pixel packed UYVY array
  • RGB555 – 16 bits per pixel with 1 bit unused and 5 bits for each RGB channel
  • RGB565 – 16 bits per pixel with 5 bits Red, 6 bits Green, and 5 bits Blue
  • RGB32 – 32 bits per pixel with 8 bits unused and 8 bits for each RGB channel
  • ARGB32 - 32 bits per pixel with 8 bits Alpha and 8 bits for each RGB channel

Support for pixel formats may vary between different video cards, therefore you should check if your card supports given format (see using the code section) before creating video surface.

2D Vertices and Matrices

Vertex is the most common primitive of Direct3D API - it has at least 3 points, X, Y and Z which describe a location in a 3D space and optionally diffuse color and a texture coordinate (discussed later). When working with 2D graphics, the Z value is unnecessary and always will be zero. Direct3D lets you construct vertex constructs of your flavor and you also need to tell the API what each field of the Vertex structure represents. In this case, I am using the following structure:

C++
struct VERTEX
{
          D3DXVECTOR3 pos;        // vertex untransformed position
          D3DCOLOR    color;      // diffuse color
          D3DXVECTOR2 texPos;     // texture relative coordinates
};
 
#define D3DFVF_CUSTOMVERTEX ( D3DFVF_XYZ | D3DFVF_DIFFUSE | D3DFVF_TEX1 )

The D3DFVF_CUSTOMVERTEX macro tells the API that the first parameter is 3D position vector, the second one is vertex color and the third is texture position. After the vertex format is defined, we can move forward and create triangles. Our goal is to create a rectangle on which the video will be rendered, so we need 2 triangles. Since these triangles are adjacent, 4 vertices will do the work.

After our rectangle is ready, we need to define matrices which transform 3D model into 2D screen space. Since our model itself is a 2D model, we only need to define Projection matrix, and set View and World matrices to identity. Direct3D uses left-handed Cartesian coordinates which means that the Z axis is pointing into the screen and Direct3DX provides D3DXMatrixOrthoOffCenterLH API for setting orthographic projection matrix according to the width and height of the render window:

C++
HRESULT D3D9RenderImpl::SetupMatrices(int width, int height)
{
    D3DXMATRIX matOrtho; 
    D3DXMATRIX matIdentity;
 
    D3DXMatrixOrthoOffCenterLH(&matOrtho, 0, width, height, 0, 0.0f, 1.0f);
    D3DXMatrixIdentity(&matIdentity);
 
    HR(m_pDevice->SetTransform(D3DTS_PROJECTION, &matOrtho));
    HR(m_pDevice->SetTransform(D3DTS_WORLD, &matIdentity));
    return m_pDevice->SetTransform(D3DTS_VIEW, &matIdentity);
}

Surfaces and Textures

In order to perform YUV -> RGB conversion and scale the video from its original size to the video window size, we need 2 surfaces. One will store YUV planar data and have dimensions of the video frames and the other will have the same size as the video window and store ARGB pixel format same as display adapter format. We can set the second surface as a render target, however using texture as render target improves performance and allows to use different effects like blending with overlays. Texture itself is a special kind of surface which can have several embedded surfaces. The render target texture is created automatically as soon as you pass the window handle of the display window:

C++
HRESULT D3D9RenderImpl::CreateRenderTarget(int width, int height)
{
    HR(m_pDevice->CreateTexture(width, height, 1, D3DUSAGE_RENDERTARGET, 
    m_displayMode.Format, D3DPOOL_DEFAULT, &m_pTexture, NULL));
    HR(m_pTexture->GetSurfaceLevel(0, &m_pTextureSurface));
    HR(m_pDevice->CreateVertexBuffer(sizeof(VERTEX) * 4, 
    D3DUSAGE_DYNAMIC | D3DUSAGE_WRITEONLY, D3DFVF_CUSTOMVERTEX, 
    D3DPOOL_DEFAULT, &m_pVertexBuffer, NULL));
    
    VERTEX vertexArray[] =
    {
        { D3DXVECTOR3(0, 0, 0),          
        D3DCOLOR_ARGB(255, 255, 255, 255), D3DXVECTOR2(0, 0) },  // top left
        { D3DXVECTOR3(width, 0, 0),      
        D3DCOLOR_ARGB(255, 255, 255, 255), D3DXVECTOR2(1, 0) },  // top right
        { D3DXVECTOR3(width, height, 0), 
        D3DCOLOR_ARGB(255, 255, 255, 255), D3DXVECTOR2(1, 1) },  // bottom right
        { D3DXVECTOR3(0, height, 0),     
        D3DCOLOR_ARGB(255, 255, 255, 255), D3DXVECTOR2(0, 1) },  // bottom left
    };
 
    VERTEX *vertices;
    HR(m_pVertexBuffer->Lock(0, 0, (void**)&vertices, D3DLOCK_DISCARD));
 
    memcpy(vertices, vertexArray, sizeof(vertexArray));
 
    return m_pVertexBuffer->Unlock();
}

The width and height belong to display window and after creating the texture and getting its first level surface, I set the vertex buffer with the rectangle on which the texture will be applied. Regarding the source surface - it must be created by you by calling CreateVideoSurface method and passing the video width, height and pixel format:

C++
HRESULT D3D9RenderImpl::CreateVideoSurface(int width, int height, D3DFORMAT format)
{
    m_videoWidth = width;
    m_videoHeight = height;
    m_format = format;
 
    HR(m_pDevice->CreateOffscreenPlainSurface(width, height, format, 
			D3DPOOL_DEFAULT, &m_pOffsceenSurface, NULL));
 
    return m_pDevice->ColorFill(m_pOffsceenSurface, NULL, D3DCOLOR_ARGB(0xFF, 0, 0, 0));
}

Please note that the C++ implementation uses D3DFORMAT from DirectX SDK and the managed C++/CLI version uses PixelFormat enumeration which is also converted to D3DFORMAT in the interop layer.

The StretchRect Magic

After successfully filling the video surface with YUV pixel data, we have to convert this data to format supported by the display adapter - usually it will be ARGB32. If you performed this conversion on CPU, you probably noticed the overhead it has on both time of conversion and CPU load. Using the DIrect3D 9 SDK, it is done in a fast and efficient way through the StretchRect API. This method takes the source surface which in this case is the video surface and blits its contents to the target surface - in this case, the surface of the texture render target. In addition, it performs scaling to fit the video frame size to the size of the display window and also can be used for changing the aspect ratio of the video frames.

When the frame is ready for display, you will call the Display method passing 3 pointers pointing to Y, U(Cb) and V(Cr) pixel planes. Data from these pointers is copied to the video surface and then stretched to the render target surface and finally displayed to the user.

Shaders

Shaders are programs written using HLSL (High Level Shadering Language) and executed on GPU. Before jumping into HLSL programming, make sure to refresh your knowledge in linear algebra since HLSL is all about scalars, vectors, matrices and operations between them. The language itself is C like without direct memory access, i.e., no pointers to video memory. Shaders were introduced in DirectX 8 and since then gained a lot of popularity in graphics programming. Direct3D 9 provides 2 shader types available in its graphics pipeline: Vertex shader and Pixel Shader. Vertex shaders are responsible for transforming vertices in 3D scene and less used in 2D graphics since the number of vertices usually will be small. Nevertheless, I use a vertex shader to perform video transformations like horizontal or vertical flipping and rotations. Pixel shader is a more interesting shader type since it is responsible for setting pixel color for each pixel visible on scene. The shader program you write in HLSL is executed once for every pixel and since pixels' colors are independent from each other - this process runs in parallel on tens or even hundreds of GPU cores with great efficiency. You benefit from both offloading CPU from intensive floating point computations and provide great user experience even in HD video with 30 fps. The ability to apply pixel shader on every frame of the video stream also introduces great flexibility as you can write custom effects and perform, a so called, video shadering. It is worth mentioning, the library can render video in any mode, i.e., with both shaders enabled, both disabled or either one of them enabled.

HLSL primer is beyond the scope of this article, however I must notice a great tool for basic shader authoring called Shazzam. It contains a lot of samples at both introduction and advanced levels. You can write the shader code in its text editor, compile it and test it on one of the images before applying it in video rendering.

Video Overlays

As described earlier, when using texture as render target, you benefit from better performance and it also allows you to add overlay geometries and blend them with the video frames. Such overlays may be lines, polygons, text and bitmaps. Each of these elements have an opacity key which determines how the underlying texture which holds the video frame's pixel are blended with the overlay.

Using the Code

This library was developed using C++ for best performance and flexibility. Using C++/CLI, it is also exposed to the .NET Framework. The most basic usage in C# would be:

C++
D3D9Renderer render = new D3D9Renderer(panel1.Handle);
if (render.CheckFormatConversion(PixelFormat.YV12))
{
   render.CreateVideoSurface(video.Width, video.Height, PixelFormat.YV12);
}
 
render.Display(planes[0], planes[1], planes[2], false);

To add overlays, you can call one of the Add-xxx-Overlay functions. Overlays are stored in a hash table and are protected by critical section so you can freely add and remove overlays during rendering. The following code adds bitmap overlay (must be a 32 bit bitmap) at point 20,20 and half transparent. After that, we add blue text with Ariel font and size 16. The overlays later may be removed by their keys 3 or 12.

C++
Image img = Image.FromFile(@"C:\Logo.png");
render.AddBitmapOverlay(12, new System.Drawing.Point(20, 20), new Bitmap(img), 135);
render.AddTextOverlay(3, "Roman", 
	new Rectangle(20, 20, 200, 200), 16, Color.Blue, "Ariel", 100);

Setting pixel shader is also an easy task. After you successfully compiled your shader code and tested it, you can set pixel or vertex shader, or both. The applied shader will be effective on the frame data only and not on the overlays, since before rendering overlays, shaders are disabled. The following code is a pretty simple shader which inverts color for each pixel.

C++
sampler2D TexSampler : register(S0);

float4 main(float2 uv : TEXCOORD) : COLOR
{
   float4 color = tex2D(TexSampler, uv );
   return float4(1 - color.rgb, color.a);
}
renders.SetPixelShader(@"Shaders\effect.fx", "main", "ps_2_0");

You can also use more advanced shaders with input scalars, vectors and matrices.

Conclusion

I had a lot of fun developing this library and writing this tutorial. This is probably one of the most complex and time consuming projects I have ever worked on and I learned a lot from it. Thanks to all the tutorials mentioned below - without them, I would have hardly succeeded. I hope you will find this library useful. You are welcome to submit your thoughts and comments regarding future directions.

References

History

  • 6th June, 2011: Initial post

License

This article, along with any associated source code and files, is licensed under The GNU Lesser General Public License (LGPLv3)


Written By
Software Developer (Senior)
Israel Israel
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
QuestionAdding overlays needs directx9 Pin
Member 951777212-Apr-17 21:19
Member 951777212-Apr-17 21:19 
QuestionDirectX 11 support Pin
Christian Knobloch22-Mar-17 4:44
Christian Knobloch22-Mar-17 4:44 
AnswerRe: DirectX 11 support Pin
Carl Are16-Jan-18 4:33
Carl Are16-Jan-18 4:33 
QuestionHow to convert RGB back to YUV Pin
Mysteryx9315-Jul-15 11:22
Mysteryx9315-Jul-15 11:22 
Question'System.Runtime.InteropServices.COMException' HRESULT: 0x8876086A Pin
Zach Saw3-May-14 3:06
Zach Saw3-May-14 3:06 
QuestionGreat Pin
bstiger14-Feb-14 1:01
bstiger14-Feb-14 1:01 
QuestionUnsupported chroma type I410 Pin
Member 104979754-Jan-14 5:51
Member 104979754-Jan-14 5:51 
QuestionCan convert rgb to yuv by shader? Pin
liaoyuandeyehuo4-Sep-13 21:02
liaoyuandeyehuo4-Sep-13 21:02 
QuestionUnsupported chroma type J422 Pin
Member 907458710-Feb-13 20:58
Member 907458710-Feb-13 20:58 
AnswerRe: Unsupported chroma type J422 Pin
roman_gin11-Feb-13 6:00
roman_gin11-Feb-13 6:00 
QuestionDoes this library have .net 3.5 version? Pin
casper_mystic@yahoo.com16-Jan-13 1:11
casper_mystic@yahoo.com16-Jan-13 1:11 
AnswerRe: Does this library have .net 3.5 version? Pin
roman_gin16-Jan-13 7:23
roman_gin16-Jan-13 7:23 
GeneralRe: Does this library have .net 3.5 version? Pin
casper_mystic@yahoo.com28-Jan-13 0:24
casper_mystic@yahoo.com28-Jan-13 0:24 
GeneralVideo stop playing when displayed in extended screen Pin
casper_mystic@yahoo.com28-Jan-13 5:17
casper_mystic@yahoo.com28-Jan-13 5:17 
GeneralRe: Video stop playing when displayed in extended screen Pin
roman_gin29-Jan-13 8:54
roman_gin29-Jan-13 8:54 
GeneralRe: Video stop playing when displayed in extended screen Pin
casper_mystic@yahoo.com29-Jan-13 19:37
casper_mystic@yahoo.com29-Jan-13 19:37 
GeneralRe: Video stop playing when displayed in extended screen Pin
roman_gin4-Feb-13 20:59
roman_gin4-Feb-13 20:59 
GeneralRe: Video stop playing when displayed in extended screen Pin
casper_mystic@yahoo.com5-Feb-13 6:51
casper_mystic@yahoo.com5-Feb-13 6:51 
GeneralRe: Video stop playing when displayed in extended screen Pin
roman_gin2-Jun-13 4:19
roman_gin2-Jun-13 4:19 
SuggestionExample program required Pin
Kozlov_Sergey30-Sep-11 4:06
Kozlov_Sergey30-Sep-11 4:06 
GeneralRe: Example program required Pin
roman_gin6-Oct-11 22:09
roman_gin6-Oct-11 22:09 
QuestionRe: Example program required Pin
Kozlov_Sergey6-Oct-11 22:12
Kozlov_Sergey6-Oct-11 22:12 
AnswerRe: Example program required Pin
roman_gin6-Oct-11 22:26
roman_gin6-Oct-11 22:26 
QuestionCompilation errors. Pin
Kozlov_Sergey30-Sep-11 3:58
Kozlov_Sergey30-Sep-11 3:58 
GeneralHow to use from C++? Pin
yafan8-Jun-11 7:00
yafan8-Jun-11 7:00 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.