Click here to Skip to main content
Click here to Skip to main content

EVR Presenter in pure C# with Direct3D Video Rendering

By , 11 Jul 2012
 

Introduction

This is my second article of customizing video output with .NET and in pure C# code. More stuff is similar to my previous post so please review it, as I will not describe common stuff. This tutorial I think for advanced developers.

Before start

First you should read the article how to build EVR Presenter on MSDN especially prerequisites part.

Multimedia threading VS .NET threading

Yes, we should understand that stuff because .NET threading is totally different. MSDN description is good but not enough.

.NET threads and objects are all works and exist in specified execution context. Some objects are not available to be accessed from different threads for example form and controls. Other objects can be accessed from another thread but that require .NET to switch between execution contexts (there object created and there it is accessed). That operation can take a while or even hang the execution. Hanging can appear due different threading model. As in our case ALL multimedia threads are works in same context so you can access object created in other thread without any problem, only don’t forget about synchronization object. Hope I clear the difference and we can proceed.

Tracing, Debugging and Exception

I want to say some words regarding that before reviewing the code. Due different threading stuff I not suggest you to use the things like Trace.Write or Debug.Write in any code which processing multimedia data or accessing unmanaged resources frequently. This is also related to issue with switching threading context and as result degrade the performance. Raising exceptions are not recommended in the same issue, so better to checking returned values, using try catch statements also recommended. The way to solve tracing output is to use of OutputDebugString API.

public static void TRACE(string _message) 
{ 
    if (!string.IsNullOrEmpty(_message)) _message += "\n"; 
    API.OutputDebugString(_message); 
}

Another helper function which will be useful along with above

public static void TRACE_ENTER() 
{ 
    MethodBase _method = (new StackTrace(1,false)).GetFrame(0).GetMethod(); 
    TRACE(string.Format("{0}::{1}", _method.ReflectedType.Name, _method.Name)); 
}

This function print to an output window caller class name and method name.

Application Overview

Demo application shows how to perform video playback using DirectShow with Enhanced Video Renderer (EVR) with custom presenter. Presenter performing allocating the surfaces for playback, performing media type negotiations, synchronization of surfaces time stamps and display frames to the user using Direct3D9. Presenting surfaces done using SlimDX (managed library for Direct3D) similar as in my previous article.

Implementation scene presenting

Presenting the scene is similar to here.

Filter Graph

Filter graph a little different but it also particular graph for playback application, just it used the Enhanced Video Renderer instead of default and look like this:

Class declaration and initialization

Same way I use my classes for graph building so the playback class declaration looks:

public class DSFilePlaybackEVR : DSFilePlayback
    , IMFVideoDeviceID 
    , IMFVideoPresenter 
    , IMFGetService 
    , IMFTopologyServiceLookupClient

Here the inherited interfaces are required for the custom EVR presenter. We also have an event delegate and event variable in class to notify the scene that the surface is ready for display. The EVR filters declaration is:

[Guid("FA10746C-9B63-4b6c-BC49-FC300EA5F256")] 
public class EVRRenderer : DSFilter 
{ 
    public EVRRenderer() 
        : base() 
    { 
    } 
}

EVR Class derived from base graph builder class which handles all basic stuff for playback via DirectShow we just need to override methods for initialization filters and connecting them:

protected override HRESULT OnInitInterfaces() 
{ 
    m_evStop.Reset(); 
    HRESULT hr; 
    hr = (HRESULT)MFHelper.DXVA2CreateDirect3DDeviceManager9(out m_DeviceResetToken, out m_DeviceManager); 
    hr.Assert(); 
    if (hr.Succeeded) 
    { 
        hr = (HRESULT)m_DeviceManager.ResetDevice(Marshal.GetObjectForIUnknown(m_Device.ComPointer), m_DeviceResetToken); 
        hr.Assert(); 
    } 
    m_Caller = new Wrapper(this);  
    m_Renderer = new EVRRenderer(); 
    IMFVideoRenderer _renderer = (IMFVideoRenderer)m_Renderer.QueryInterface(typeof(IMFVideoRenderer)); 
    hr = (HRESULT)_renderer.InitializeRenderer(null, (IMFVideoPresenter)this); 
    hr.Assert(); 
    m_Renderer.FilterGraph = m_GraphBuilder; 
    hr = base.OnInitInterfaces(); 
    hr.Assert(); 
    return hr; 
}

Code fairly simple: we initialize DXVA2 device manager, create EVR filter and put it into the graph. After performing initialization EVR filter with our Presenter.

Implementing Presenter Interfaces

Now time for hardest part and you will know why I describe things regarding threading.

Invoker

Some methods of interfaces provided by my class for EVR Presenter is called from different threads, as if there will be one thread we will have no issues. But in there at least 2, commonly 3: user interaction (Play Pause Stop), media streaming (Samples delivering) and Clock (Samples Synchronization). Interfaces which are called at same context are IMFVideoDeviceID and IMFGetService. With other intarfaces we should do something to make it work properly, how to do so? The answer is simple: to make them to be called in same thread so context will be same. Hope you good enough with threading and synchronization to understand following code. Let’s look how implemented IMFTopologyServiceLookupClient:

public int InitServicePointers(IntPtr pLookup)
{
    Wrapper.CCallbackHandler _handler =
                 new Wrapper.InitServicePointersHandler(m_Caller, pLookup);
    _handler.Invoke();
    return _handler.m_Result;
}

public int ReleaseServicePointers()
{
    Wrapper.CCallbackHandler _handler =
                 new Wrapper.ReleaseServicePointersHandler(m_Caller);
    _handler.Invoke();
    return _handler.m_Result;
}

Here we are simplify creates the specified pre-defined async invoker, wait for it result and return it. The invoker callback base class looks next:

public class CCallbackHandler
{
    public bool m_bAsync = false;
    public CallType m_Type = CallType.Unknown;
    public EventWaitHandle m_Notify = new ManualResetEvent(false);
    public int m_Result = S_OK;
    private Wrapper m_Invoker = null;

    #region Constructor

    public CCallbackHandler(Wrapper _Invoker)
    {
         m_Invoker = _Invoker;
    }

    #endregion

    #region Methods

    public void Invoke()
    {
         m_Invoker.InvokeThread(this);
         WaitHandle.WaitAny(new WaitHandle[] { this.m_Notify, m_Invoker.m_Quit });
    }

    #endregion
}

And the actual invokers thread methods:

private void InvokeThread(object _param)
{
	lock (m_LockThread)
	{
         		m_Parameter = _param;
                  m_Notify.Set();
	}
	WaitHandle.WaitAny(new WaitHandle[] { m_Quit, m_Ready });
}

private void ThreadProc(object _state)
{
    while (true)
    {
                 int nWait = WaitHandle.WaitAny(new WaitHandle[] { m_Quit, m_Notify });
                 if (nWait == 1)
                 {
                         object _param;
                         lock (m_LockThread)
                         {
                                _param = m_Parameter;
                         }
                         m_Ready.Set();
                         AsyncInvokerProc(_param);
                 }
                 else
                 {
                          break;
                 }
    }
}

We put the caller object as parameter after set the notify event which wakes up the thread to process the parameter and waits until the parameter will be retrieved. The thread got the parameter and executes passed callback; so all access to managed resources become from single thread.

Advanced Marshaling

Hope you still here; as previous part was not last hard code. Once the method is called with the invoker in same thread we can without any problems hold COM interfaces in our class. But someone who try to make EVR Presenter in .NET I’m sure had an issue with the InitServicePointers method of IMFTopologyServiceLookupClient interface, right? The issue appear, as I remember (I wrote that code year or two ago) was with query IMFTopologyServiceLookUp from pLookUp. Issue happened because of COM have aggregation and the returned interface may not be the interface of an actual object and the .NET doesn’t handle that it just call the QueryInterface and fail if it not in there, but we are know it is here. To solve this we just can access the vtable (table of virtual methods) of interface we interested in. All entry in that table are the pointers to the functions in order of interface inheritance and interface methods (actually interface it is structure with the specified methods entries and nothing else). As an example first 3 entries in interface are always implementation of IUnknown in order QueryInterface, AddRef and Release, That is true for all managed interfaces too, plus also for managed objects, as all managed objects are COM objects by default (that just hidden from developers). So I made the helper class which allows accessing COM object by it vtable:

public class VTableInterface : COMHelper,IDisposable
{
    #region Delegates

    private delegate int QueryInterfaceProc(
        IntPtr pUnk,
        ref Guid riid,
        out IntPtr ppvObject
        );

    #endregion

    #region Variables

    protected IntPtr m_pUnknown = IntPtr.Zero;

    #endregion

    #region Constructor

    protected VTableInterface(IntPtr pUnknown)
    {
        if (pUnknown != IntPtr.Zero)
        {
            m_pUnknown = pUnknown;
            Marshal.AddRef(m_pUnknown);
        }
    }
    ~VTableInterface()
    {
        Dispose();
    }

    #endregion

    #region Methods

    public int QueryInterface(ref Guid riid, out IntPtr ppvObject)
    {
        ppvObject = IntPtr.Zero;
        if (m_pUnknown == IntPtr.Zero) return E_NOINTERFACE;
        QueryInterfaceProc _Proc = GetProcDelegate<QueryInterfaceProc>(0);
        if (_Proc == null) return E_UNEXPECTED;
        return (HRESULT)_Proc(m_pUnknown,ref riid,out ppvObject);
    }
    #endregion

    #region Helper Methods

    protected T GetProcDelegate<T>(int nIndex) where T : class
    {
        IntPtr pVtable = Marshal.ReadIntPtr(m_pUnknown);
        IntPtr pFunc = Marshal.ReadIntPtr(pVtable, nIndex * IntPtr.Size);
        return (Marshal.GetDelegateForFunctionPointer(pFunc, typeof(T))) as T;
    }

    #endregion

    #region IDisposable Members

    public void Dispose()
    {
        if (m_pUnknown != IntPtr.Zero)
        {
            Marshal.Release(m_pUnknown);
            m_pUnknown = IntPtr.Zero;
        }
    }
    #endregion
}

The main interesting method here is GetProcDelegate which allows getting method from vtable by it index. How it works you can see in QueryInterface implementation. So to implement IMFTopologyServiceLookUp without any problems we make next code:

public class MFTopologyServiceLookup : VTableInterface, IMFTopologyServiceLookup
…
private delegate int LookupServiceProc(
            IntPtr pUnk,
            MFServiceLookUpType Type,
            uint dwIndex,
            [In, MarshalAs(UnmanagedType.LPStruct)] Guid guidService,
            [In, MarshalAs(UnmanagedType.LPStruct)] Guid riid,
            [Out, MarshalAs(UnmanagedType.LPArray, ArraySubType = UnmanagedType.SysInt)] IntPtr[] ppvObjects,
            [In, Out] ref uint pnObjects);
….
public int LookupService(MFServiceLookUpType _type, uint dwIndex, Guid guidService, Guid riid, IntPtr[] ppvObjects, ref uint pnObjects)
{
            if (m_pUnknown == IntPtr.Zero) return E_NOINTERFACE;
            LookupServiceProc _lookUpProc = GetProcDelegate<LookupServiceProc>(3);
            if (_lookUpProc == null) return E_UNEXPECTED;
            return (HRESULT)_lookUpProc(
                        m_pUnknown,
                        _type,
                        dwIndex,
                        guidService,
                        riid,
                        ppvObjects,
                        ref pnObjects
                        );
}

Not so hard I think. Forgot to mention the interface delegate function differ from method declaration in interface in one additional argument. First argument should be the pointer to the vtable object or our IntPtr, why that necessary you can find over web I think.

Samples Scheduler and notify of free sample

If you look at the EVR Presenter C++ example from Microsoft you can find that it define free samples while it released, I mean called Release with specified notification set on that. .NET doesn’t allow us to use that method as we have no access to the IUnknown directly. So I solve that with usage of events (probably for someone better to use semaphores, but this is just an example).

Main Application

Implementation of main form is very easy, most interesting methods I describe here. Variable declaration for scene and playback:

private Scene m_Scene = null; 
private DSFilePlayback m_Playback = null;

Creating scene object:

m_Scene = new Scene(this.pbView);

Here pbView control on which we’ll do presenting the video. Starting playback code:

m_Playback = new DSFilePlaybackEVR(m_Scene.Direct3DDevice); 
m_Playback.OnPlaybackStop += new EventHandler(btnStart_Click); 
((DSFilePlaybackEVR)m_Playback).OnSurfaceReady += new EVR.SurfaceReadyHandler(m_Scene.OnSurfaceReady); 
m_Playback.FileName = this.tbFileName.Text; 
if (m_Playback.Start().Succeeded) 
{ 
    btnStart.Text = "Stop"; 
    btnBrowse.Enabled = false; 
}

In this code we create EVR rendering graph with specified Scene device. After we provide event handler for surface ready notify, and starting playback. Stopping code is fairly simple – just Dispose the playback:

m_Playback.Dispose(); 
m_Playback = null;

History

Initial Version 11-07-2012

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

Maxim Kartavenkov
Software Developer (Senior)
Russian Federation Russian Federation
I'm a professional multimedia developer (more than 10 years) in any kind of applications and technologies related to it, such as DirectShow, Direct3D, WinMM, OpenGL, MediaFoundation, WASAPI, Windows Media and other including drivers development of Kernel Streaming, Audio/Video capture drivers and audio effects. Have experience in following languages: C, C++, C#, delphi, C++ builder, VB and VB.NET. Strong knowledge in math and networking.

Sign Up to vote   Poor Excellent
Add a reason or comment to your vote: x
Votes of 3 or less require a comment

Comments and Discussions

 
You must Sign In to use this message board.
Search this forum  
    Spacing  Noise  Layout  Per page   
QuestionDXVA Hardware Decodingmemberdamikez27-Jan-13 15:31 
Thanks for the sample! I'm learning a lot! Were you able to figure out how to get DXVA Hardware Decoding running on this?
AnswerRe: DXVA Hardware DecodingmvpMaxim Kartavenkov6-Feb-13 20:40 
Yes you can configure and use DXVA2 decoding with that sample.
QuestionMemory Leakmember2fast4all16-Nov-12 1:32 
Hallo Maxim,
 
first at all: thank you for that great piece of code Smile | :) .
 
I got a little Problem with it: I tried to use the code for a tiny media-player which plays a 10sec. media file (DNxHD Codec - LAV Splitter as decoder...) in a loop, but it doesn`t run very long until it consumes a huge ammount of memory.
 
I also reproduced this behaviour with your sample application, do you have any idea?
 
Best regards,
Andi
AnswerRe: Memory LeakmemberMaxim Kartavenkov16-Nov-12 4:32 
Not sure that your memory leaks related to my code - as you can see in there I allocate fixed number of render targets and use them along playback - so memory allocation perfomed one time.
 
Try use other spliters or decoders in your playback probably issue is in there. Try to reproduce graph in GraphEdit.
 
Maxim.
GeneralRe: Memory Leakmember2fast4all16-Nov-12 4:54 
Thank you for your fast reply. As you say, maybe it`s not in your code: i tried it out with GraphEdit, and it also leaks, but only if I use EVR as renderer not with VMR.
 
The funny thing: First at all I wanted to use the WMP-OCX but this also leaks badly - and seemingly ther`s no solution for that (there are many other guys having the same problem with WMP-OCX...).
 
So for me the only solution which doesn`t leak is rendering with Quartz and VMR, but this has some side effects in conjunction with Windows 7s Aero (if Aero is turned on, the interpolation is ugly if the video resolution doesn`t match screen resolution exactly).
 
Andi
GeneralRe: Memory LeakmemberMaxim Kartavenkov16-Nov-12 5:02 
Try to unregister one by one the filter used by default for your file playback, so graph manager will be using other ones instead of them and you will figure onu which filter eat memory.
 
Maxim.
GeneralMy vote of 5memberBavarian12-Nov-12 8:14 
great work
QuestionDXVA2 decodermemberghiaccio19-Oct-12 5:16 
Thank you for excellent example code.
 
I want to play h264 or wmv with DXVA2 decoder.
This code use EVR, but DXVA2 decoder device was not created,
using software decoder(checked with DXVA_checker_2.9.1).
 
DXVA2 decoder device was created when I use GraphEdit player,
using same filter graph.(checked m_playback[n].name)
 
Is any solution known?
AnswerRe: DXVA2 decodermemberMaxim Kartavenkov19-Oct-12 5:38 
hello,
 
How are you determine if dxva2 used or not?
 
dxva used in case if the decoder uses it. If decoder supports dxva it usually query for upstream filter Direct3D device. And create query caps and the allocator based on it. I didn't check dxva in that sample so I can only suggest you to figure that.
Following things will help you find the solution:
Check if the connected decoder supports dxva2.
Check device creation settings - maybe necessary to change configuration in there.
Check media types negotiations as in sample is uses swap chains, maybe better to change that with dxva surface and other formats and performs blitting to a target surface before render.
 
Regards,
Maxim.
QuestionRe: DXVA2 decodermemberghiaccio22-Oct-12 2:23 
Thank you for reply.
 
For determine if DXVA2 use or not, I used DXVA checker's Trace log.
http://bluesky23.yu-nagi.com/en/DXVAChecker.html[^]
 
I checked mp4(avc) file with below filter graph.
source(mp4/avc) -> Haali Splitter ->Microsoft DTV-DVD Decoder -> EVR
 
Graphedit trace log
DXVA2_DecodeDeviceCreated, graphedt, 00:00:02.2734124
DXVA2_DecodeDeviceBeginFrame, graphedt, 00:00:02.3222269
DXVA2_DecodeDeviceGetBuffer, graphedt, 00:00:02.3224265
DXVA2_DecodeDeviceGetBuffer, graphedt, 00:00:02.3224391
DXVA2_DecodeDeviceExecute, graphedt, 00:00:02.326083
created dxva2 bob device,processor device, and ModeH264_VLD_NoFGT(DXVA2 decoder device).
 
EVRPlayback trace log(same mp4 source, and same filter-graph)
DXVA2_ProcessDeviceCreated, EVRPlayback.vshost, 00:00:06.0879228
DXVA2_ProcessDeviceCreated, EVRPlayback.vshost, 00:00:06.0880062
DXVA2_ProcessBlt, EVRPlayback.vshost, 00:00:06.0908226
DXVA2_ProcessBlt, EVRPlayback.vshost, 00:00:06.1158033
DXVA2_ProcessBlt, EVRPlayback.vshost, 00:00:06.1163385
DXVA2_ProcessBlt, EVRPlayback.vshost, 00:00:06.1290979
 
created dxva2 bob device and processor device.
Although a part of dxva2 functions are used, it use software decoder.
 
I would like to know whether EVRPlayback can use dxva2 decoder.
I guess it is necessary to implement some additional interfaces( IDirectXVideoDecoderService,
IDirectXVideoAccelerationService..)
AnswerRe: DXVA2 decodermemberMaxim Kartavenkov22-Oct-12 4:23 
Hi,
Not sure in that necessary to make that interfaces. Maybe expose them over GetService.
Try to research in things I listed in previous reply.
 
Regards,
Maxim.
QuestionEVR hangs while playing large .ts filememberrevengeoffallen10-Aug-12 3:17 
Hi,
I am facing tough problem out here. We had to switch from VMR9 to EVR for performance and quality reasons.
When we play large .ts files, and move slider 3-4 times, entire app hangs. Same thing happened even on GraphEdit.
Video Decoder : AVC H264,
Audio Decoder : AAC
please help
AnswerRe: EVR hangs while playing large .ts filememberMaxim Kartavenkov10-Aug-12 3:33 
This can be bcs of one of the following issues: mpeg spliiter used is not good or H264 decoder is not good. Send me your GraphEdit screen shots. Try to use ffdshow decoder - it's works on CPU but decoding properly in most cases, also try DivX H264 Decoder, for splitter try MPC Splitter. Also you can try to play your file in MPC as I remember it allows to try different renderers.
 
Regards,
Maxim.
QuestionVery nicememberAlois Kraus23-Jul-12 22:51 
This is a nice article. You are right that Debug and Trace stuff takes a global lock thus hurting performance but if you are writing it via OutputDebugString and you have a listener attached to it (the debugger or DbgView) it is much slower anyway.
At least it is much slower than to write directly to a file. In your next article you mention TRACE_ENTER which uses Reflection whic is veery slow. Much slower than to take a lock. There are ways to improve perf via using GetStackFramesInternal or by passing the method name as plain string.
In fact I have written a tracing library here which should suit your perf needs.
 
You seem to have written your own SynchronizationContext class implementation. I think it should be possible to set for the threads that require marshalling your own derived Synchronization context class and use this for marshalling.
Yours,
Alois Kraus

AnswerRe: Very nicememberMaxim Kartavenkov8-Aug-12 17:32 
Hi,
Sure any debugging stuff slow down the performance either in .NET or in native code. But that is only for testing purpose and in DEBUG build. Writing directly to a file is slower than using OutputDebugString. Reflection not so slow as usage Debug and Trace classes and works fine but sure necessary to think there to put it in code and remove once it not required.
Yes in here implemented common context thread stuff, but with some modifications with COM marshaling it can be removed but then will be bit harder for testing.
 
Maxim.

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Permalink | Advertise | Privacy | Mobile
Web04 | 2.6.130617.1 | Last Updated 12 Jul 2012
Article Copyright 2012 by Maxim Kartavenkov
Everything else Copyright © CodeProject, 1999-2013
Terms of Use
Layout: fixed | fluid