LeakMon: Part 3 - Opening a file in Visual Studio through automation

This is the 3^rd post related to the LeakMon tool. In this post, we will see how the DumpViewer (DumpViewer is an application used to analyze the leak dump created by LeakMon) is able to open a particular file in Visual Studio and highlight the specified line.

Step 1: Finding Out the Version of Visual Studio to Use

It is 2011 now and we have Visual Studio 2010. But there are people or companies who still use Visual Studio 6. Things have changed a lot from Visual Studio 6 to Visual Studio 10, even the EXE name has changed. So the first task is to find out which version of Visual Studio to use.

DumpViewer uses the same Visual Studio version that is registered as the default program for documents such as .c, .cpp, .h, .hpp on your machine. For instance, say we have the following line shown in DumpViewer:

func2 t:\naveen\pgms\cpp\2008\mulitthread\mulitthreaddlg.cpp(178)

DumpViewer will extract the file path and line number from the above line and pass it to the FindExecutable function. This function will return the path of executable that is registered for the above file types, which will usually be Visual Studio. If it is not Visual Studio, sorry, it will fail!

In the file name of the executable returned by FindExecutable function, we will check whether there is a string "msdev.exe" or "devenv.exe". If there is "msdev.exe" in the string, code that does the automation of Visual Studio 6 is called, otherwise the code to handle higher version of Visual Studio is called.

const LPCTSTR VC_6_APP_NAME = _T("msdev.exe");
const LPCTSTR VC_8_APP_NAME = _T("devenv.exe");
if( 0 == csExe.CompareNoCase( VC_6_APP_NAME ))
{
 // launch vc6
 OpenUsingVS<IApplication>( IID_IDispatch, L"MSDEV.Application",csFileName_i, nLineNo );
}
else if( 0 == csExe.CompareNoCase( VC_8_APP_NAME ) )
{
 // launch vs7 or any other higher versions of VS
 OpenUsingVS<_DTE>( EnvDTE::IID__DTE, 
   L"VisualStudio.DTE",csFileName_i, nLineNo );
}

Step 2: Getting a Pointer to Already Running Instance of Visual Studio

Since we are going to open a file through automation, we need a pointer of either IApplication or _DTE (IApplication for Visual Studio 6 and _DTE for Visual Studio 7 and higher). We can create an instance of the above type using the CoCreateInstance by passing the IID and CLSID. But this will create a new instance of Visual Studio each time we try to open a new file. So what if we want is to get a pointer to IApplication / _DTE of an already running instance of Visual Studio if any. This can be done by enumerating the Moniker in the ROT (Running Object Table).

template<class T>
bool OpenUsingVS( IID riid, CString csProgID, CString csFileName_i, int nLineNo)
{
    HRESULT hRes; 
    CLSID clsid; 
    CComPtr<IUnknown> punk; 
    CComPtr<T> dte; 
    RETURN_ON_FAIL( ::CLSIDFromProgID( csProgID.operator LPCTSTR(), &clsid) ); 

    // Search through the Running Object Table for an instance of Visual Studio 
    // to use that either has the correct solution already open or does not have  
    // any solution open. 
    CComPtr<IRunningObjectTable> ROT; 
    RETURN_ON_FAIL( GetRunningObjectTable( 0, &ROT ) ); 

    CComPtr<IBindCtx> bindCtx; 
    RETURN_ON_FAIL( CreateBindCtx( 0, &bindCtx ) ); 

    CComPtr<IEnumMoniker> enumMoniker; 
    RETURN_ON_FAIL( ROT->EnumRunning( &enumMoniker ) ); 

    CComPtr<IMoniker> dteMoniker; 
    RETURN_ON_FAIL( CreateClassMoniker( clsid, &dteMoniker ) ); 

    CComPtr<IMoniker> moniker; 
    CComPtr<IMoniker> moniker2; 
    ULONG monikersFetched = 0; 
    while ( enumMoniker->Next( 1, &moniker, &monikersFetched ) == S_OK) 
    { 
        moniker2 =NULL;
        moniker->Reduce( bindCtx, MKRREDUCE_ALL, NULL, &moniker2 );
        IMoniker* pMon = (0 != moniker2)?moniker2:moniker;
        if ( moniker2->IsEqual( dteMoniker ) ) 
        { 
            hRes = ROT->GetObject( moniker, &punk ); 
            if ( hRes == S_OK ) 
            { 
                dte = punk; 
 
                if ( dte ) 
                { 
                    // We have got an instance pointer. Let's try to open
                    // the file using that pointer
                    if( OpenFile( dte,csFileName_i, nLineNo ))
                        return true;
                    dte = 0;
                } 
            } 
            punk = NULL; 
        } 
        moniker = NULL; 
    } 

    if ( !dte ) 
    {                
        // we didn't get any pointer from ROT. So let's go and create a new
        // instance.
        RETURN_ON_FAIL( ::CoCreateInstance
	( clsid, NULL, CLSCTX_LOCAL_SERVER, riid , (LPVOID*)&punk ) ); 
        dte = punk; 
        if ( !dte ) 
            return false; 
        OnLaunch( dte );
        OpenFile( dte,csFileName_i, nLineNo );
    }
    return true;
}

As you can see in the above code, if we failed to get a valid instance pointer of Visual Studio, we create a new one using CoCreateInstance. In Visual Studio 7 and higher, as soon as we release the com pointer that we got from CoCreateInstance, Visual Studio will terminate. To prevent Visual Studio from exiting like this, we need to call put_UserControl of DTE interface.

Step 3: Opening a File and Selecting a Line from it

From this part on, the processing is different for Visual Studio 6 and other versions of Visual Studio.

In Visual Studio 6

#define dsWindowStateMaximized 1
bool OpenFile(IApplication* m_spApplication,CString csFileName_i, int nLineNo )
 {
    CComPtr<IDispatch> pDispatch;
    // Make it visible just in case if it is not
    m_spApplication->put_Visible( VARIANT_TRUE );
    // Maximize the window
    m_spApplication->put_WindowState( dsWindowStateMaximized );
    // Get a pointer to the IDocument interface
    m_spApplication->get_Documents( &pDispatch );
    CComQIPtr<IDocuments> pDocs = pDispatch;
    CComVariant type="Auto";
    CComVariant read="False";
    pDispatch = 0;
    // The Open function of IDocument opens the file.
    pDocs->Open( CComBSTR(csFileName_i),type, read, &pDispatch );
    // Now for selecting a line, we need to get pointer to ITextSelection
    // Interface
    CComQIPtr<ITextDocument> pDoc = pDispatch;
    pDispatch = 0;
    pDoc->get_Selection( &pDispatch );
    CComQIPtr <ITextSelection> pTextSelection = pDispatch;
    CComVariant varReserved( FALSE );
    // now just set the cursor to the correct line and call SelectLine of ITextSelection
    pTextSelection->StartOfDocument( varReserved );
    pTextSelection->LineDown( varReserved, CComVariant(nLineNo-1));
    Sleep( 1000 );
    pTextSelection->SelectLine();
    return true;
 }

In Visual Studio 7 and Higher

bool OpenFile( _DTE* pDTE,CString csFileName_i, int nLineNo )
{
    HRESULT hRes = 0; 
    CComPtr<ItemOperations> Operations;
    // Get the ItemOperations pointer
    RETURN_ON_FAIL(pDTE->get_ItemOperations( &Operations ));
    CComPtr<Window> Wnd;
    // Open the file using ItemOperations pointer
    RETURN_ON_FAIL(Operations->OpenFile( CComBSTR( csFileName_i),
                   CComBSTR(vsViewKindTextView), &Wnd ))
    
    // It might take some time to open the file.
    // so we will wait in a loop
    CComPtr<Document> pDoc;
    for( int nIdx =0; nIdx < 5; nIdx ++ )
    {
        pDTE->get_ActiveDocument( &pDoc );
        //Sleep( 100 );
        if( pDoc )
        {
            break;
        }
        if( nIdx == 4 )
        {
            return false;
        }
        Sleep( 100 );
    }
    // Active the main window. 
    CComPtr<Window> pMainWnd;
    RETURN_ON_FAIL(pDTE->get_MainWindow( &pMainWnd ))
    pMainWnd->Activate();
    
    // Get the TextSelection pointer
    CComPtr<IDispatch> pDisp = 0;
    RETURN_ON_FAIL(pDoc->get_Selection( &pDisp ));
    CComQIPtr<TextSelection> pSelection = pDisp;
    //GotoLine with VARIANT_TRUE will make the line selected
    pSelection->GotoLine( nLineNo, VARIANT_TRUE);
    return true;
}