Accessing an API through a closed source C++ DLL

• Download source code - 10.6 KB

Introduction

This article describes a useful technique for using a closed source C++ DLL, loaded at run-time, to access an API for a popular consumer peripheral. It’s assumed the developer does not have access to import libraries or source code.

Background

This is a technical follow-up to my Continuum blog post describing how I automated a simple software task to enable an automatic pan-and-zoom feature of a consumer webcam that didn't provide an API. We needed to do this without disturbing the user experience, so standard program automation tools like AutoIt were off the table.

Throughout the process, I realized some of the pitfalls in making calls on C++ DLLs and identified workarounds. The inspiration for this article was a blog post from Recx Ltd, Working with C++ DLL Exports without Source or Headers. An example project is provided, but our specific case relied on a proprietary DLL that could not be included due to licensing.

API Monitor is an invaluable tool for monitoring the activity of API calls. It has more comprehensive features for Windows API calls, but it can also be used to view calls to external DLLs. You can spawn processes
from within the tool or attach to a running process. The state of the call stack can be seen before and after API calls are made, return values are visible, and it even provides breakpoint functionality.

With API Monitor, I spawned an instance of the web cam controller application, which allowed me to see which DLLs were being loaded. Monitoring from process execution (as opposed to attaching to running process) can be important when trying to see initialization behavior.

The module dependency view revealed a large list of DLLs. Most of them were Windows system DLLs or related to the Qt framework. One in particular stood out for what we needed: CameraControls_Core.dll. I set up the API Monitor to log all calls to this DLL and this was the relevant output:

LWS::CameraControls::CameraControls_Core::Init ( ... )
LWS::CameraControls::CameraControls_Core::GetCurrentVideoDevice ( ... )
LWS::CameraControls::CameraControls_Core::SetCurrentVideoDevice ( ... )
LWS::CameraControls::CameraControls_Core::GetFaceTracking ( ... ) 

1 0 00001366 ??0CameraControls_Core@CameraControls@LWS@@QAE@XZ
2 1 000015C3 ??1CameraControls_Core@CameraControls@LWS@@UAE@XZ
3 2 00038740 ??_7CameraControls_Core@CameraControls@LWS@@6B@   

I ran the output of the dumpbin tool into Microsoft’s undname utility to undecorate the names.

This provided a more coherent output:

1    0 00001366 public: __thiscall LWS::CameraControls::CameraControls_Core::CameraControls_Core(void)
2    1 000015C3 public: virtual __thiscall LWS::CameraControls::CameraControls_Core::~CameraControls_Core(void)
3    2 00038740 const LWS::CameraControls::CameraControls_Core::`vftable'
...
40   27 000012E9 public: bool __thiscall LWS::CameraControls::CameraControls_Core::Init(class QString)  
...
86   55 000016D1 public: long __thiscall LWS::CameraControls::CameraControls_Core::SetFaceTracking(unsigned long,long) 

It was around this point in time that I stumbled on the above mentioned Recx Ltd article. I realized that our particular DLL would present a few challenges that made a simple script-kiddie application of this technique impossible. I moved a copy of the DLL into my application’s working directory and attempted to write some code to load it dynamically.

static HINSTANCE CameraControlDllHandle = NULL;
CameraControlDllHandle = LoadLibrary(L"CameraControls_Core.dll"); 

Stepping through the code revealed the first dependency library of many: Qt4Gui4. As the errors presented themselves, I copied the dependencies into the working directory. It turns out ten additional DLLs were required. I ran through the code one last time, and got an error about MSVCR90.DLL being missing.

Placing this DLL in the directory results in an error about the C-runtime being loaded improperly.

MSVCR90.DLL is the C-runtime library for Visual C++ 2008. I tried re-building my project to run with this runtime, so it would use the same runtime as the DLL, but it didn't mitigate the problem. As it turns out, Microsoft introduced a new form of system-wide DLL management in Windows 98 for allowing conflicting DLLs to exist simultaneously in memory (Wikipedia DLL Hell). DLL’s complying with this standard have a manifest file that is imported into the DLL to inform a calling process which dependencies need to be loaded.

To overcome the C run-time dependency, I needed to create a manifest file pointing to the run-time library and embed it into my CameraControls_Core.dll. This MSDN article outlines how to use Microsoft’s MT.EXE utility to embed a manifest file into an already built executable or DLL.

The manifest file I needed to use for MSVCR90.DLL is as follows:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
  <dependency>
    <dependentAssembly>
      <assemblyIdentity type="win32" name="Microsoft.VC90.DebugCRT" 
        version="9.0.21022.8" processorArchitecture="x86" 
        publicKeyToken="1fc8b3b9a1e18e3b"></assemblyIdentity>
    </dependentAssembly>
  </dependency>
  <dependency>
    <dependentAssembly>
      <assemblyIdentity type="win32" name="Microsoft.VC90.CRT" 
        version="9.0.21022.8" processorArchitecture="x86" 
        publicKeyToken="1fc8b3b9a1e18e3b"></assemblyIdentity>
    </dependentAssembly>
  </dependency>
</assembly>

After this step, the library was loading into memory without exceptions! Now, I just needed to verify method calls would work. Early attempts at calling essentially any of the methods were throwing exceptions. I realized this was because some critical init function was probably not being called. The reader should read through Working with C++ DLL Exports without Source or Headers first.

 #include <QString>
//------------------------------------------------------------------------------
// Function Pointer Typedefs
//------------------------------------------------------------------------------
typedef long (__thiscall *_SetFaceTracking)(DWORD *pThis, unsigned long, long boolValue);
typedef long (__thiscall *_GetFaceTracking)(DWORD *pThis, unsigned long, long *boolValue);
typedef void (__thiscall *_CameraControls_Core)(DWORD *pThis);
typedef bool (__thiscall *_Init)(DWORD *pThis, class QString param1);
typedef long (__thiscall *_GetCurrentVideoDevice)(DWORD *pThis, unsigned long *param);
typedef long (__thiscall *_SetCurrentVideoDevice)(DWORD *pThis, unsigned long param);
//--------------------------------------------------------------------------------------------
// Static Variables
//--------------------------------------------------------------------------------------------
static BOOL            freeResult;
static _SetFaceTracking    CameraSetFaceTracking;
static _GetFaceTracking     CameraGetFaceTracking;
static DWORD            dwFakeObject[512];
static HINSTANCE        CameraControlDllHandle = NULL;
static unsigned long    currentVideoDevice = 0;
PRINCEDLL_API HRESULT APIENTRY InitLibrary()
{
    long retValue = 0;
    QString inputString = "";
 
    CameraControlDllHandle = LoadLibrary(L"CameraControls_Core.dll");
    
    if (CameraControlDllHandle == NULL)
        return E_FAIL;
    _CameraControls_Core CameraImportedConstructor = (_CameraControls_Core) GetProcAddress(
      CameraControlDllHandle,"??0CameraControls_Core@CameraControls@LWS@@QAE@XZ");
    if (CameraImportedConstructor == NULL)
        return E_FAIL;
    CameraSetFaceTracking = (_SetFaceTracking) GetProcAddress (CameraControlDllHandle,
      "?SetFaceTracking@CameraControls_Core@CameraControls@LWS@@QAEJKJ@Z");
    if (CameraSetFaceTracking == NULL)
        return E_FAIL;
    CameraGetFaceTracking = (_GetFaceTracking) GetProcAddress (CameraControlDllHandle, 
      "?GetFaceTracking@CameraControls_Core@CameraControls@LWS@@QAEJKPAJ@Z");
    if (CameraGetFaceTracking == NULL)
        return E_FAIL;
    _Init CameraInitializer = (_Init) GetProcAddress (CameraControlDllHandle, 
      "?Init@CameraControls_Core@CameraControls@LWS@@QAE_NVQString@@@Z");
    if (CameraInitializer == NULL)
        return E_FAIL;
    _SetCurrentVideoDevice CameraSetCurrentVideoDevice = (_SetCurrentVideoDevice) GetProcAddress (
      CameraControlDllHandle, "?SetCurrentVideoDevice@CameraControls_Core@CameraControls@LWS@@QAEJK@Z");
    if (CameraSetCurrentVideoDevice == NULL)
        return E_FAIL;
    _GetCurrentVideoDevice CameraGetCurrentVidDevice = (_GetCurrentVideoDevice) GetProcAddress (
      CameraControlDllHandle, "?GetCurrentVideoDevice@CameraControls_Core@CameraControls@LWS@@QAEJPAK@Z");
    if (CameraGetCurrentVidDevice == NULL)
        return E_FAIL;
    memset (dwFakeObject, 0x00, 512);
    CameraImportedConstructor(dwFakeObject);
    retValue = CameraInitializer(dwFakeObject, inputString);
    retValue = CameraGetCurrentVidDevice (dwFakeObject, &currentVideoDevice);
    retValue = CameraSetCurrentVideoDevice (dwFakeObject, currentVideoDevice);
    return S_OK;
} 

In __cdecl, all the parameters are passed on the call stack. The object instance this pointer is passed last. Our DLL used the __thiscall convention. __thiscall, it turns out, informs the callee that the this pointer is being passed via the ECX register (not on the call stack). The __thiscall keyword tells the compiler that the first parameter should be placed in the ECX register. Technically, the first parameter should be a pointer to the object instance.

Since our DLLs Init() function requires a QString as a parameter, I needed to determine the version of Qt being used by the target DLL, build Qt, and statically link it to my DLL. Fortunately, passing an empty QString to the Init function completed initialization without errors and allowed the other methods to be called. The dwFakeObject array is required because C++ instance methods expect to be passed a reference to the object they are designed to work on. We reserve an area in memory and treat this as a
reference to a dummy object that is passed to the methods within the DLL.

Points of Interest

Hopefully this will prove insightful to someone going through a similar situation with attempting to use third party DLLs. Without access to source for the definitions of the underlying interface and data
types, the problem can be more or less complex than this example. However, it serves to illustrate that there isn’t always a general purpose solution for accessing DLL functions and some cases will require an ad-hoc approach.

Topics touched on:

• Embedded DLL Manifest files
• Viewing export functions and making calls from C++ DLLs
• APIMonitor as a tool for spying on API calls
• Calling conventions

History

17 Jan 2013 - Initial publication.