Automated unit testing became very popular in the Java world and then marched victoriously into the .NET territory, thanks to an excellent tool called nUnit.
However, nUnit has one serious limitation: it works only with managed code. Good old C++ is not going anywhere, and we, C++ programmers, also want to enjoy the wonders of nice and easy automated unit testing.
GenTestAsm is the tool that makes it happen. It allows you to write unit tests in (unmanaged) C++, and then run them in nUnit.
Unit Testing C++ Code
When it comes to unit testing C++ code, there are essentially three choices:
- Do not do unit testing at all.
- Use one of the unit testing packages designed specifically for C++, e.g. TUT C++ unit test Framework.
- Find a way to run C++ tests in nUnit.
Not doing unit testing at all is a very risky approach. The code becomes brittle and the risk of making changes is too high. TUT is a nice tool, but it does not provide a GUI test runner like nUnit. Also, having to switch between two different tools for managed and unmanaged code looks like a nuisance.
Therefore, I concentrated on the last approach - finding a way to run C++ tests in nUnit.
The Battle Plan
The general battle plan was as follows:
- Make unmanaged tests callable from the outside world via DLL exports.
- Write a tool that takes an unmanaged DLL, enumerates its exports, and generates a managed assembly loadable by nUnit. I called this tool
- For each exported unmanaged function, automatically create a managed method marked with
- A managed method calls an unmanaged function via P/Invoke.
Enumerating DLL Exports
Sadly, Win32 does not provide out-of-the box API for enumerating DLL exports. Fortunately, the format of DLL files is publicly available from Microsoft. I extract the list of exports by opening the executable file and analyzing the bytes. It is a little tedious, but not a very complex task. The biggest annoyance is that the PE file format uses relative virtual memory addresses (RVAs) instead of file offsets. This is great when the file is loaded in memory, but requires constant recalculations when working with the file on disk.
Generating Test Assembly
To generate test assembly, I first create C# source code and then compile it using
CSharpCodeProvider class. This proved to be simpler and more straightforward than building the code through CodeDOM. This is also easier to test. If something goes wrong with the generated assembly, one can always look at the generated source code and scan it for abnormalities. I added an option to
GenTestAsm that outputs generated source code instead of compiled binary.
Test Exports vs. Other Exports
It is definitely possible that a DLL with unmanaged tests exports a function that is not a test. When
GenTestAsm creates the managed wrapper, it needs to know which exports are tests and which are not. nUnit separates tests from non-tests using attributes, but there are no attributes in the unmanaged world. I decided to use a simple naming convention instead.
GenTestAsm generates managed test wrappers only for the exports whose names begin with a certain prefix (by default
UnitTest). Other exports are ignored.
The next problem is how to handle test failures. In the nUnit world, a test is usually considered successful if it runs to completion, and failed if it throws an exception. Since my tests are written in unmanaged C++, their exceptions would be unmanaged C++ exceptions. I cannot let these exceptions escape into the managed wrapper. Therefore, I need some other mechanism to report test failures. I decided to use the test's return value. Unmanaged tests must have the signature:
Return value of
NULL means success, anything else means failure, and the returned
string is the error message. I chose
BSTR over regular
BSTR has well-defined memory management rules, and .NET runtime knows how to free it.
Writing a Trivial Test
BSTR from the C++ test is nice, but it makes writing a test a little difficult. The author of the test must make sure that unhandled C++ exceptions don't escape the test. He must also format the error message and convert it to
BSTR. If this were done by hand in each and every test, the code would become too verbose to be practical. Let's take a trivial test in C#:
public void CalcTest()
Assert.AreEqual( 4, Calculator.Multiply(2,2) );
and see how an equivalent test in C++ would look like:
int const expected = 4;
int actual = Calculator::Multiply(2,2);
if (expected != actual)
msg << "Error in " << __FILE__ << " (" << __LINE__ << "): "
<< "expected " << expected << ", but got " << actual;
return SysAllocString( msg.str().c_str() );
return SysAllocString("Unknown exception");
This is too much boiler plate code. We need a support library here.
With the help of a tiny
#include file we can squeeze our C+ test back to 3 lines of code:
ASSERT_EQUAL( 4, Calculator::Multiply(2,2) );
TEST macro that encapsulates the details of exception handling and
BSTR conversion. It also defines a couple of
ASSERT macros such as
The Big Lockdown
However, there is one catch. As you remember, I use P/Invoke to call my unmanaged tests. Internally, P/Invoke loads the unmanaged DLL and keeps it loaded until the managed process exits. In other words, if I used P/Invoke blindly, once you executed the tests, your managed DLL would become locked. You would not be able to recompile it until you closed nUnit GUI. This is an unpleasant speed bump.
One Way Out
Instead of invoking unmanaged DLL directly,
GenTestAsm could, of course, call
LoadLibrary(), and then
GetProcAddress(). It could then do
Marshal.GetDelegateForFunctionPointer() and invoke the resulting delegate. The problem is, this API is available only in .NET 2.0. I wanted
GenTestAsm to be compatible with .NET 1.1, so I had to find a different solution.
Another Way Out
If something must be loaded forever, let it not be the test DLL, but some other, helper DLL that never changes. Current version of
GenTestAsm P/Invokes into unmanaged helper (thunk), which then calls
FreeLibrary(). This way, it is the thunk that gets locked, while the real test DLL remains free.
typedef BSTR (*TestFunc)();
BSTR __cdecl RunTest( LPCSTR dll, LPCSTR name )
HMODULE hLib = LoadLibrary(dll);
if (hLib == NULL) return SysAllocString(L"Failed to load test DLL");
TestFunc func = (TestFunc)GetProcAddress(hLib, name);
if (func == NULL) return SysAllocString(L"Entry point not found");
BSTR result = func();
I put the thunk DLL as a resource into GenTestAsm.exe, and it is always written alongside generated managed assembly. Having two additional DLL files hanging around is a little annoying, but it is better than being unable to recompile your code.
Specifying Version of nUnit
GenTestAsm creates C# source code of the managed test assembly and then compiles it using .NET Framework C# compiler. The test assembly references nunit.framework.dll. The location of this DLL is specified in the gentestasm.exe.config file as follows:
value="C:\Program Files\NUnit 2.2\bin\nunit.framework.dll" />
Using nUnit for .NET Framework 2.0
If you use nUnit for .NET 2.0,
GenTestAsm may have difficulties working with it. You might get the following error when creating your managed assembly:
fatal error CS0009: Metadata file
'c:\Program Files\NUnit-Net-2.0 2.2.8\bin\nunit.framework.dll'
could not be opened -- 'Version 2.0 is not a compatible version.'
This error occurs because
GenTestAsm is a .NET 1.1 application, and by default uses .NET 1.1 C# compiler (when it is available). This compiler cannot reference an assembly created for a newer version of the Framework. To work around this problem, we must force
GenTestAsm to use .NET 2.0 libraries, including .NET 2.0 C# compiler. This is achieved by adding a
supportedRuntime element to the configuration file:
value="C:\Program Files\NUnit-Net-2.0 2.2.8\bin\nunit.framework.dll" />
GenTestAsm is a tool that allows to run unmanaged (typically, C++) tests in popular nUnit environment. A tiny support library provides authors of unmanaged tests with basic assertion facilities, similar to those of nUnit. With
GenTestAsm, a team can use a more uniform approach to unit testing of managed and unmanaged code. The same tool is used to run the tests, and test syntax is similar.