UTPP - Another Unit Test Framework

• Download utpp.zip - 25.5 KB
• GitHub repository

Introduction

There are countless articles explaining the benefits of testing but still there are people who find it too boring, too time-consuming or too difficult to implement. Let me get over this subject briefly before delving into the description of this particular framework.

Writing small tests should be the bread and butter of your workday as a software developer. While you design your system top-down looking first at the big picture and decomposing it into smaller tasks, you implement your system from bottom up, making or choosing small parts that you assemble into bigger and bigger assemblies. In this phase, tests are the scaffolding that holds together your yet unfinished building. You want to have the confidence that you made something durable before moving up to the next higher level. I find myself writing tests immediately after finishing a piece of code to verify that it does what I expected it to do and check different corner cases that would be hard to verify with the complete system. I also write tests later on in the integration phase or even after the system has been shipped in response to a bug report. In these cases, the test serves first to "illuminate" the bug and then to show that it has been solved by the code change. These tests serve as "guard rails" later on when an upgrade fails regression testing. Tests added in this phase also serve to show how incomplete my original design was and how many requirements I forgot to take into account (of course, that's just me and you design complete systems with well defined specifications and your users never change their mind about what the system should do - grin).

Please note that I'm not advocating here for "test driven development". I find the syntagm a bit silly and sounds like a construction engineer advocating for "scaffold driven construction". Scaffolds are important tools that have to be used when needed but they don't drive a construction any more than a crane or an excavator do.

Why a New Framework

There are many test frameworks you can choose from but I found myself particularly attracted to UnitTest++, a framework created by Noel LLopis and Charles Nicholson. Before making UnitTest++ framework, in [another article] (http://gamesfromwithin.com/exploring-the-c-unit-testing-framework-jungle), Noel spelled out some of his requirements for a test framework:

• Minimal amount of work needed to add new tests
• Easy to modify and port
• Supports setup/tear-down steps (fixtures)
• Handles exceptions and crashes well
• Good assert functionality
• Supports different outputs
• Supports suites

UnitTest++ was based on these requirements and fulfills most of them. However, I found a problem: the implementation is not very tight with WAY too many objects and unfinished methods for my taste. Instead of choosing another framework, I decided to re-implement UnitTest++ and that's how UTPP (Unit Test Plus Plus) came into existence. It borrows the API from UnitTest++ but the implementation is all new.

The latest version of this library is header-only. That means there is no libary to build or link. You just include the header file.

Using the Framework

Here is a short example of how to use the test framework:

#include <utpp/utpp.h>

bool earth_is_round ();
double earth_radius_km ();

TEST (EarthShape)
{
  CHECK (earth_is_round ());
}

TEST (HowBigIsEarth)
{
  CHECK_CLOSE (6371., earth_radius_km(), 1.);
}

TEST_MAIN (int argc, char** argv)
{
  return UnitTest::RunAllTests ();
}

The program contains two tests: one that checks if the earth_is_round function returns true and another one that checks if the earth_radius_km function is close enough to the expected value. The main program runs all the tests and, if all goes well, returns 0.

Tests are introduced by the TEST macro followed by a block of code. Throughout the test, you can check different conditions using one of the CHECK_... macros. The example above showed two of these macros: CHECK verifies that a condition is true, while CHECK_CLOSE verifies that two values are closer than a specified limit.

There are many macros to verify different condition during a test. Below is a list of those macros and the conditions they verify:

• CHECK(condition) - condition is true
• CHECK_EX (condition, message) - condition is true. If not, it produces the specified message.
• CHECK_EQUAL (expected, actual) - actual value is equal to expected value. The two values can be of any type that has an equality operator.
• CHECK_CLOSE (expected, actual, tolerance) - actual value is within tolerance from expected value. Specially useful for floating point values.
• CHECK_ARRAY_EQUAL (expected, actual, count) - actual array is equal to expected array. Each array has count elements. This macro is for C-style array. C++ containers that know their size can use CHECK_EQUAL macro.
• CHECK_ARRAY_CLOSE (expected, actual, count, tolerance) - actual array is with tolerance from expected array
• CHECK_ARRAY2D_EQUAL (expected, actual, rows, columns) - actual two-dimensional array is equal to expected array
• CHECK_ARRAY2D_CLOSE (expected, actual, rows, columns, tolerance) - actual two-dimensional array is within tolerance from expected array

There is nothing special to be done when adding new tests: you just write them and they will get executed. Tests can be in the same source file or in different ones. They will still be picked up automatically and executed. There is however no guarantee about the execution order.

Here is another example using CHECK_EQUAL macro:

const char *planet_name () {
  return "Earth";
}

TEST (PlanetName)
{
  CHECK_EQUAL ("Earth", planet_name ());
} 

This macro can compare numbers, strings or in general any values for which an equality operator is defined.

You can also test if an exception is thrown using CHECK_THROW macro:

class flat_earth_exception : public std::exception {
public:
  const char *what () { return "Earth is not flat!"; }
};

void go_to_end_of_earth ()
{
  throw flat_earth_exception();
}
TEST (EndOfTheEarth)
{
  CHECK_THROW (flat_earth_exception, go_to_end_of_earth ());
}

Exceptions thrown outside of a CHECK_THROW macro are considered failures and are caught by try... catch blocks that wrap each test.

Fixtures

When performing a test, you need certain objects and values to be in a known state before the beginning of the test. This is called a fixture. In UTPP, any object with a default constructor can be used as a fixture. Your tests will be derived from that object and the state of the object is defined by the fixture constructor.

Example:

void exchange_to_eur (double& usd, double& eur);

struct Account_fixture {
  Account_fixture () : amount_usd (100), amount_eur (0), amount_chf (0) {}
  ~Account_fixture () {}
  double amount_usd;
  double amount_eur;
  double amount_chf;
};
TEST_FIXTURE (Account_fixture, TestExchangeEur)
{
  exchange_to_eur (amount_usd, amount_eur);
  CHECK_EQUAL (0, amount_usd);
  CHECK (amount_eur > 0);
}

A test that uses a fixture is defined using a TEST_FIXTURE macro that takes as arguments the name of the fixture and the name of the test. The fixture constructor is invoked right before the beginning of the test and it insures that amount_usd is set to 100. Because the test object is derived from the fixture object, any public or protected members of the fixture are directly available in the test body. When the test finishes, the fixture destructor gets called to release any resources allocated by the constructor.

More than one test can use the same fixture and it will be setup the same way at the beginning of each test:

void exchange_to_chf (double& usd, double& chf);
  
TEST_FIXTURE (Account_fixture, TestExchangeChf)
{
  exchange_to_chf (amount_usd, amount_chf);
  CHECK_EQUAL (0, amount_usd);
  CHECK (amount_chf > 0);
}

In this case, both tests, TestExchangeEur and TestExchangeChf start with the same configuration.

Result Handling - Reporters

All output from the different CHECK_... macros together with other general messages are sent an object called a reporter. This object is responsible for generating the actual output. The default reporter sends all results to stdout. There is another reporter for generating an XML file in a format similar with NUnit. Here is a fragment from a tests results XML file:

<?xml version="1.0" encoding="UTF-8"?>
<utpp-results total="167" failed="0" failures="0" duration="17.834">
 <start-time>2021-12-30 21:18:19Z</start-time>
 <command-line>&quot;C:\development\mlib\tests\exe\x86\Release\mlib_test.exe&quot;</command-line>
 <suite name="DefaultSuite">
  <test name="Base64_Encode" time_ms="0"/>
  <test name="Base64_Encode_Zero_Length" time_ms="0"/>
  <test name="Base64_Decode" time_ms="0"/>
  <test name="dirname" time_ms="0"/>
...

Yet another reporter can send all output using the OutputDebugString function.

To change the reporter used, you create a reporter object and pass it to RunAllTests function:

std::ofstream os ("test.xml");
UnitTest::ReporterXml xml (os);
UnitTest::RunAllTests (xml);

Test Grouping

Tests can be grouped in suites:

SUITE (many_tests)
{
  TEST (test1) { }
  TEST (test2) { }
}

All tests from one suite are going to be executed before the next suite begins. If the main program invokes the tests by calling UnitTest::RunAllTests() function, there are no guarantees as to the order of execution of each suite or for the order of tests within the suite. There is however a function:

int UnitTest:RunSuite (const std::string& suite_name);

that runs only one suite.There is also a function that prevents a suite from running:

void UnitTest::DisableSuite (const std::string& suite_name);

Internally, a suite is implemented as a namespace and that helps preventing clashes between test names: you have to keep test names unique only within a suite.

Timing

Each test can have a limit set for its running time. You define these local time limits using the UNITTEST_TIME_CONSTRAINT(ms). This macro creates an object of type UnitTest::TimeConstraint in the scope where it was invoked. When this object gets out of scope, if the preset time limit has been exceeded, it generates a message that is logged by the reporter.

In addition to these local time limits, the UnitTEst::RunAllTests takes an additional parameter that is the default time limit for every test. Again, if a test fails this global time limit, the reporter generates a message. If using the global time limit, a test can be exempted from this check by invoking the UNITTEST_TIME_CONSTRAINT_EXEMPT macro.

Architecture

In its simplest form, a test is defined using the TEST macro using the following syntax:

TEST (MyFirstTest)
{
  // test code goes here
}

A number of things happen behind the scenes when TEST macro is invoked:

1. It defines a class called TestMyFirstTest derived from Test class. The new class has a method called RunImpl and the block of code following the TEST macro becomes the body of the RunImpl method.

2. It creates a small factory function (called MyFirstTest_maker) with the following body:

   C++

   Test* MyFirstTest_maker ()
   {
    return new MyFirstTest;
   }

   We are going to call this function the maker function.

3. A pointer to the maker together with the name of the current test suite and some additional information is used to create a TestSuite::Inserter object (with the name MyFirstTest_inserter). The current test suite has to be established using a macro like in the following example:

   C++

   SUITE (LotsOfTests)
   {
     // tests definitions go here
   }

   If no suite has been declared, tests are by default appended to the default suite.

4. The TestSuite::Inserter constructor appends the newly created object to current test suite.

5. There is a global SuitesList object that is returned by GetSuitesList() function. This object maintains a container with all currently defined suites.

The main program contains a call to RunAllTests() that triggers the following sequence of events:

1. One of the parameters to the RunAllTests() function is a TestReporter object, either one explicitly created or the default reporter that sends all results to stdout.

2. The RunAllTests() function calls SuitesList::RunAll() function.

3. SuitesList::RunAll() iterates through the list test suites mentioned before and, for each suite calls the TestSuite::RunTests() function.

4. TestSuite::RunTests() iterates through the list of tests and for each test does the following:

   • Calls maker function to instantiate a new Test-derived object (like TestMyFirstTest).
   • Calls the Test::Run method which in turn calls the TestMyFirstTest::RunImpl. This is actually the test code that was placed after the TEST macro.
   • When the test has finished, the Test-derived object is deleted.

Throughout this process, different methods of the reporter are called at appropriate moments (beginning of test suite, beginning of test, end of test, end of suite, end of run).

CHECK... macros evaluate the condition and, if false, call the ReportFailure function, which in turn calls Reporter::ReportFailure function to record all failure information (file name, line number, message, etc.). To determine if a condition is true, the CHECK_EQUAL and CHECK_THROW_EQUAL macros invoke a template function:

template <typename Expected, typename Actual>
bool CheckEqual (const Expected& expected, const Actual& actual, std::string& msg)
{
  if (!(expected == actual))
  {
    std::stringstream stream;
    stream << "Expected " << expected << " but was " << actual;
    msg = stream.str ();
    return false;
  }
  return true;
}

The template function can be instantiated for any objects that support the equality operator.

The latest version of UTPP is a header-only library but it still needs a couple of global variables. C++ versions prior to C++17 are not very friendly to global data in header files. The solution was to replace the main function with a macro TEST_MAIN(ARGC, ARGV) with the same signature as the main function. The macro take care of defining the global variables before defining the main function. If you are using C++17 or newer, you don't have to use the TEST_MAIN macro.

Conclusion

To finalize, let's review the requirements listed at the beginning of this article and see how UTPP fares against them:

1. Minimal amount of work needed to add tests. You just write the test using TEST or TEST_FIXTURE macros. Test registration is automatic and tests can be in different source files.

2. Easy to modify and port. The code is very clean and well documented.

3. Supports setup/tear-down steps (fixtures). Any object with a default constructor can become a fixture. Fixtures are integrated into a test using TEST_FIXTURE macro. Object destructor takes care of tear-down.

4. Handles exceptions and crashes well. Tests can check for exceptions using CHECK_THROW and CHECK_THROW_EX macros. All other exceptions are caught and logged.

5. Good assert functionality. There are a variety of CHECK_... macros. As they translate internally into function templates, they can take arbitrary parameters.

6. Supports different outputs. The use of reporter objects makes it easy to redirect output to different venues. As is the library can direct output to stdout, debug output or an XML file but users can create their own reporters.

7. Supports suites. Yes, it does.

Although there is no shortage of unit test frameworks, if you spend a bit of time with UTPP, you might begin to like it. You can find the latest version on GitHub at: https://github.com/neacsum/utpp and any contributors are welcome.

History

• 21^st April 2020: Initial version
• 1^st February 2022: New, header-only library