Practical Unit Testing - a manual

Introduction

Recently, with the advent of agile programming methods and especially “Extreme Programming”, a technique somewhat older has been projected into the public view. This technique, Unit Testing is the general subject of this article. A particular aspect of it, Automated Unit Testing, has gained popularity with the release of JUnit, an automated testing tool for Java. Since then, a plethora of books, articles and papers have been written that have Unit Testing as their subject. Unfortunately, most of the literature on the topic has been focused on the whys of Unit Testing and on the tools of automated unit testing demonstrated for their own sake. These writings generally present Automated Unit Testing merely as a supporting technique for one or another agile programming method. Furthermore, beyond the documentation (help files, “cookbooks”) provided with these tools, code examples are rarely given and real examples of the application of this powerful technique are few and far between.

This article is the first in a series of articles that aim at filling the gap that exists, providing a complete introduction to Unit Testing together with code examples that will try to keep as close as possible to real life scenarios. Obviously, Unit Testing can be done on many platforms and in many languages but this article will restrict the scope of investigations to the .NET platform and will be using C# as its implementation language.

The meaning of Unit Testing

As can be immediately understood by its name, Unit Testing is a form of testing. More precisely, it is an attempt to prove the correctness of one or more pieces of software by putting them to work and validating their functioning/results against a known set of values/behavioural elements (the contract). This process can take many forms but the object of this article will be writing code to test code. Basically, given a piece of code (implementation code) of which the correctness needs to be established, some more code (test code) is written to exercise the implementation code and verify that operations performed by it correspond to the model that is being implemented. One key feature of Unit Testing, in fact the feature that gives it its name, is that it is all about testing discrete functional units of the software and not the software system as a whole. This differentiates it from the aptly named System Testing. A unit test ensures the validity of a well delimited piece of the system (a unit). In practice, this will mean that what we test are components, classes, packages, compilation units… depending on the terminology that applies in the environment for which we develop. Since our main focus of interest here is .NET which is an object-orientated programming framework, we will be applying Unit testing to test classes and components.

What do you test for?

As briefly stated above, what we test for is the adherence of our code to a set of pre-determined decisions about its workings. That set of decisions is generally referred to as the contract. The contract reflects what we wanted to do when we set out to write that piece of code. It also states what our users (business customers or other developers using our libraries) expect from our code. As might be expected, the contract encompasses many things, all of them important to the users. But generally, three areas can be isolated as being the target of our testing efforts.

• Accuracy

The general compliance of the results given by our code with what is exhibited by the model we are trying to implement. Basically, we are making sure here that given correct inputs/working conditions, our code will produce the right results/perform the proper operations. I.e. if our code implements the addition process, it must give the correct sum of the input elements given to it. Compliance with this part of the contract determines the correctness of our code.

• Stress

This area of the contract states the behaviour expected of our code when the size of inputs given to it is large to the point of being near overwhelming or when the resources available to our code are restricted in some way (not enough memory, no ports available, …) It also describes the expected behaviour of the code in a multi-threaded environment. If our code performs image processing operations, it must behave properly (not crash or hang the machine) when given an image that is larger than the memory available. Compliance with this part of the contract determines the robustness of our code.

• Failure

A part of the contract that spells out the behaviour expected of our code when it is presented with invalid or wrong input. To reuse the image processing code example, we assume that our code is asked to open a file that does not contain an image but is in fact a renamed music file; in that case, the code must not crash or hang or worse “open” that file and display a nonsensical picture to the user.

Compliance with this part of the contract determines the reliability of our code.

Benefits of Unit Testing

Like all other kinds of testing, Unit Testing finds errors, and just like other kinds of testing it is limited in its scope because you can not be sure to have found every single error. But Unit Testing helps in finding a number of them. A particularly interesting application of Unit Testing is in regression testing; that helps ensure that previously found (and eliminated) errors do not creep back into the software. Because of its discrete nature, Unit Testing is easy to perform. Each piece of software being tested is small (relative to the whole) and therefore well-determined. Because of that, Unit Testing gives an incredible amount of confidence in one’s code and enables developers to work and apply changes to complex areas of software knowing that unit tests are there to maintain the correctness of their work.

Basically two “schools of thought” exist and each has its prescribed way of doing Unit Testing. The one that is most vocal advocates for Test-Driven Development or “test first programming”. What this means is that given the contract that the implementation code must fulfil, the programmer starts by writing code to test the implementation (which hasn’t been written yet). Following that, the rest of the development consists in making sure that the tests formerly written all pass. This way of doing things is mostly associated with the Extreme Programming camp. Other developers prefer to write the code first and then write unit tests to ensure the validity of their code. Obviously, each “side” has many arguments to justify its position and the intensity of the debate around this is nearly as great as the one surrounding coding style or language preference. In the next installment in this series we will see a bit of both ways.

A glance at Unit Testing

Keep a running update of any changes or improvements you've made here.

Let us take a quick look at what Unit Testing is really about. We will be using, for the sake of this example, a very simplistic situation. The intent here is to illustrate the process of writing unit tests. Let us assume that we have to write some code to divide integer numbers. We intend to do so by creating a class ArithmeticOperations that will expose a method Division that takes two integer parameters and returns an integer, the quotient of the division of the first parameter by the second. We want to be sure that this code performs properly by writing some unit tests for it. First, we start by listing known facts about division. This step is important because these facts are the yardstick against which we will measure the correctness of our division code.

1. Any number divided by 1 yields itself
2. Any number divided by itself yields 1
3. 19 divided by 7 yields 2
4. 15 divided by 5 yields 3
5. 3 divided by 8 yields 0
6. No number can be divided by zero

What you can notice here is that “Fact” 1 and “Fact” 2 are general facts about the result of a division. “Fact” 3, “Fact” 4 and “Fact” 5 each show a special case for a division. They have been chosen to illustrate the three main situations of a division (and this is important, unit test code should be written to exercise all types of situations your code will be confronted with); i.e. an even division, a division with a remainder and a division that gives 0. “Fact” 6 shows us a case where division cannot be performed (remember the failure tests?). It is good to list among the “Facts”, special cases for which we know the result or behaviour to expect and general rules about the result of our operations. So armed with these facts, we now write another class to test ArithmeticOperations (actually we could put our tests in the same class but it is cleaner to separate them from the implementation code).

We make sure that we provide inputs that match the conditions of each “Fact”, and we verify that the result provided or behaviour exhibited by our class corresponds to what is specified in the “Fact”.

Our implementation code would look like this

/// <summary>
/// Class that implements arithmetic operations for integer numbers.
/// This class simply calls the standard CLR operators.
/// </summary>
public class ArithmeticOperations
{
/// <summary>
/// Empty constructor
/// </summary>
public ArithmeticOperations()
{
}
/// <summary>
/// Performs an integer division
/// </summary>
/// <param name="a">Dividend. Can be any integer</param>
/// <param name="b">Divisor. Can be any non-zero integer</param>
/// <returns>Integer quotient of the division of a by b</returns>
/// <exception cref="ArgumentException">If b is zero</exception>
public int Division(int a, int b)
{
if (b != 0)
{
return a / b;
}
else
{
throw new ArgumentException("Cannot divide by zero", "b");
}
}
}

Notice that our code throws an exception when zero is passed as a divisor. It is very important that all code (not only contrived examples) actively reject parameters for which the operation they are supposed to perform is undefined or impossible to perform.

Our test code would look like this

/// <summary>
/// Example test class for ArithmeticOperations.
/// This class has a method for every "Fact" identified
/// during the test preparation phase.
/// </summary>
public class TestArithmeticOperations
{
/// <summary>
/// variable used to determine the number of
/// repetitions used for certain tests
/// </summary>
private const int numberOfTests = 100;
/// <summary>
/// We need to make sure that our tests are ALWAYS the same.
/// So we make sure to use a well-defined constant seed
/// for random number generation
/// </summary>
private const int randomNumbersSeed = 13547;
/// <summary>
/// Empty constructor
/// </summary>
public TestArithmeticOperations()
{
}
/// <summary>
/// Tests Fact1.
/// Any number divided by itself gives 1.
/// We perform this test on a certain number of
/// randomly generated numbers. Because we need
/// our tests to be always the same, we generate
/// those random numbers from a constant seed
/// defined in this class
/// </summary>
public void TestFact1()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next();
int r = ao.Division(a, a);
if (r != 1)
{
Console.WriteLine("Error dividing {0} by {0}", a);
}
}
}
/// <summary>
/// Tests Fact2.
/// Any number divided by 1 gives itself.
/// We perform this test on a certain number of
/// randomly generated numbers. Because we need
/// our tests to be always the same, we generate
/// those random numbers from a constant seed
/// defined in this class
/// </summary>
public void TestFact2()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
//we want a number in the range [1, numberOfTests]
int a = rnd.Next(1, numberOfTests);
int r = ao.Division(a, 1);
if (r != a)
{
Console.WriteLine("Error dividing {0} by 1", a);
}
}
}
/// <summary>
/// Tests Fact3.
/// 19 divided by 7 gives 2.
/// This tests a non-even division.
/// </summary>
public void TestFact3()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(19, 7);
if (r != 2)
{
Console.WriteLine("Error dividing {0} by {1}", 19, 7);
}
}
/// <summary>
/// Tests Fact4.
/// 15 divided by 5 gives 3.
/// This tests a standard even division.
/// </summary>
public void TestFact4()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(15, 5);
if (r != 3)
{
Console.WriteLine("Error dividing {0} by {1}", 15, 5);
}
}
/// <summary>
/// Tests Fact5.
/// 3 divided by 7 gives 0.
/// This tests a division when the dividend
/// is lower than the divisor
/// </summary>
public void TestFact5()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(3, 7);
if (r != 0)
{
Console.WriteLine("Error dividing {0} by {1}", 3, 7);
}
}
/// <summary>
/// Tests Fact6.
/// No number can be divided by 0.
/// We therefore attempt to do so while
/// expecting the exception specified by
/// our code's contract to be thrown.
/// If the exception is not thrown, then the
/// test has failed because our code is returning
/// a results where none exists.
/// We perform this test on a certain number of
/// randomly generated numbers. Because we need
/// our tests to be always the same, we generate
/// those random numbers from a constant seed
/// defined in this class
/// </summary>
public void TestFact6()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next();
try
{
ao.Division(a, 0);
Console.WriteLine("Our code pretends to divide by 0");
}
catch(ArgumentException)
{
}
}
}
}

“Fact” 1 through “Fact” 5 are accuracy tests while “Fact” 6 is a failure test. This article does not illustrate any stress test. This is partly because the example is trivial, and also because stress testing requires a discussion of its own. This is therefore left for the next installment to address.

All there is left to do is create a console project that will reference the subject class and the test class, and call each method of the test class.

We now have a full battery of tests that we can run on our original class (i.e. ArithmeticOperations). As long as our code complies with the “Facts” we listed during the test preparation phase, the tests will run smoothly (no error message). It should be clear by now that the more “Facts” we know about the code we are writing, the more comprehensive our testing will be. In the particular case we are studying some more facts could have been listed (e.g. when multiplying the quotient by the divisor, one obtains a number which when subtracted from the dividend gives a number always lower than the divisor. Goodness that sounds complicated!!) but the case is so trivial that more tests would have made our study disproportionately complicated (if that’s not already the case J). A trap to avoid is the listing of “Facts” that include one another. For example, listing a “Fact” that states a rule about division by 2 and another “Fact” that states a rule about division by even numbers.

Automated Unit Testing and its tools

The way of doing things displayed in the preceding paragraph, though functional, tends to be cumbersome and a bit inefficient. Specifically:

• We need to write some more code to call every method of our test code.
• In our test code, we have to call methods to alert the user of problems.
• We are limited to running our tests in a console environment
• It would take some more code to get statistics about our tests.

This is where Automated Unit Testing comes into play. What we need is a tool that would provide us with a framework to write our tests and would enable us to run those tests. That tool would reduce the amount of infrastructure code we need to write (all those

Console.WriteLine(...)

NUnit can be gotten at http://www.nunit.orgg . The latest version is v2.2 which looks a bit different than the version used in this article v2.1 but nevertheless implements the same concepts.

Automated Unit testing tools use a specific terminology that makes talking about tests easier.

• Every method that tests one of the “Facts” mentioned earlier is called a test case.A class (like TestArithmeticOperations) which combines a group of test cases is called a test fixture
• When testing a component (basically many classes that work together to provide a user with a specific interface to accomplish a well-defined task), one writes many test fixtures. The combination of those test fixtures is called a test suite.

if (r != 4){Console.WriteLine("Error");}

using System;
//needed to access NUnit attributes and classes
using NUnit.Framework;
namespace SimplisticExamples
{
/// <summary>
/// Example test class for ArithmeticOperations.
/// This class has a method for every "Fact" identified
/// during the test preparation phase.
/// </summary>
[TestFixture]
public class NUnitTestArithmeticOperations
{
/// <summary>
/// variable used to determine the number of
/// repetitions used for certain tests
/// </summary>
private const int numberOfTests = 100;
/// <summary>
/// We need to make sure that our tests are ALWAYS the same.
/// So we make sure to use a well-defined constant seed
/// for random number generation
/// </summary>
private const int randomNumbersSeed = 13547;
/// <summary>
/// Empty constructor
/// </summary>
public NUnitTestArithmeticOperations()
{
}
/// <summary>
/// Tests Fact1.
/// Any number divided by itself gives 1.
/// We perform this test on a certain number of
/// randomly generated numbers. Because we need
/// our tests to be always the same, we generate
/// those random numbers from a constant seed
/// defined in this class
/// </summary>
[Test]
public void TestFact1()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next(1, numberOfTests);
int r = ao.Division(a, a);
Assertion.AssertEquals("Dividing " + a + " by " + 1, 1, r);
}
}
/// <summary>
/// Tests Fact2.
/// Any number divided by 1 gives itself.
/// We perform this test on a certain number of
/// randomly generated numbers. Because we need
/// our tests to be always the same, we generate
/// those random numbers from a constant seed
/// defined in this class
/// </summary>
[Test]
public void TestFact2()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next(1, numberOfTests);
int r = ao.Division(a, 1);
Assertion.AssertEquals("Dividing " + a + " by " + a, a, r);
}
}
/// <summary>
/// Tests Fact4.
/// 19 divided by 7 gives 2.
/// This tests a non-even division.
/// </summary>
[Test]        public void TestFact3()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(19, 7);
Assertion.AssertEquals("Error dividing " + 19 + " by " + 7, 2, r);
}
/// <summary>
/// Tests Fact4.
/// 15 divided by 5 gives 3.
/// This tests a standard even division.
/// </summary>
[Test]
public void TestFact4()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(15, 5);
Assertion.AssertEquals("Error dividing " + 15 + " by " + 5, 3, r);
}
/// <summary>
/// Tests Fact5.
/// 3 divided by 7 gives 0.
/// This tests a division when the dividend
/// is lower than the divisor
/// </summary>
[Test]        public void TestFact5()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(3, 7);
Assertion.AssertEquals("Error dividing " + 3 + " by " + 7, 0, r);          }
/// <summary>
/// Tests Fact6.
/// No number can be divided by 0.
/// We therefore attempt to do so while
/// expecting the exception specified by
/// our code's contract to be thrown.
/// If the exception is not thrown, then the
/// test has failed because our code is returning
/// a results where none exists.
/// We perform this test on a certain number of
/// randomly generated numbers. Because we need
/// our tests to be always the same, we generate
/// those random numbers from a constant seed
/// defined in this class
/// </summary>
[Test]
public void TestFact6()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next(1, numberOfTests);
try
{
ao.Division(a, 0);
Assertion.Fail("Our code pretends to divide by zero");
}
catch(ArgumentException)
{
}
}
}
}
}

/// <summary>t;
/// Performs an integer division
/// Utilises a different algorithm based
/// on subtraction of numbers.
/// </summary>
/// <param name="a">Dividend. Can be any integer</param>
/// <param name="b">Divisor. Any non-zero integer</param>
/// <returns>Integer quotient of the division of a by b</returns>
public int Division(int a, int b)
{
int r = 1;
//We keep subtracting the divisor from the dividend
//while incrementing the quotient until the dividend
//becomes lower than 1
while((a - b != a) && (a -= b) > 0)
r++;
return r;
}