Click here to Skip to main content
Click here to Skip to main content

Practical Unit Testing - a manual

, 11 Oct 2004
Rate this:
Please Sign up or sign in to vote.
An article on the details and HowTos of Unit Testing on the .NET platform

Introduction

Recently, with the advent of agile programming methods and especially “Extreme Programming”, a technique somewhat older has been projected into the public view. This technique, Unit Testing is the general subject of this article. A particular aspect of it, Automated Unit Testing, has gained popularity with the release of JUnit, an automated testing tool for Java. Since then, a plethora of books, articles and papers have been written that have Unit Testing as their subject. Unfortunately, most of the literature on the topic has been focused on the whys of Unit Testing and on the tools of automated unit testing demonstrated for their own sake. These writings generally present Automated Unit Testing merely as a supporting technique for one or another agile programming method. Furthermore, beyond the documentation (help files, “cookbooks”) provided with these tools, code examples are rarely given and real examples of the application of this powerful technique are few and far between.

This article is the first in a series of articles that aim at filling the gap that exists, providing a complete introduction to Unit Testing together with code examples that will try to keep as close as possible to real life scenarios. Obviously, Unit Testing can be done on many platforms and in many languages but this article will restrict the scope of investigations to the .NET platform and will be using C# as its implementation language.

The meaning of Unit Testing

As can be immediately understood by its name, Unit Testing is a form of testing. More precisely, it is an attempt to prove the correctness of one or more pieces of software by putting them to work and validating their functioning/results against a known set of values/behavioural elements (the contract). This process can take many forms but the object of this article will be writing code to test code. Basically, given a piece of code (implementation code) of which the correctness needs to be established, some more code (test code) is written to exercise the implementation code and verify that operations performed by it correspond to the model that is being implemented. One key feature of Unit Testing, in fact the feature that gives it its name, is that it is all about testing discrete functional units of the software and not the software system as a whole. This differentiates it from the aptly named System Testing. A unit test ensures the validity of a well delimited piece of the system (a unit). In practice, this will mean that what we test are components, classes, packages, compilation units… depending on the terminology that applies in the environment for which we develop. Since our main focus of interest here is .NET which is an object-orientated programming framework, we will be applying Unit testing to test classes and components.

What do you test for?

As briefly stated above, what we test for is the adherence of our code to a set of pre-determined decisions about its workings. That set of decisions is generally referred to as the contract. The contract reflects what we wanted to do when we set out to write that piece of code. It also states what our users (business customers or other developers using our libraries) expect from our code. As might be expected, the contract encompasses many things, all of them important to the users. But generally, three areas can be isolated as being the target of our testing efforts.

  • Accuracy

The general compliance of the results given by our code with what is exhibited by the model we are trying to implement. Basically, we are making sure here that given correct inputs/working conditions, our code will produce the right results/perform the proper operations. I.e. if our code implements the addition process, it must give the correct sum of the input elements given to it. Compliance with this part of the contract determines the correctness of our code.

  • Stress

This area of the contract states the behaviour expected of our code when the size of inputs given to it is large to the point of being near overwhelming or when the resources available to our code are restricted in some way (not enough memory, no ports available, …) It also describes the expected behaviour of the code in a multi-threaded environment. If our code performs image processing operations, it must behave properly (not crash or hang the machine) when given an image that is larger than the memory available. Compliance with this part of the contract determines the robustness of our code.

  • Failure

A part of the contract that spells out the behaviour expected of our code when it is presented with invalid or wrong input. To reuse the image processing code example, we assume that our code is asked to open a file that does not contain an image but is in fact a renamed music file; in that case, the code must not crash or hang or worse “open” that file and display a nonsensical picture to the user.

Compliance with this part of the contract determines the reliability of our code.

Benefits of Unit Testing

Like all other kinds of testing, Unit Testing finds errors, and just like other kinds of testing it is limited in its scope because you can not be sure to have found every single error. But Unit Testing helps in finding a number of them. A particularly interesting application of Unit Testing is in regression testing; that helps ensure that previously found (and eliminated) errors do not creep back into the software. Because of its discrete nature, Unit Testing is easy to perform. Each piece of software being tested is small (relative to the whole) and therefore well-determined. Because of that, Unit Testing gives an incredible amount of confidence in one’s code and enables developers to work and apply changes to complex areas of software knowing that unit tests are there to maintain the correctness of their work.

Basically two “schools of thought” exist and each has its prescribed way of doing Unit Testing. The one that is most vocal advocates for Test-Driven Development or “test first programming”. What this means is that given the contract that the implementation code must fulfil, the programmer starts by writing code to test the implementation (which hasn’t been written yet). Following that, the rest of the development consists in making sure that the tests formerly written all pass. This way of doing things is mostly associated with the Extreme Programming camp. Other developers prefer to write the code first and then write unit tests to ensure the validity of their code. Obviously, each “side” has many arguments to justify its position and the intensity of the debate around this is nearly as great as the one surrounding coding style or language preference. In the next installment in this series we will see a bit of both ways.

A glance at Unit Testing

Keep a running update of any changes or improvements you've made here.

Let us take a quick look at what Unit Testing is really about. We will be using, for the sake of this example, a very simplistic situation. The intent here is to illustrate the process of writing unit tests. Let us assume that we have to write some code to divide integer numbers. We intend to do so by creating a class ArithmeticOperations that will expose a method Division that takes two integer parameters and returns an integer, the quotient of the division of the first parameter by the second. We want to be sure that this code performs properly by writing some unit tests for it. First, we start by listing known facts about division. This step is important because these facts are the yardstick against which we will measure the correctness of our division code.

  1. Any number divided by 1 yields itself
  2. Any number divided by itself yields 1
  3. 19 divided by 7 yields 2
  4. 15 divided by 5 yields 3
  5. 3 divided by 8 yields 0
  6. No number can be divided by zero

What you can notice here is that “Fact” 1 and “Fact” 2 are general facts about the result of a division. “Fact” 3, “Fact” 4 and “Fact” 5 each show a special case for a division. They have been chosen to illustrate the three main situations of a division (and this is important, unit test code should be written to exercise all types of situations your code will be confronted with); i.e. an even division, a division with a remainder and a division that gives 0. “Fact” 6 shows us a case where division cannot be performed (remember the failure tests?). It is good to list among the “Facts”, special cases for which we know the result or behaviour to expect and general rules about the result of our operations. So armed with these facts, we now write another class to test ArithmeticOperations (actually we could put our tests in the same class but it is cleaner to separate them from the implementation code).

We make sure that we provide inputs that match the conditions of each “Fact”, and we verify that the result provided or behaviour exhibited by our class corresponds to what is specified in the “Fact”.

Our implementation code would look like this

/// <summary>
/// Class that implements arithmetic operations for integer numbers.
/// This class simply calls the standard CLR operators.
/// </summary>
public class ArithmeticOperations
{
/// <summary>
/// Empty constructor
/// </summary>
public ArithmeticOperations()
{
}
/// <summary>
/// Performs an integer division
/// </summary>
/// <param name="a">Dividend. Can be any integer</param>
/// <param name="b">Divisor. Can be any non-zero integer</param>
/// <returns>Integer quotient of the division of a by b</returns>
/// <exception cref="ArgumentException">If b is zero</exception>
public int Division(int a, int b)
{
if (b != 0)
{
return a / b;
}
else
{
throw new ArgumentException("Cannot divide by zero", "b");
}
}
}

Notice that our code throws an exception when zero is passed as a divisor. It is very important that all code (not only contrived examples) actively reject parameters for which the operation they are supposed to perform is undefined or impossible to perform.

Our test code would look like this

/// <summary>
/// Example test class for ArithmeticOperations.
/// This class has a method for every "Fact" identified
/// during the test preparation phase.
/// </summary>
public class TestArithmeticOperations
{
/// <summary>
/// variable used to determine the number of
/// repetitions used for certain tests
/// </summary>
private const int numberOfTests = 100;
/// <summary>
/// We need to make sure that our tests are ALWAYS the same.
/// So we make sure to use a well-defined constant seed
/// for random number generation
/// </summary>
private const int randomNumbersSeed = 13547;
/// <summary>
/// Empty constructor
/// </summary>
public TestArithmeticOperations()
{
}
/// <summary>
/// Tests Fact1.
/// Any number divided by itself gives 1.
/// We perform this test on a certain number of
/// randomly generated numbers. Because we need
/// our tests to be always the same, we generate
/// those random numbers from a constant seed
/// defined in this class
/// </summary>
public void TestFact1()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next();
int r = ao.Division(a, a);
if (r != 1)
{
Console.WriteLine("Error dividing {0} by {0}", a);
}
}
}
/// <summary>
/// Tests Fact2.
/// Any number divided by 1 gives itself.
/// We perform this test on a certain number of
/// randomly generated numbers. Because we need
/// our tests to be always the same, we generate
/// those random numbers from a constant seed
/// defined in this class
/// </summary>
public void TestFact2()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
//we want a number in the range [1, numberOfTests]
int a = rnd.Next(1, numberOfTests);
int r = ao.Division(a, 1);
if (r != a)
{
Console.WriteLine("Error dividing {0} by 1", a);
}
}
}
/// <summary>
/// Tests Fact3.
/// 19 divided by 7 gives 2.
/// This tests a non-even division.
/// </summary>
public void TestFact3()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(19, 7);
if (r != 2)
{
Console.WriteLine("Error dividing {0} by {1}", 19, 7);
}
}
/// <summary>
/// Tests Fact4.
/// 15 divided by 5 gives 3.
/// This tests a standard even division.
/// </summary>
public void TestFact4()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(15, 5);
if (r != 3)
{
Console.WriteLine("Error dividing {0} by {1}", 15, 5);
}
}
/// <summary>
/// Tests Fact5.
/// 3 divided by 7 gives 0.
/// This tests a division when the dividend
/// is lower than the divisor
/// </summary>
public void TestFact5()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(3, 7);
if (r != 0)
{
Console.WriteLine("Error dividing {0} by {1}", 3, 7);
}
}
/// <summary>
/// Tests Fact6.
/// No number can be divided by 0.
/// We therefore attempt to do so while
/// expecting the exception specified by
/// our code's contract to be thrown.
/// If the exception is not thrown, then the
/// test has failed because our code is returning
/// a results where none exists.
/// We perform this test on a certain number of
/// randomly generated numbers. Because we need
/// our tests to be always the same, we generate
/// those random numbers from a constant seed
/// defined in this class
/// </summary>
public void TestFact6()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next();
try
{
ao.Division(a, 0);
Console.WriteLine("Our code pretends to divide by 0");
}
catch(ArgumentException)
{
}
}
}
}

“Fact” 1 through “Fact” 5 are accuracy tests while “Fact” 6 is a failure test. This article does not illustrate any stress test. This is partly because the example is trivial, and also because stress testing requires a discussion of its own. This is therefore left for the next installment to address.

All there is left to do is create a console project that will reference the subject class and the test class, and call each method of the test class.

We now have a full battery of tests that we can run on our original class (i.e. ArithmeticOperations). As long as our code complies with the “Facts” we listed during the test preparation phase, the tests will run smoothly (no error message). It should be clear by now that the more “Facts” we know about the code we are writing, the more comprehensive our testing will be. In the particular case we are studying some more facts could have been listed (e.g. when multiplying the quotient by the divisor, one obtains a number which when subtracted from the dividend gives a number always lower than the divisor. Goodness that sounds complicated!!) but the case is so trivial that more tests would have made our study disproportionately complicated (if that’s not already the case J). A trap to avoid is the listing of “Facts” that include one another. For example, listing a “Fact” that states a rule about division by 2 and another “Fact” that states a rule about division by even numbers.

Automated Unit Testing and its tools

The way of doing things displayed in the preceding paragraph, though functional, tends to be cumbersome and a bit inefficient. Specifically:

  • We need to write some more code to call every method of our test code.
  • In our test code, we have to call methods to alert the user of problems.
  • We are limited to running our tests in a console environment
  • It would take some more code to get statistics about our tests.

This is where Automated Unit Testing comes into play. What we need is a tool that would provide us with a framework to write our tests and would enable us to run those tests. That tool would reduce the amount of infrastructure code we need to write (all those Console.WriteLine(...) we had in our test code and the extra project we have to create in order to run the tests). That tool would allow us to write the tests once and run them in multiple environments (console mode, graphical mode). Finally that tool would provide us with complete statistics about our tests (how many did we run, how many passed, which ones passed, where did the ones who failed fail…)

NUnit can be gotten at http://www.nunit.orgg . The latest version is v2.2 which looks a bit different than the version used in this article v2.1 but nevertheless implements the same concepts.

Automated Unit testing tools use a specific terminology that makes talking about tests easier.

  • Every method that tests one of the “Facts” mentioned earlier is called a test case.A class (like TestArithmeticOperations) which combines a group of test cases is called a test fixture
  • When testing a component (basically many classes that work together to provide a user with a specific interface to accomplish a well-defined task), one writes many test fixtures. The combination of those test fixtures is called a test suite.

So, NUnit allows the programmer to write test suites, test fixtures and test cases. The mechanism through which this is done is by putting assertionss about your operations in your test code. For example, assuming there is a variable r in your code that, after some computation, should have the value 4. Test code as we wrote it in the previous paragraph would state

if (r != 4){Console.WriteLine(“Error”);}

Assertion.AssertEquals(“Error”, 4, r);That code means, [make sure the value of r is 4; if it is not, fail the test case with the message “Error”]

There are different types of assertions:

  • Check for exp being null Assertion.AssertNull(msg, exp)
  • Check for exp not being null Assertion.AssertNotNull(msg, exp)
  • Check for exp2 being equal to exp1 Assertion.AssertEquals(msg, exp1, exp2)
  • Check for exp2 being the same as exp1 Assertion.AssertSame(msg, exp1, exp2)
  • Check for exp being a true boolean expression Assertion.Assert(msg, exp)

All assertions make the test case fail if the condition they are testing for is not verified. In that case they issue the message specified by msg.

A special kind of assertion fails: Assertion.Fail(msg)

Once the methods containing the assertions have been written, they need to be signalled as being test cases. The NUnit framework defines a way for the programmer to do just that. All that needs to be done is to add the attribute [Test]] to the method.

TestFixture] attribute to the class.

No special marker is needed to signal a group of classes as a test suite.

The code to our tests now looks like:e:

using System;
//needed to access NUnit attributes and classes
using NUnit.Framework;
namespace SimplisticExamples
{
/// <summary>
/// Example test class for ArithmeticOperations.
/// This class has a method for every "Fact" identified
/// during the test preparation phase.
/// </summary>
[TestFixture]
public class NUnitTestArithmeticOperations
{
/// <summary>
/// variable used to determine the number of
/// repetitions used for certain tests
/// </summary>
private const int numberOfTests = 100;
/// <summary>
/// We need to make sure that our tests are ALWAYS the same.
/// So we make sure to use a well-defined constant seed
/// for random number generation
/// </summary>
private const int randomNumbersSeed = 13547;
/// <summary>
/// Empty constructor
/// </summary>
public NUnitTestArithmeticOperations()
{
}
/// <summary>
/// Tests Fact1.
/// Any number divided by itself gives 1.
/// We perform this test on a certain number of
/// randomly generated numbers. Because we need
/// our tests to be always the same, we generate
/// those random numbers from a constant seed
/// defined in this class
/// </summary>
[Test]
public void TestFact1()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next(1, numberOfTests);
int r = ao.Division(a, a);
Assertion.AssertEquals("Dividing " + a + " by " + 1, 1, r);
}
}
/// <summary>
/// Tests Fact2.
/// Any number divided by 1 gives itself.
/// We perform this test on a certain number of
/// randomly generated numbers. Because we need
/// our tests to be always the same, we generate
/// those random numbers from a constant seed
/// defined in this class
/// </summary>
[Test]
public void TestFact2()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next(1, numberOfTests);
int r = ao.Division(a, 1);
Assertion.AssertEquals("Dividing " + a + " by " + a, a, r);
}
}
/// <summary>
/// Tests Fact4.
/// 19 divided by 7 gives 2.
/// This tests a non-even division.
/// </summary>
[Test]        public void TestFact3()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(19, 7);
Assertion.AssertEquals("Error dividing " + 19 + " by " + 7, 2, r);
}
/// <summary>
/// Tests Fact4.
/// 15 divided by 5 gives 3.
/// This tests a standard even division.
/// </summary>
[Test]
public void TestFact4()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(15, 5);
Assertion.AssertEquals("Error dividing " + 15 + " by " + 5, 3, r);
}
/// <summary>
/// Tests Fact5.
/// 3 divided by 7 gives 0.
/// This tests a division when the dividend
/// is lower than the divisor
/// </summary>
[Test]        public void TestFact5()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(3, 7);
Assertion.AssertEquals("Error dividing " + 3 + " by " + 7, 0, r);          }
/// <summary>
/// Tests Fact6.
/// No number can be divided by 0.
/// We therefore attempt to do so while
/// expecting the exception specified by
/// our code's contract to be thrown.
/// If the exception is not thrown, then the
/// test has failed because our code is returning
/// a results where none exists.
/// We perform this test on a certain number of
/// randomly generated numbers. Because we need
/// our tests to be always the same, we generate
/// those random numbers from a constant seed
/// defined in this class
/// </summary>
[Test]
public void TestFact6()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next(1, numberOfTests);
try
{
ao.Division(a, 0);
Assertion.Fail("Our code pretends to divide by zero");
}
catch(ArgumentException)
{
}
}
}
}
}

Once this test code has been written, what is the next step? First of all, we make sure the whole thing compiles (implementation code and test code). You may choose to have two separate projects (the best choice for large scale development) or combine test code and implementation code in the same project. If you have separate projects, be sure to add a reference to the implementation project in the test project.

NUnit offers two operating modes Console and GUI modes. Let us look at the console mode:

The console application nunit-console.exe located in the bin sub-folder of the NUnit installation folder runs the test fixtures present in the test assembly with which it is invoked.

nunit-console <test assembly>

Here is the invocation and results of the tests we just wrote on the ArithmeticOperations class.

As you can see, we had 6 tests run (the number of the “Facts” we defined), none of them failed; we also had the amount of time it took to run these tests.nunit-console offers more sophisticated options to run the tests. Help on those are listed by calling nunit-console without any option. The other mode offered by NUnit is the GUI mode.

The Windows Form application nunit-console.exe located in the bin sub-folder of the NUnit installation folder runs the test fixtures present in the test assembly with which it is invoked.

nunit-gui <test assembly>

After using that command line, the Main NUnit form is loaded

Click on the ‘Run’ button and see the progress bar unroll: Green at the beginning and shifting to Red as soon as a single test is failed. In our case, the bar remains green all through and we end up with this

ArithmeticOperations class; namely:

/// <summary>t;
/// Performs an integer division
/// Utilises a different algorithm based
/// on subtraction of numbers.
/// </summary>
/// <param name="a">Dividend. Can be any integer</param>
/// <param name="b">Divisor. Any non-zero integer</param>
/// <returns>Integer quotient of the division of a by b</returns>
public int Division(int a, int b)
{
int r = 1;
//We keep subtracting the divisor from the dividend
//while incrementing the quotient until the dividend
//becomes lower than 1
while((a - b != a) && (a -= b) > 0)
r++;
return r;
}

This way of implementing the division (which we could have chosen for whatever reason) intentionally has bugs which our tests can help us “discover”.

Running the tests from the command prompt again give us this:

Running the tests from the Graphical User Interface gives us this:

In the right window pane, we have a view of all the tests, those that passed in green, the ones that failed in red. The progress bar is entirely red to signify that our class/component is not correct. A window below the progress bar shows us the details of the test failures (same as in the console mode).

Conclusion

  • Add-ins to Visual Studio.NET (NUnitAdd-in, VSNUnit…)
  • Testing frameworks for ASP.NET (NUnitAsp)
  • Test reporting tools, test coverage tools… (NUnit2Report…)

Even though Unit testing is not the ultimate all-error remover weapon (studies have shown that it detects only 30%-40% of all bugs), it is a great tool for making sure that those bugs that have been removed remain out of the software and as such boosts developer productivity while helping to maintain software quality standards.

Like any other powerful tool, it must be applied with discipline or it will harm the development process. One particular pitfall to watch for is the accumulation over time of unit tests making the software builds prohibitively long and causing developers to waste an incredible amount of time running tests every time they want to check in new code to the repository (or worse stopping to run the tests because of the time they take). Special policies about unit tests and the build process must then be made and followed. These issues have to do with continuous integration and will be the object of a future article.

In all, when properly applied, unit testing has benefits that you as a developer cannot afford to pass.

In the next installment in this series, we will look at a general method for writing unit tests building upon the recommendations given in this article and we will see some detailed examples of real-world development with unit tests.

We will touch upon Test-Driven Development with an example of such a process and we will delve into stress testing discussing some of its complexities.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

Francois Bonin

Satellite Provider Satellite Provider
No Biography provided

Comments and Discussions

 
GeneralSome Deeper Questions PinmemberRick Pingry4-Nov-04 15:04 
GeneralRe: Some Deeper Questions PinmemberTheCois21-Dec-04 6:49 
GeneralRe: Some Deeper Questions PinmemberRick Pingry21-Dec-04 9:09 
GeneralRe: Some Deeper Questions PinmemberTheCois22-Dec-04 12:07 
GeneralRe: Some Deeper Questions PinmemberRick Pingry27-Dec-04 5:37 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.140721.1 | Last Updated 12 Oct 2004
Article Copyright 2004 by Francois Bonin
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid