Introduction
Recently, with the
advent of agile programming methods and especially “Extreme Programming”, a
technique somewhat older has been projected into the public view. This
technique, Unit Testing is the general subject of this article. A particular
aspect of it, Automated Unit Testing, has gained popularity with the release of JUnit, an automated testing tool for Java. Since then, a plethora of books,
articles and papers have been written that have Unit Testing as their subject.
Unfortunately, most of the literature on the topic has been focused on the whys
of Unit Testing and on the tools of automated unit testing demonstrated for
their own sake. These writings generally present Automated Unit Testing merely
as a supporting technique for one or another agile programming method.
Furthermore, beyond the documentation (help files, “cookbooks”) provided with
these tools, code examples are rarely given and real examples of the application
of this powerful technique are few and far between.
This article is the first in a series of articles that aim at
filling the gap that exists, providing a complete introduction to Unit Testing
together with code examples that will try to keep as close as possible to real
life scenarios. Obviously, Unit Testing can be done on many platforms and in
many languages but this article will restrict the scope of investigations to the
.NET platform and will be using C# as its implementation
language.
The meaning of Unit Testing
As can be immediately understood by its name,
Unit Testing is a form of testing. More precisely, it is an attempt to prove the
correctness of one or more pieces of software by putting them to work and
validating their functioning/results against a known set of values/behavioural
elements (the contract). This process can take many forms but the object of this
article will be writing code to test code. Basically, given a piece of code (implementation code) of which the
correctness needs to be established, some more code (test code) is written to exercise the
implementation code and verify that operations performed by it correspond to the
model that is being implemented. One key feature of Unit Testing, in fact the
feature that gives it its name, is that it is all about testing discrete
functional units of the software and not the software system as a whole. This
differentiates it from the aptly named System Testing. A unit test ensures the
validity of a well delimited piece of the system (a unit). In practice, this
will mean that what we test are components, classes, packages, compilation
units… depending on the terminology that applies in the environment for which we
develop. Since our main focus of interest here is .NET which is an
object-orientated programming framework, we will be applying Unit testing to
test classes and components.
What
do you test for?
As briefly stated above, what we test for is the adherence of our
code to a set of pre-determined decisions about its workings. That set of
decisions is generally referred to as the
contract. The contract reflects what we wanted to do when we set out to
write that piece of code. It also states what our users (business customers or
other developers using our libraries) expect from our code. As might be
expected, the contract encompasses many things, all of them important to the
users. But generally, three areas can be isolated as being the target of our
testing efforts.
The general compliance of the results
given by our code with what is exhibited by the model we are trying to
implement. Basically, we are making sure here that given correct inputs/working
conditions, our code will produce the right results/perform the proper
operations. I.e. if our code implements the addition process, it must give the
correct sum of the input elements given to it. Compliance with this part of the
contract determines the correctness of our code.
This area of the contract states the
behaviour expected of our code when the size of inputs given to it is large to
the point of being near overwhelming or when the resources available to our code
are restricted in some way (not enough memory, no ports available, …) It also
describes the expected behaviour of the code in a multi-threaded environment. If
our code performs image processing operations, it must behave properly (not
crash or hang the machine) when given an image that is larger than the memory
available. Compliance with this part of the contract determines the robustness
of our code.
A part of the contract that spells
out the behaviour expected of our code when it is presented with invalid or
wrong input. To reuse the image processing code example, we assume that our code
is asked to open a file that does not contain an image but is in fact a renamed
music file; in that case, the code must not crash or hang or worse “open” that
file and display a nonsensical picture to the user.
Compliance with this part of the
contract determines the reliability of our code.
Benefits of Unit Testing
Like all other kinds of testing, Unit Testing finds errors, and
just like other kinds of testing it is limited in its scope because you can not
be sure to have found every single error. But Unit Testing helps in finding a
number of them. A particularly interesting application of Unit Testing is in
regression testing; that helps ensure that previously found (and eliminated)
errors do not creep back into the software. Because of its discrete nature, Unit
Testing is easy to perform. Each piece of software being tested is small
(relative to the whole) and therefore well-determined. Because of that, Unit
Testing gives an incredible amount of confidence in one’s code and enables
developers to work and apply changes to complex areas of software knowing that
unit tests are there to maintain the correctness of their
work.
Basically two “schools of thought” exist and
each has its prescribed way of doing Unit Testing. The one that is most vocal
advocates for Test-Driven Development or “test first programming”. What this
means is that given the contract that the implementation code must fulfil, the
programmer starts by writing code to test the implementation (which hasn’t been
written yet). Following that, the rest of the development consists in making
sure that the tests formerly written all pass. This way of doing things is
mostly associated with the Extreme Programming camp. Other developers prefer to
write the code first and then write unit tests to ensure the validity of their
code. Obviously, each “side” has many arguments to justify its position and the
intensity of the debate around this is nearly as great as the one surrounding
coding style or language preference. In the next installment in this series we
will see a bit of both ways.
A
glance at Unit Testing
Keep a running update of any changes or improvements
you've made here.
Let us take a quick look at what Unit Testing is really about. We
will be using, for the sake of this example, a very simplistic situation. The
intent here is to illustrate the process of writing unit tests. Let us assume
that we have to write some code to divide integer numbers. We intend to do so by
creating a class ArithmeticOperations
that will expose a method
Division
that
takes two integer parameters and returns an integer, the quotient of the
division of the first parameter by the second. We want to be sure that this code
performs properly by writing some unit tests for it. First, we start by listing
known facts about division. This step is important because these facts are the
yardstick against which we will measure the correctness of our division
code.
- Any number divided by 1 yields itself
- Any number divided by itself yields 1
- 19 divided by 7 yields 2
- 15 divided by 5 yields 3
- 3 divided by 8 yields 0
- No number can be divided by zero
What you can notice here is that “Fact” 1 and “Fact” 2 are general facts about the result of a
division. “Fact” 3, “Fact” 4 and “Fact” 5 each show a special case for a division. They have
been chosen to illustrate the three main situations of a division (and this is
important, unit test code should be
written to exercise all types of situations your code will be confronted with);
i.e. an even division, a division with a remainder and a division that gives 0.
“Fact” 6 shows us a case where division cannot be performed (remember the
failure tests?). It is good to list among the “Facts”, special cases for which
we know the result or behaviour to expect and general rules about the result of
our operations. So armed with these facts, we now write another class to test
ArithmeticOperations (actually we could put our tests in the same class but it
is cleaner to separate them from the implementation
code).
We make sure that we provide inputs that match the conditions of
each “Fact”, and we verify that the result provided or behaviour exhibited by
our class corresponds to what is specified in the
“Fact”.
Our implementation code would look like
this
public class ArithmeticOperations
{
public ArithmeticOperations()
{
}
public int Division(int a, int b)
{
if (b != 0)
{
return a / b;
}
else
{
throw new ArgumentException("Cannot divide by zero", "b");
}
}
}
Notice that our code throws an
exception when zero is passed as a divisor. It is very important that all code
(not only contrived examples) actively reject parameters for
which the operation they are supposed to perform is undefined or impossible to
perform.
Our test code would look like this
public class TestArithmeticOperations
{
private const int numberOfTests = 100;
private const int randomNumbersSeed = 13547;
public TestArithmeticOperations()
{
}
public void TestFact1()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next();
int r = ao.Division(a, a);
if (r != 1)
{
Console.WriteLine("Error dividing {0} by {0}", a);
}
}
}
public void TestFact2()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next(1, numberOfTests);
int r = ao.Division(a, 1);
if (r != a)
{
Console.WriteLine("Error dividing {0} by 1", a);
}
}
}
public void TestFact3()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(19, 7);
if (r != 2)
{
Console.WriteLine("Error dividing {0} by {1}", 19, 7);
}
}
public void TestFact4()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(15, 5);
if (r != 3)
{
Console.WriteLine("Error dividing {0} by {1}", 15, 5);
}
}
public void TestFact5()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(3, 7);
if (r != 0)
{
Console.WriteLine("Error dividing {0} by {1}", 3, 7);
}
}
public void TestFact6()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next();
try
{
ao.Division(a, 0);
Console.WriteLine("Our code pretends to divide by 0");
}
catch(ArgumentException)
{
}
}
}
}
“Fact” 1 through “Fact” 5 are accuracy tests while “Fact” 6 is a
failure test. This article does not illustrate any stress test. This is partly
because the example is trivial, and also because stress testing requires a
discussion of its own. This is therefore left for the next installment to
address.
All there is left to do is create a console project that will
reference the subject class and the test class, and call each method of the test
class.
We now have a full battery of tests that we
can run on our original class (i.e. ArithmeticOperations
). As long as our code
complies with the “Facts” we listed during the test preparation phase, the tests
will run smoothly (no error message). It should be clear by now that the more
“Facts” we know about the code we are writing, the more comprehensive our
testing will be. In the particular case we are studying some more facts could
have been listed (e.g. when multiplying the quotient by the divisor, one obtains
a number which when subtracted from the dividend gives a number always lower
than the divisor. Goodness that sounds complicated!!) but the case is so trivial
that more tests would have made our study disproportionately complicated (if
that’s not already the case J). A trap to avoid is the listing of “Facts” that include one
another. For example, listing a “Fact” that states a rule about division by 2
and another “Fact” that states a rule about division by even
numbers.
Automated Unit Testing and its
tools
The way of doing things displayed in the preceding paragraph,
though functional, tends to be cumbersome and a bit inefficient.
Specifically:
- We need to write some more code to call every
method of our test code.
- In our test code, we have to call methods to
alert the user of problems.
- We are limited to running our tests in a
console environment
- It would take some more code to get statistics
about our tests.
This is where Automated Unit Testing comes into play. What we need
is a tool that would provide us with a framework to write our tests and would
enable us to run those tests. That tool would reduce the amount of
infrastructure code we need to write (all those
Console.WriteLine(...)
we had in
our test code and the extra project we have to create in order to run the
tests). That tool would allow us to
write the tests once and run them in multiple environments (console mode,
graphical mode). Finally that tool would provide us with complete statistics
about our tests (how many did we run, how many passed, which ones passed, where
did the ones who failed fail…)
<pwell, there="" are="" in="" fact="" many="" tools="" like="" that:="" csunit,="" nunit="" and=""
mbunit.="" all="" of="" them="" more="" or="" less="" descendants="" the="" famed="" junit="" (the="" first=""
automated="" unit="" testing="" tool="" to="" have="" gained="" massive="" popularity).="" they="" implement=""
almost="" same="" concepts="" as="" junit,="" released=""
open-source="" software="" but="" unlike="" that="" was="" directed="" towards="" java,="" these=""
tools="" written="" for="" .net.="" this="" article,="" we="" will="" direct="" our="" attention=""
nunit="" it="" is="" most="" widely="" used="" one="" therefore="" has="" support,=""
plug-ins…
<="" p="">
NUnit can be gotten at http://www.nunit.orgg . The latest
version is v2.2 which looks a bit different than the version used in this
article v2.1 but nevertheless implements the same
concepts.
Automated Unit testing tools use a specific terminology that makes
talking about tests easier.
- Every method that tests one of the “Facts”
mentioned earlier is called a test
case.A class (like
TestArithmeticOperations
) which
combines a group of test cases is called a test
fixture
- When testing a component (basically many
classes that work together to provide a user with a specific interface to
accomplish a well-defined task), one writes many test fixtures. The
combination of those test fixtures is called a test
suite.
So, NUnit allows the programmer to write test suites, test
fixtures and test cases. The mechanism through which this is done is by putting
assertionss about your operations in
your test code. For example, assuming there is a variable r in your code that, after some
computation, should have the value 4. Test code as we wrote it in the previous
paragraph would state
if (r != 4){Console.WriteLine("Error");}
<pin nunit,="" the="" code="" would="" look="" like=""
this
<="" p="">
Assertion.AssertEquals(“Error”, 4,
r);That code means, [make sure the value of r is 4; if it is not, fail the test
case with the message “Error”]
There are different types of
assertions:
- Check for exp being null Assertion.AssertNull(msg,
exp)
- Check for exp not being null Assertion.AssertNotNull(msg,
exp)
- Check for exp2 being equal to exp1 Assertion.AssertEquals(msg,
exp1, exp2)
- Check for exp2 being the same as exp1
Assertion.AssertSame(msg,
exp1, exp2)
- Check for exp being a true boolean expression
Assertion.Assert(msg,
exp)
All assertions make the test case fail if the condition they are
testing for is not verified. In that case they issue the message specified by msg.
A special kind of assertion <balways <="" b="">fails: Assertion.Fail(msg)
Once the methods containing the
assertions have been written, they need to be signalled as being test cases. The
NUnit framework defines a way for the programmer to do just that. All that needs
to be done is to add the attribute [Test]]
to the
method.
<ponce all="" the="" test="" cases="" have="" been=""
written,="" you="" signal="" class="" they="" belong="" to="" as="" being="" a="" fixture="" by="" adding=""
the="" [<b="">TestFixture] attribute to the
class.
No special marker is needed to signal a group of classes as a test
suite.
<all these="" attributes="" are="" defined="" in="" the="" nunit.framework="" namespace=""
which="" therefore="" needs="" to="" be="" “used”="" inside="" each="" source="" file.="" and=""
nunit.framework.dll="" assembly="" (copied="" upon="" installing="" nunit="" on="" your="" system)=""
to="" added="" as="" a="" reference="" test="" project.
<="" p="">
The code to our tests now looks like:e:
using System;
using NUnit.Framework;
namespace SimplisticExamples
{
[TestFixture]
public class NUnitTestArithmeticOperations
{
private const int numberOfTests = 100;
private const int randomNumbersSeed = 13547;
public NUnitTestArithmeticOperations()
{
}
[Test]
public void TestFact1()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next(1, numberOfTests);
int r = ao.Division(a, a);
Assertion.AssertEquals("Dividing " + a + " by " + 1, 1, r);
}
}
[Test]
public void TestFact2()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next(1, numberOfTests);
int r = ao.Division(a, 1);
Assertion.AssertEquals("Dividing " + a + " by " + a, a, r);
}
}
[Test] public void TestFact3()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(19, 7);
Assertion.AssertEquals("Error dividing " + 19 + " by " + 7, 2, r);
}
[Test]
public void TestFact4()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(15, 5);
Assertion.AssertEquals("Error dividing " + 15 + " by " + 5, 3, r);
}
[Test] public void TestFact5()
{
ArithmeticOperations ao = new ArithmeticOperations();
int r = ao.Division(3, 7);
Assertion.AssertEquals("Error dividing " + 3 + " by " + 7, 0, r); }
[Test]
public void TestFact6()
{
ArithmeticOperations ao = new ArithmeticOperations();
Random rnd = new Random(randomNumbersSeed);
for (int num = 0; num < numberOfTests; ++num)
{
int a = rnd.Next(1, numberOfTests);
try
{
ao.Division(a, 0);
Assertion.Fail("Our code pretends to divide by zero");
}
catch(ArgumentException)
{
}
}
}
}
}
Once this test code has been written, what is the next step? First
of all, we make sure the whole thing compiles (implementation code and test
code). You may choose to have two separate projects (the best choice for large
scale development) or combine test code and implementation code in the same
project. If you have separate projects, be sure to add a reference to the
implementation project in the test project.
<now, it="" is="" time="" for="" the="" automated="" testing="" part=""
(finally!).
<="" p="">
NUnit offers two operating modes Console and GUI modes. Let us
look at the console mode:
The console application nunit-console.exe located in the bin
sub-folder of the NUnit installation folder runs the test fixtures present in
the test assembly with which it is invoked.
nunit-console <test assembly>
Here is the invocation and results of the tests we just wrote on
the ArithmeticOperations
class.
As you can see, we had 6 tests run (the number of the “Facts” we
defined), none of them failed; we also had the amount of time it took to run
these tests.nunit-console offers more
sophisticated options to run the tests. Help on those are listed by calling
nunit-console without any option. The other mode offered by NUnit is the GUI
mode.
The Windows Form application nunit-console.exe located in the bin
sub-folder of the NUnit installation folder runs the test fixtures present in
the test assembly with which it is invoked.
nunit-gui <test assembly>
After using that command line, the Main NUnit form is
loaded
Click on the ‘Run’ button and see the
progress bar unroll: Green at the beginning and shifting to Red as soon as a
single test is failed. In our case, the bar remains green all through and we end
up with this
<pnow, for="" the="" sake="" of="" example,="" let="" us="" assume="" we="" had="" implemented="" a=""
different="" way="" doing="" division="" in="" our="" <code="">ArithmeticOperations class;
namely:
public int Division(int a, int b)
{
int r = 1;
while((a - b != a) && (a -= b) > 0)
r++;
return r;
}
This way of implementing the division
(which we could have chosen for whatever reason) intentionally has bugs which
our tests can help us “discover”.
Running the tests from the command prompt again give us
this:
<pthis time,="" you="" can="" see="" that="" we="" have="" 3=""
failures="" and="" a="" list="" of="" the="" tests="" failed="" together="" with="" messages="" specified=""
while="" writing="" assertions="" in="" our="" code.="" here="" lot=""
functionality="" is="" provided="" for="" which="" did="" not="" to="" write="" code="" (i.e.="" name=""
the="" test="" case="" failed,="" expected="" behaviour,="" actual="" message=""
associated="" failure)
<="" p="">
Running the tests from the Graphical User Interface gives us
this:
In the right window pane, we have a view of all the tests, those
that passed in green, the ones that failed in red. The progress bar is entirely
red to signify that our class/component is not correct. A window below the
progress bar shows us the details of the test failures (same as in the console
mode).
Conclusion
<unit testing="" is="" not="" a="" new="" technique;="" programmers="" have="" been=""
their="" code="" by="" writing="" for="" as="" long="" there="" were="" complex="" programs.="" these=""
past="" few="" years="" seen="" it="" come="" to="" the="" front="" of="" development="" techniques=""
result="" widespread="" object-orientated="" programming="" (which="" lends="" itself=""
bit="" better="" unit="" testing)="" and="" apparition="" agile=""
methodologies.="" together="" with="" automated="" tools,="" makes="" powerful=""
technique="" that="" no="" programmer="" can="" dispense="" with.="" sudden="" increase=""
testing="" add-ins="" environments="" bears="" strong="" testimony="" fact=""
that="" love="" here="" stay.="" brief="" list=""
most="" useful="" complementary="" tools="" available:
<="" p="">
- Add-ins to Visual Studio.NET (NUnitAdd-in,
VSNUnit…)
- Testing frameworks for ASP.NET
(NUnitAsp)
- Test reporting tools, test coverage tools…
(NUnit2Report…)
Even though Unit testing is not the
ultimate all-error remover weapon (studies have shown that it detects only
30%-40% of all bugs), it is a great tool for making sure that those bugs that
have been removed remain out of the software and as such boosts developer
productivity while helping to maintain software quality standards.
Like any other powerful tool, it must be applied with discipline
or it will harm the development process. One particular pitfall to watch for is
the accumulation over time of unit tests making the software builds
prohibitively long and causing developers to waste an incredible amount of time
running tests every time they want to check in new code to the repository (or
worse stopping to run the tests because of the time they take). Special policies
about unit tests and the build process must then be made and followed. These
issues have to do with continuous integration and will be the object of a future
article.
In all, when properly applied, unit testing has benefits that you
as a developer cannot afford to pass.
In the next installment in this
series, we will look at a general method for writing unit tests building upon
the recommendations given in this article and we will see some detailed examples
of real-world development with unit tests.
We will touch upon Test-Driven
Development with an example of such a process and we will delve into stress
testing discussing some of its complexities.
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.