The benefits of automated unit testing

Introduction

Writing software is very difficult. Writing bug-free software all the time is virtually impossible. There are many reasons why this is so; anyone who has written an untrivial piece of software knows how easily a human can miss a tiny detail, and how the slightest of omissions can cause a most spectacular crash, most likely at the most awkward moment.

Unfortunately, the task of writing software promises to be no easier in the future. While software technologies do advance, so does the complexity of the software we write. There is no magical trick in sight in the foreseeable future that would make writing bug-free software simple and straightforward. Still, not all hope is lost - there are some things we can do to substantially improve the quality of software we write.

Ad-hoc Testing

The development of new features to a software product goes in phases. No matter what the methodology, the developer usually starts with the requirements, proceeds to design, then writes the code, debugs it, and finally checks the code into the source code version control system. At this point, the feature is considered done. But how does the developer know that the code actually works?

In many smaller companies and companies that are not really software houses but still develop software for their own use, the developer spends a day or so testing the feature, simply running the application and trying out the functionality with different inputs while observing that the outputs come out as designed. After having completed this testing, the developer has some confidence in that the code works the way it is supposed to. But every developer who has ever written code to an actual product knows the nagging feeling of uncertainty. How good a coverage did the testing have? Did I test every single feature in the application – otherwise how can I be sure that the new code did not accidentally break some other, apparently unrelated, piece of functionality? What if new code written by somebody else two weeks later will adversely affect this feature?

Using developer time for manual ad hoc testing is not a cost effective way of ensuring software quality. Developer time is expensive and is better spent on writing software rather than test-running it ad nauseum. This kind of testing finds out whether the new code affected some other piece of functionality only if the developer tests every single feature in the application. Further, it gives no guarantee whatsoever that the feature still works two months down the road as more features have been added to the system.

Quality Assurance

Some companies have a more or less formal quality assurance process with a team dedicated for testing. The team may have test case scripts that the team goes through and manually tests the software. This activity is carried out after every internal release of the software. Customers will see a new version of the software only after the tests pass and the QA team leader has signed it off.

This kind of testing is better than ad-hoc testing but it is far from optimal. Human testers can and will make mistakes. While hired tester time is cheaper than developer time, the cheapness is offset by the fact that the problems are uncovered relatively late in the development cycle. Every time QA acceptance tests fail, the developers get notified; they fix the problems and give a new version of the software to QA. Time and energy is wasted on communication while the software bounces back and forth between the developers and the QA team. In addition to wasted time and energy, this trashing can have side effects such as creating a ‘ghetto’ mentality in QA personnel.

Quality assurance teams may also have automated testing systems. This is more effective than manual testing, but still is not the optimal situation. Test scripting systems tend to be expensive. Writing good test scripts and maintaining them is something that a tester who was hired for a summer job is not likely able to do. Hiring someone who can write and maintain tests will likely cost as much as hiring a developer in the first place.

Automated Unit Tests

I am not going to describe how to write unit tests in this article. There are already articles on this site that can get you started. Suffice it to say that writing cost effective unit tests is not trivial. It is just like writing any other type of software: it takes time, effort and skill. For now, I’m just going to take the easy way out and list some very basic guidelines for writing and running unit tests.

A unit test is a piece of code that exercises another piece of code. There are essentially two flavors of unit tests: point unit tests exercise a small piece of functionality, e.g., just one method or methods of a class. End-to-end tests exercise one feature across many, sometimes all, layers of an application. An ideal end-to-end test works like an end user story: "the user logs in, invokes the monthly revenue report, sets the date range of the report from July 1st to July 31st, runs the report and observes that all the numbers come up as expected and add up". This is a fairly liberal definition of a unit test; for the purposes of this article anything that you would write e.g., as a NUnit test case is a unit test.

When implementing a feature, a developer should write the code together with a unit test that ensures that the feature actually works. In practice this means writing code that e.g., calls an entry point method of the feature and then verifies that the code did what it was supposed to. For example, if you have a method which transfers money from one account to another, the unit test would call that method with accounts with known balances and then verify that the balances come out the way they are expected to.

When the feature and the unit tests are complete, the developer checks in the source code - including the unit test code - to the source version control system. It is vital that the unit test code is checked in to the version control too. This allows integrating the unit tests with the build. It also allows other developers to check out the unit tests on their local machines and run them at will.

As a rule, before a developer checks in any code into the main source control repository, she should get the latest versions of the unit tests from the version control on her development machine and run all the tests locally. Only after all unit tests pass should the developer check in new code. This ensures that the source code in the version control system remains healthy at all times and can be used to rebuild the system at any time.

In order to get the full benefit from unit tests, the test suite should be run as part of the build process – this is what I mean by ‘automated’ unit tests. The build along with the unit tests should be scheduled to run automatically once or twice a day. If you have unit tests but do not run them as part of a scheduled build, then you are not getting the full benefit from the tests. By running the tests as part of a scheduled build, you test early and often. This again ensures that the source code in the main repository remains healthy. In real life, developers may accidentally check in code that does not work. The daily build should be your main line of defense against these bugs. There are some wonderful and cheap tools for implementing build and test systems, e.g., if you are developing for .NET, you can use NAnt and NUnit frameworks. How to implement a build process is again a fairly large topic and resources on this site and elsewhere on the web can get you started.

Writing unit tests is an extra effort if compared to not writing tests at all, and this should not be ignored. In my experience writing unit tests adds roughly 10-30% to the time it takes to complete a feature. End-to-end tests tend to be the most time consuming type of tests to write. Then again, when writing end-to-end tests one usually very quickly creates a small framework as the common code in the tests is factored out. Thus writing end-to-end tests comes easier after the first few test cases have been written.

Both types of tests - end-to-end and point tests – are needed for best coverage. End-to-end tests find different kinds of problems as they exercise a number of components that participate in the implementation of a feature. Point unit tests are also needed for the most important components because they can verify the functionality of a component more thoroughly.

Benefits of an Automated Unit Test Suite

An automated unit test suite brings along a number of important, tangible advantages as compared to other testing strategies.

First, unit tests find problems early in the development cycle. The common wisdom in software development is that the earlier problems are found, the cheaper it is to fix them. An automated unit test suite finds problems effectively as early as possible, long before the software reaches a customer, and even before it reaches the QA team. Most of the problems in new code are already uncovered before the developer checks the code into source control.

Second, an automated unit test suite watches over your code in two dimensions: time and space. It watches over your code in the time dimension because once you’ve written a unit test, it guarantees that the code you wrote works now and in the future. It watches over your code in space dimension because unit tests written for other features guarantee that your new code did not break them; likewise it guarantees that code written for other features does not adversely affect the code you wrote for this feature.

Third, developers will be less afraid to change existing code. Over time, software systems become more and more change resistant because developers are reluctant to change old code. This is natural because when changing old code, there is always the risk of breaking it or some other part of the system through a side-effect.

The fear of changing code is bad because in one respect software is just like any other physical system: it is subject to the law of entropy. Over time, as new features are added and bugs are fixed, the overall quality of the code degrades. This is even more so if the software is successful since there will be pressure to add new features which do not quite fit into the original architecture. No practical architecture can be designed to accommodate every thinkable feature, and in the changing world there is no way we can foresee all future requirements anyway. The only way to keep adding new features to software and retain the internal quality and clean design over time is by way of refactoring. Without occasional refactoring, the code grows more and more internally tangled until every class knows of every other class. Refactoring and cleaning up existing code is a really scary thing - unless you have automated unit tests.

Fourth, the development process becomes more flexible. Sometimes it may be necessary to fix a problem and to deploy the fix quickly. Despite best efforts, a bug may slip in and an important feature may stop working. The customers cannot purchase products, the users cannot work and your boss is breathing over your shoulder asking you to fix the problem immediately. Releasing quick fixes makes us feel uneasy because we are not certain what side-effects the changes might have. Running the unit tests with the fixes applied saves the day as they should reveal undesirable side-effects. Publishing hotfixes is something we hope we never have to do, and a unit test suite should already decrease the need for such things anyway. But if you ever have to publish a hotfix, a unit test suite improves your chances of doing so without introducing new problems.

Fifth, having a unit test suite improves your project’s truck factor. Truck factor is the minimum number of developers that if hit by a truck the project would grind to a halt. If you have one key developer who works on a subsystem that no-one else knows, then your truck factor is effectively “1”. Obviously, having truck factor of “1” is a risk on the project.

A comprehensive unit test suite improves truck factor because it makes it easier for a developer to take over a piece of code she is not intimately familiar with. A developer can start working on code written by others because the unit tests will guide the developer by pointing it out if she makes an error. Losing a key developer for any reason just before a release is less of a catastrophe if you have the safety net of unit tests.

Sixth, an automated unit test suite reduces the need for manual testing. Some manual testing will always be needed because humans excel at discovering bugs that involve complex data and workflow processes. Writing a unit test for the most complex cases might be so prohibitively time consuming that it is not cost effective any more. The QA team can concentrate discovering the hard-to-find bugs while the unit tests do most of the mundane testing.

The net effect of the benefits listed above is that software development will become more predictable and repeatable – in a word, a bit more like a real engineering discipline. The design and coding phases still have a fair amount of ‘art’ in them, and this will not go away. Once the coding is done, the build process builds and tests the software much like physical products are built on an assembly line. This removes much of the ad-hoc nature in software development which is the underlying reason for many of the problems that plague software projects.

The Future

In addition to the immediate and tangible benefits listed in the previous section, I foresee a future in which unit tests will help in answering to change pressures from two directions. At this point, I would like to point out that the following discussion applies to different kinds of software in varying degrees – it applies best to business software and applications, e.g., CRM, ERP and such. I do not see similar major changes taking place in the development of operating systems, device drivers, database management systems etc.

The raison d’être of a software developer, and thus that of a software company, is to make the buyers and users of the software happy. In more precise terms, the developer should provide the end user with software that corresponds to the user’s real data processing needs in a reliable and predictable manner. This is a very difficult task to carry out and traditionally we software developers have not been very good at it.

I have seen an organization where developers became so frustrated with the end users’ constantly changing requirements that in the end they had the end users’ representative sign off the requirements document before they started to work on the implementation. Later, when the users complained that the software does not address their actual needs, the developers showed them the document: “look, this is what you signed off, and that is exactly what was implemented”. The developers got their moral victory but in the end both the developers and the users lost.

The way for both to win is to start by recognizing the fact that the world is constantly changing. A company’s business needs are always changing and a company that does not adapt will eventually die. To address this, we need to start developing software with what I would like to call Extreme Customer Oriented Rapid Application Development (E-CORAD). The end user’s software will be a smart client with pluggable modules that can be updated independently. The developers will release a module early in the development cycle so an end-user dedicated for this task can try it out; the user will give constant feedback to developers allowing them to refine the module through frequent and small incremental releases. This method of development guarantees that the final product will correspond to end users’ real needs. Automated unit tests will be the major enabler that will allow this cycle to take place frequently without generating chaos. CORAD as such is not a new invention but technological advances such as .NET combined with unit tests will allow us to take it to new extremes.

There are certainly plenty of technical and non-technical challenges in E-CORAD but I do believe the challenges are solvable.

The second direction from which there will be pressure to change the way we develop software is that of service-oriented architectures (SOA). Smart clients will pair with loosely coupled, service-oriented architectures on server side (how’s this for buzzwords in one sentence?). These new architectures will fundamentally change the way we design, develop, deploy and maintain software. Client and server side software will be developed and deployed independently from each other, using abstract contracts as points of contact. Loose coupling translates in practice to less type safety, and thus we will have less help from the compiler in catching those stupid and simple mistakes we make all the time. All this flexibility will result in utter chaos unless we do serious automated testing. A unit test can exercise a Web Service to ensure that it abides to its contract. A unit test can test a Web Service consumer by having it consume mock services that return predefined data.

We need automated unit tests in order to realize the promises of SOA. Also, with SOA, software needs to be highly adaptable, and that again means we will develop software in a frequently occurring cycle of small, incremental releases. This is only possible if we have the automatic assembly line with unit tests that will allow the cycle to take place often without compromising system integrity.

From these changes will emerge a ‘Brave New World’ of software; this is a topic on which I hope to write another article soon.

Unit Testing in the Real World

Despite all the benefits automated unit tests bring us, it has to be said that they are not the silver bullet that will rid us from buggy software for good. Trusting unit tests blindly as in: "if unit tests pass, code is ready for production" is a recipe for shipping bugs. Even if we have a comprehensive unit test suite that is run frequently, bugs can still sneak in and some amount of manual testing by humans is needed.

When a bug makes it into production despite a unit test suite, we need to do post mortem analysis. Whenever this happened in the projects I’ve worked in, one of the following things turned out to be true:

• There were two bugs in the code: the original bug in the application code and another one in the test code. Usually when this happens, it is due to special circumstances in the code being tested. For example, in a web application project I worked on, there was a feature that allowed the user to reset her user account password without logging in. The unit test framework performed a login automatically in the beginning of each test case, which made perfect sense for 99% of the tests. For this particular feature, the automatic login hid a bug that manifested itself only when the user was not logged in. The "reset password" unit test ran happily while the actual feature failed miserably after I added some code that as a side-effect required the user to be logged in. The moral of the story is that while bugs may still creep into production code, it will take two bugs instead of one. The chances for the two bugs occurring together are slimmer than for the first bug occurring alone.
• The production environment is different from the test environment. For example, in the same web application mentioned above, there was a different version of a database driver in the production system than in the test environment. A bug in a query that produced seemingly sensible results in the test environment generated a syntax error in production. The obvious conclusion is that the test environment should reflect the real environment as closely as possible.
• The third possibility of course is that the feature in question did not have a unit test to begin with. This is an open invitation for bugs to come in and crash the party. Even the simplest smoke test that does basic verification is usually worth writing.

Every time a bug makes it into production, we go back to the unit tests and add a test case to specifically verify that this bug never occurs again. This way the noose gets tighter and tighter on bugs and it becomes more and more difficult for the bugs to slip through.

There are developers who think that writing unit tests is not worth the time and effort. Sometimes this objection is due to the fact that the developers equate unit tests with micro unit tests, e.g., unit tests that exercise a single get/set property of a class. Writing tests that test trivial code is not worth the time spent. It is more cost effective to write a test that tests a piece of functionality that would fail if the property did not work properly. As a side-note, with .NET it is so easy to write a generic test that tests get/set properties of a given class through reflection that it actually makes sense to write it once and have it do the trivial tests. Another typical objection is that writing more advanced unit tests requires skill, takes time and can be boring. To this I can only say that the unit test should be considered an integral part of a feature, and writing effective unit tests is like writing any kind of software: it requires skill and expertise.

The downside of having a unit test suite is that it needs to be maintained. If interfaces in application code change, unit tests that exercise the code will break. There is no easy solution to this issue other than convincing the reader that the benefits of the tests outweigh the extra effort in developing and maintaining the tests.

Summary

The recipe for better software with less people is simple: unit test early, unit test often, and refactor when needed. Unit tests find problems automatically, early and will never grow tired of testing the same feature again and again.

A comprehensive unit test suite that runs together with the daily build is the heart beat of a software project. It gives a sense of progress and stability. Email notifications of successful builds with unit tests can boost project morale. An email notification of a unit test failure, on the other hand, tells all project developers where they should focus their attention to in order to get the problem fixed.

Despite all the benefits that unit tests bring, some amount of manual ad-hoc testing is still needed. The developer needs to run the application and use her best judgment to see if the code really does what it is supposed to. A dedicated QA team is also needed in bigger projects. But with a carefully written unit test suite, the software is self-testing and the need for manual testing and separate QA personnel will be reduced. The benefits of automatic unit tests do outweigh the extra time and effort in writing and maintaining the tests.