The Purpose of a Unit Test

I would like to clarify the purpose and intention of a unit test for every role even tangentially related to the development of software. I have observed a steady upward trend, over the last 15 years, for the importance and value of automating the software validation process. I think this is fantastic! What I am troubled by is the large amount of misinformation that exists in the attempts to describe how to unit test. I specifically address and clarify the concept of unit testing in this entry.

There is no doubt the Agile Programming methodologies have contributed to the increase of awareness, content and focus of unit tests. The passion and zeal developers gain for these processes is not surprising. Many of these methodologies make our lives easier, our jobs become more enjoyable. There are many of us that like to pass on what we have learned. Unfortunately, there is a large discrepancy in what each person believes is a unit test, and the information that is written regarding unit tests frightens me.

The Definition of a Unit Test

This definition of a unit test is the most clear and succinct definition that I have found so far:

A unit test is used to verify a single minimal unit of source code. The purpose of unit testing is to isolate the smallest testable parts of an API and verify that they function properly in isolation.

API Design for C++, p295; Martin Reddy

I would leave it at that, however, I don't think that simple definition of what a unit test is will resolve all of the discrepancies, misunderstandings, and misleading advice that exists. I believe that it will require a little bit of context, and answering a few fundamental questions to ensure everybody understands unit testing and software verification in general.

The Goal of Testing Software

We test software to manage risk. Risk is the potential for a problem to be realized. A lower level of risk implies fewer problems. A patch of ice on the sidewalk is not a problem. It merely creates the risk for someone to slip and fall. The problem is realized when someone travels the path that takes them over the ice, and they slip and fall. Poorly written code is like ice on the sidewalk. It may not exhibit any problems. However, when the right set of inputs sends execution down the path with the ice-like code, a problem may occur.

There are many forms risk with software. I believe these risks can be categorized into one of the categories below.

Correct Behavior

A dry, safe sidewalk is useless for us to travel on if it will not lead us to our intended destination. Therefore, an important aspect of software verification is to prove correctness. We want to verify what we wrote, does both what we intended, and expected to create. The computer always does what I instruct it to do, but did I instruct it to do what I intended? We want to verify the software operates as it was designed to function.

Robustness

We want to ensure that our software is robust. Robust software properly manages resources and handles errors gracefully. Software that uses proper resource management is free of memory leaks, avoid deadlock situations, and responsibly manages system resources so the rest of the system can continue to operate properly. Graceful error handling simply means the application does not crash or continue to operate on invalid data.

The Software Unit Test

Now remember, the goal of a software unit test is to manage risk at the smallest unit possible. The target sizes to consider for a unit of code are single objects, their public member functions and global functions. I think we have covered enough definitions to start to correct misunderstandings that many people have regarding software unit tests. When I say people, I am including all of the roles that have any input or direction as to how the software is developed: Architects, programmers, software testers, build configuration managers, project managers and potentially others.

Unit Tests Do Not Find Bugs

Unit tests verify expected behavior based on what they are created to test. Seems obvious now that I stated it right?! It can be so easy to fall into that trap, especially when the word automated is used so freely with "unit test". When the person responsible for the schedule discovers that the tests are not free, they begin to argue that the tests are unnecessary because we have Software Test verify the software before we "Ship It!" It is much more efficient and cost effective to prevent creation of defects, than to try to find them after they have been created. I would like to give some context to where a unit test fits into the overall development process, and how they can improve the predictability for when your product will be ready for release.

One Size Does Not Fit All

It is important to keep in mind that there are many different forms of testing. This holds true whether we are discussing the physical world or the realm of software. Imagine a state of the art television that is on the assembly line. Before the components have made their way to the manufacturing floor, most likely some sort of quality control test was performed to verify these components met specifications. Next these basic pieces are assembled into larger components, such as the LCD display, or encoder/decoder module. These components may be validated as well. Finally, the television assembly is completed by integrating the larger components to the final system.

The unit test phase is similar to the very first quality control check in the television analogy. The objects and the functions created by the developer are verified, in isolation, for quality and that they meet the specified requirements. Compare this phase to the definition that I presented at the beginning of the essay. These unit tests should verify the smallest unit of testable code possible.

At the moment, we are only concerned with unit tests. The graphic below illustrates the different types of testing that I described above. The image correlates the primary beneficiaries and the type of resources that are the most effective for each phase.

Unit Tests are for the Software Developer

Verifying every individual component of the final television will not guarantee that the final television will meet specifications or even work properly. The same holds true for software and unit tests. Unit tests verify the software building blocks that the programmers will use to build more complex components, and then combine the components into a final system or application.

The unit tests can also be organized and run automatically as part of the build process. Each time the software is built, the unit tests will be run along with any other regression and verification tests that have been put in place. Unit tests will continue to provide value throughout the development lifetime of the software. However, unit tests are the most beneficial to the software developers, because they verify the units of logic in isolation before integration.

Unit Tests Will Affect the Schedule

The schedule is always a touchy subject, because it is directly tied to the budget, and indirectly tied to profits. The common perception is that adding the extra task of writing a test along with the code, will extend the amount of time needed to complete the project. This would be true if we developed perfect code, did not re-introduce bugs that we have previously fixed, and always had a complete set of requirements at the beginning of the project. Unfortunately, all three of those are rarely true. These are the circumstances where unit tests will reduce the amount of time required from your schedule.

The diagram below is inspired by a highly esteemed engineer I work with. He simply drew a timeline for two different versions of a project, one that develops unit tests, and one that doesn't. While it may be true that developing unit tests will require more development time, unit tests help ensure that the quality level stays constant or increases, but never decreases. With quality checks like this put in place during each phase of development, the schedule will be more deterministic. When the quality is allowed to waver through the development process, the end of the schedule becomes less predictable.

The situations remain similar regardless of the type of project you are creating, a project with hardware, or a software only product. Unit tests help keep the software portable and adaptable. This means the possibility of developing the logic on different hardware than the intended target and emulators is more of a possibility. This allows dependencies on hardware to be eliminated until the final system integration phase is planned. If your hardware is delayed, or limited in supply, software engineers can continue to work. For software only project, the end of the schedule is simply more predictable.

The Software Developer Writes the Unit Tests

The programmer that creates the software should also write the unit test. I know what many of you are thinking at this point, "The person that created the product should not be the person to inspect its quality." This is absolutely correct. However, remember, we are simply at the smallest possible scale for code at this point. The code units that we are referring to are not products; they are building blocks for what will become the final product. The Software Test team will develop the test methodologies for Acceptance Testing. Therefore a different group is still responsible for verifying the quality of the product.

A process like Test Driven Development (TDD) requires the same person to write both the code and the tests. Software products of even moderate size are too complex to account for design detail before the software itself is developed. The unit tests become much like a development sandbox in which the engineer can experiment. This part of the process happens regardless. To have a separate person write the tests would impose another restriction on the developer.

The developers are responsible for maintaining the unit tests throughout the lifetime of the products development. With an entire set of unit tests in place, changes that break expected behavior can be caught immediately. Running the entire set of unit tests before new code is delivered back to the repository should be made a requirement. This makes each developer take responsibility for all of their changes, even if the changes break tests from other units of code. I think the developer who made the change is the most qualified to determine why the other tests broke, because he/she know what was changed.

Unit Test Frameworks

If you give each developer the direction to "Write unit tests, I don't care how! Just do it!" You will end up with many tests, that will generally only be usable by the developer that wrote the tests. That is why you should select a unit test framework. A unit test framework provides consistency for how the unit tests for your project are written. There are many test frameworks to choose from for just about any language you want to program with, including Ada. Just like programming language, coding guidelines and caffeinated beverage, almost every programmer has a strong opinion which test framework is the best. Research what's out there and use the one that meets the needs of your organization.

The framework will provide a consistent testing structure to create maintainable tests with reproducible results. From a product quality and business view-point, those are the most valuable reasons to use a unit test framework. When I am writing code, I think the most valuable reason is a quick and simple way to develop and verify your logic in isolation. Once I know I have it working solidly by itself, I can integrate it into the larger solutions with confidence. I have saved an enormous amount of time during component and integration phases because I was able to pare down the code to search through when debugging issues.

Unit Tests Are an Asset

I would like to emphasize to anyone in software development, unit tests are an asset. They are an extremely valuable asset, almost as valuable as the code that they verify. These should be maintained just as if they were part of the code required to compile your product. The last thing you want is an uninitiated programmer commenting out or deleting unit tests so they can deliver their code. Because the unit tests can be carried forward, they become a part of your automated regression test set. If you lose any of the tests, a previous bug may creep back in.

How to Unit Test

I'm sorry, it's not that simple. I am not going to profess that I have The Silver Bullet process to unit testing or any other part of software development, because there isn't one. If there were such a process, we wouldn't still be repeating the same mistakes as described by Fred Brooks, in The Mythical Man Month. For those of you that have not read or even heard of this book, it was first printed in 1975. The book is a set of essays based on Brooks' experiences while managing the development of the IBM OS/360. This may possibly be on of the reasons why a new development process emerges and gains traction every 5-8 years.

I will revisit this topic in the near future. There is much knowledge and experience for me to share with you, as well as the techniques that have been the most successful for me. Each new project has brought on new challenges. Therefore, I will be sure to relay the context in which the techniques were successful and when they caused trouble.

Conclusion

Unit testing is such a broad subject that multiple books are required to properly cover the topic. I have chosen to focus only on the intended purpose of software unit tests. I wanted to clarify many of the misconceptions associated with unit tests. Quality control should exist at many levels in the development process. It is very important for everyone in the development process to understand that unit tests alone are not enough to verify the final product. Moreover, having a sufficient set of unit tests in place should significantly reduce the amount of time required to verify and release the final product.

Software unit tests provide a solid foundation on which to build the rest of your product. These tests are small, verify tiny units of logic in isolation, and are written by the programmers that wrote the code. Unit tests can be automated as part of the build process and become your products first set of regression tests. Unit tests are very valuable, and should be maintained long with the code for your product. Keep these points in mind for the next strategy that you develop to verify a product that requires software.