Checklist for Unit & Unit Integration Testing

Introduction

Over time, I compiled a list of things regarding unit and unit integration tests; books, articles, forums and blogs. I have not always kept track of where I found some piece of information. So the references at the bottom of this article are not complete.

Recently I discovered the XUnit Test Patterns website by Gerad Meszaros (I wished I found that one earlier). Still I wanted to share with you the following list. This can maybe help to verify your effort in organizing and/or conducting unit (integration) tests. Maybe you have points to add? Or maybe you disagree? In any case, I would like to hear from you.

Background

Unit testing is a process of testing the individual subprograms (classes), subroutines or procedures (methods) in a program. Most of the times, these “units” are tightly coupled with other modules. Sometimes the latter are stubbed out to assure that we only test the unit under test and not the secondary units. Unit integration testing concerns the inter-operation between the different units a developer has made. (Integration in the small.)

Points of Attention

• Do you know how to incorporate the unit testing into the development process?
  • Unit tests are written by the developer who writes the code and is executed by the developer.
  • Feedback period between coding and finding error is smaller.
  • Coder and tester is the same person.
  • Accounted in the development budget.
  • Continuous testing versus phase-end testing.
  • Avoid arguments like: I don’t have the time to conduct unit test. I must deliver ASAP.
  • Don’t rely on phase-end tests to find your errors.
• Do you have a strategy to incorporate unit test code in production code organization?
  • Be prepared to see as much production code as test code so good organization is primordial.
  • Integration in source control system is needed.
• Are you aware of the type of errors unit testing tries to find?
  • Some errors are easier to find with unit (integration) test than in formal phase-end tests. Or at least the effort is bigger to find those errors.
• Do you use an automated tool such as NUnit or VS2005 Integrated testing Framework to make the unit-testing process easy and repeatable?
  • The same Framework is available for every developer. The same framework can be used by every developer. Automation in two ways: invoking the tests and checking the results.
  • Can be integrated into automatic build/continuous integration scheme.
• Do you know which type (layer) of code benefits the most of unit testing?
  • Some type of code is more appropriate for unit testing than others, for example business logic versus User interface.
  • Whenever you are tempted to write something into a debug.writeLine type of thing, write a test for it.
• Do you know how to deal with code that is too hard to unit test?
  • Consider other ways of testing
  • GUI skin code, main methods, asynchronous method, multi-threading
• Do you know how to deal with developers that don’t write unit code?
  • Education
  • Take away resistance by showing advantages
• Do you how to avoid that unit-testing becomes de-prioritized when deadlines come near?
  • Include into Planning (10%- 30% extra?)
• Do you have the right mindset to conduct unit test?
  • Showing that it works versus finding errors versus verifying expectations (specification) compared to the realisation (code).
  • Are you prepared to scrutinize your work?
• Do you know when to write unit test?
  • Write the tests as you go along.
  • Don’t leave them for a “later” that may never come.
• Do you know when to execute unit tests?
  • Run unit tests as often as possible to decrease feedback period between coding and finding error.
• Do you know how to integrate unit testing into the build process /continuous integration scheme?
  • Unattended execution of unit tests to let the test run as much as possible.
• Do you have a policy about how to deal with failed unit test?
  • Make sure code passes unit tests and integration tests before you check it into the source control system.
• Do you make a distinction between pure unit test and unit integration test and when to use them?
  • Make tests for collaborating units in function instead of big bang approach in system testing.
  • Unit tests are very good for regression testing during development.
• Do you have a policy to distribute knowledge about conduction unit testing and unit (integration) testing and automated testing in general?
  • Optimal use of resources
• Do you have a policy to use unit test for (formal) regression testing?
  • Changes to an existing product may not result in breaking unit tests.
• Do you have a policy to keep the quality of unit tests high?
  • Avoid that the first unit tests are good and the remaining unit tests are of lesser quality (coverage) because of time pressure.
• Do you consider the unit test code on the same level as production code?
  • Be prepared to see as much production code as test code.
  • Test must be written and maintained to the same professional standards as your production code
• Did you code to Interfaces rather than Classes?
  • To plug in different implementations at runtime or test time.
  • Makes it easy to provide test stubs that don’t need a sophisticated implementation of collaborators’ functionality and hence are freed from needing to depend on supporting infrastructure.
• Do you use Inversion of Control/dependency injection pattern?
  • Makes it easier to feed the class under test with alternative implementations of collaborating classes.
• Does each class have a well-defined set of responsibilities?
  • So it will be more easy to come up with unit tests. High cohesion, low coupling
• Did you expose too much of a class for the sake of testability?
  • Don’t mark a method public just for the sake of testability.
  • Keep encapsulation in mind. VS2005 test framework can deal with class and methods marked friend or private.

• Did you separate unit test code from the code under test?
  • Unnecessary code to deploy.
• Is there one test class per class under test?
  • Code generation of VS2005 test framework encourages it and uses it when you generate additional test method skeletons

• Is there a separate assembly per assembly under test?
  • Code generation of VS2005 test framework encourages it and uses it when you generate additional test class skeletons
• Is there a separation between tests that need configuration files and the ones that don’t need configuration files?
  • You can more easily identify which test should be able to run on any machine without setup (eg. Build server)
  • Pay attention on place of test data /configuration files because of the way xUnit framework works.
• Is there a separation pure unit test and unit integration test?
  • With unit integration tests, you must always verify if the collaborating units didn’t cause an error when a test should fail.
• Did you separate database access code test?
  • These kind of tests require much more attention in preparation, configuration and execution speed.
• Is the unit testing code under source control? Is it part of the code base.
  • Any developer must be able to use it afterwards.
  • An automatic system (Build/CI) must be able to use it.
• Can all test methods run autonomous so they don’t rely on other tests or require that they are run in a particular order?
  • Test method can be selected/de-selected via the Test UI so test must run independent of each other.
  • VS2005 test framework allows ordered test runs (more functional like testing, use case scripts)

• Is every test able to run over and over again, in any order, and produce the same results?
  • Otherwise automation of the assertions is difficult.
• Did you use mocks or stubs to accomplish test repeatability?
  • Use mock objects /stubs help you to isolate the item under test and keep it independent from the environment because they give always the expected values.
• Did you consider positive unit tests?
  • To exercise the code as intended and verify the right result.
• Did you consider negative unit tests?
  • To intentionally misuse the code and verify for robustness and appropriate error handling.
• Did you consider "Stress" tests?
  • To check the code for its limits (overflow , etc).
• Did you consider data conformance test?
  • To check whether an result value conforms to an expected format.
  • To check how the code deals with input value that conforms with expected data format and vice versa.
• Did you consider Ordering tests?
  • To check whether the set of result values are ordered or unordered as appropriate.
  • To check how the code deals with input values that are ordered or unordered.
  • Did you consider range tests?
  • To check whether the result falls within reasonable minimum and maximum values
  • To check how the code deals with input values that fall within reasonable minimum and maximum values.
  • To check how the code deals with input values that fall outside the boundaries of reasonable minimum and maximum values.
• Did you consider reference tests?
  • At the end of the method, post conditions are those things that you guarantee your method will make happen.
  • Direct results returned by the method are one obvious thing to check, but if the method has any side-effects, then you need to check those as well.
• Did you consider existence tests?
  • To test whether the result exist (e.g., is non-null, non- zero, present in a set, etc.)?
  • To test how the code deals with empty or missing input values (such as 0, 0.0, “”, or null).
• Did you consider cardinality tests?
  • To test whether there are exactly enough values in the result.
  • To see how the code deals with input value list that is of the correct size or contains duplicates.
• Did you consider process logic test?
  • To check if all control structure constructs (if, else, case, etc.) function correctly. Did you consider which part of the code is reasonable for testing invalid parameters?
  • Check input at the boundaries of the system, and you won't have to duplicate those tests inside the system. Internal components can trust that if the data has made it this far into the system, then it must be okay. You still can take the pessimistic way but then be prepared to see a lot of extra unit tests.
• Do you use code coverage tools?
  • In order to verify the thoroughness of the unit tests
• Do you write tests for bug-fixes?
  • To demonstrate the bug isn’t present anymore.
• Do you test for specific number/datetime problems like roundings , regional settings?
  • Decimal sign, DD/MM/YYY versus MM/DD/YYYY
• Does the unit test method only test one specific thing?
  • When a unit test breaks, you can more easily pinpoint the problem. Maybe even without debugging.
• Did you consider the use of a Mock library or a stub while unit integration testing?
  • A stub is a piece of code used to stand in for some other programming functionality. A stub may simulate the behaviour of existing code (such as a procedure on a remote machine) or be a temporary substitute for yet-to-be-developed code. Stubs are therefore useful in unit testing.
  • Mock objects are simulated objects that mimic the behaviour of real objects in controlled ways. In a unit test, mock objects can simulate the behaviour of complex, real (non-mock) objects and are therefore useful when a real object is difficult or impossible to incorporate into a unit test.
• Are the test suites enough self-documenting?
  • Documentation in the form of meaningful test case names and assertion messages is usually more important than comments on test cases itself.
  • When a test fails, the output will show the failed assertion (or an error) and the name of the test in question
• Is the name of the test method intention-revealing?
  • Name unit test shows up in interactive screen reports and console output
• Did you avoid test method names with appended numbers to distinguish between them?
  • That doesn’t do a very good job of communicating your intentions.
• Is there a consistent naming convention for the unit test code?
  • So you know what are the test methods and the help methods.
• Did you consider putting setup code that is used in every test method into separate initialization XUnit framework methods that are triggered on test class or test method level?
  • Excessive data setup can obfuscate a unit test beyond comprehension so hide it if needed.
  • All state is known in advance and will always be the same no matter where or when your test is run.
  • To avoid one unit test leaves some kind of dirty data hanging around
  • Preconditions that must be true when you want to run the method under test.
• Is your unit test fast?
  • Many unit tests are launched often so fast execution avoids friction and encourages you to run them often.
• Did you consider factory methods that create the object instance under test for you?
  • You can then reuse that method to get fresh instances of your class under test in other test methods.
  • This helps to keep the tests maintainable across time and guards your tests from unforeseen changes to the code under test.
• Is the actual act of executing the method under test separated from the act of asserting on the result by creating a result variable on a different line?
  • The invocation against the object under test may be very long and might make your Assert line stretch all the way beyond the edge of the screen, forcing the test reader to scroll to the right.
• Does the variable that contains the result have a readable name?
  • Makes your Assert line very understandable and easy to read
• Did you consider using separate verification methods? This is a reusable method in your test class that contains an Assert statement but that can take different inputs and verify something on them. You can use these verification methods when you are asserting the same thing over and over again with varying inputs. Even though the Assert is located in a different method, if the Assert fails you'll still get an assert exception and the original calling test will be shown in the test failure output window. Can improve readability.
• Did you avoid multiple Asserts in a Single Unit Test?
  • After a failure, subsequent Asserts aren't executed.
  • These unused Asserts could provide valuable data (or symptoms) that would help you quickly narrow your focus and discover the underlying problem.
  • Put the additional Asserts should be run in separate, self-contained unit tests so that you have a good opportunity to see what fails.
• Did you use an informative assert message?
  • A good Assert message should always explain either what should have happened or what happened and why it's wrong. Shows up in interactive screen reports and console output Test code maintenance
• Did you avoid code duplication?
  • Refactor the Test Suite as necessary and identify areas in which you can reuse code to speed up writing individual test cases

References & Recommend Reading

• xUnit Test Patterns: Refactoring Test Code, Gerard Meszaros, 2007 by Addison Wesley Professional
• The Art of Unit Testing, Roy Osherove, Manning 2008 (MEAP 2007)
• Next Generation Java Testing: TestNG and Advanced Concepts, Cédric Beust and Hani Suleiman, Addison-Wesley 2007
• Pragmatic Unit Testing in Java with JUnit, Andy Hunt and Dave Thomas, Pragmatic Programmers LLC 2004
• Pragmatic Unit Testing in C# with NUnit: The Pragmatic Starter Kit, Volume II, Andy Hunt and Dave Thomas, Pragmatic Programmers LLC © 2004
• Coder to Developer: Tools and Strategies for Delivering Your Software, Mike Gunderloy, Sybex 2004
• EJB Design Patterns, Floyd Marinescu John Wiley & Sons 2002
• Code Craft: The Practice of Writing Excellent Code, Pete Goodliffe No Starch Press 2007
• Code Complete, Second Edition, Steve McConnell, Microsoft Press 2004