Click here to Skip to main content
Click here to Skip to main content
Go to top

Getting started with automated white box testing (and Pex)

, , 28 Jan 2009
Rate this:
Please Sign up or sign in to vote.
Pex is a new tool that helps in understanding the behavior of .NET code, debugging issues, and in creating a test suite that covers all corner cases -- fully automatically.

Update: see this article as a video on Channel9!

newpexillustration.png

Introduction

Pex is a new tool that helps in understanding the behavior of .NET code, debugging issues, and in creating a test suite that covers many corner cases -- fully automatically. Internally, Pex operates very much like a professional white box tester: it runs the code, learns about its behavior by monitoring, and finally comes up with new inputs to cover previously uncovered branches by thinking hard about the program behavior.

It only takes a single click to launch Pex from the editor, are you ready to run it on your code?

This articles provides a step by step introduction to Pex in Visual Studio 2008 or 2010 CTP. In particular,

  • how to analyze existing code with a few clicks in the code editor,
  • how to create test cases that reproduce issues that Pex finds,
  • how to debug such issues,
  • how to let Pex generate and save an entire test suite, which is small, yet often achieves high code coverage,
  • how to write parameterized unit tests,
  • why parameterized unit tests will change the way you write unit tests.

Running Pex for the first time

If you want to follow this tutorial on your machine, we assume that you have installed Pex on your machine.

Let's start from scratch, and write some code. We will write a method that turns a string consisting of separate words into a capitalized identifier, where the leading characters of the original words have been turned into upper-case, and dots have been escaped to underscores.

We start by creating a new C# class library to hold the StringExtensions class, and give a first shot at the Capitalize implementation.

public static class StringExtensions 
{
    // convert 'hello world' to 'HelloWorld'
    // punctuation is turned into '_', others are ignored.
    public static string Capitalize(string value) 
    {
        // WARNING: this sample is for demonstration only: it *contains* bugs.
        var sb = new StringBuilder();
        bool word = false;
        foreach (var c in value)
        {
            if (char.IsLetter(c))
            {
                if (word)
                    sb.Append(c);
                else
                {
                    sb.Append(char.ToUpper(c));
                    word = true;
                }
           }
           else
           {
                if (c == '!')
                    sb.Append('_');
                word = false;
            }
        }
 
        return sb.ToString();
    }
}

Right-click into the code of the method, and select Run Pex Explorations in the context menu.

DiggerRunPexExplorations.png

What does it mean to Run Pex Explorations? Pex will run your code, possibly many times, with different inputs. Don't run Pex on code that could launch real rockets!

After a brief moment of deep thinking, Pex starts showing the results of its analysis as a table in a separate Pex Exploration Results window. Each row of the table represents an actual input/output behavior of Capitalize. The values of the input parameter value are shown in the second column, and the corresponding return values in the result column. If an exception is raised, it is displayed in the Summary/Exception column.

DiggerTestTable.png

Why did Pex choose these inputs? Are they special, or simply random? At first sight, the inputs might appear to be just randomly chosen character sequences. But, take a closer look: The inputs contain only the characters ':', 'p', 'J', and '\0'. These characters are representative of punctuation, lower-case characters, upper-case characters, and everything else. Pex picked exactly one representative of these equivalence classes that the program distinguishes. You might see slightly different characters, depending on the mood of the constraint solver that Pex uses under the hood. Think about the following: What would be the probability of randomly chosen inputs to contain punctuation characters, when there are 2^16=65536 different characters?

When we click on a row, we see more details on the right. Here, we see a stack trace of the exception. We could click on the Details header to see details about the inputs that Pex chose for this row.

DiggerTestDetails.png

The .NET guidelines say that a method should never throw a NullReferenceException. We could fix this issue ourselves, or we could simply drop someone an email or file a work item, to let someone else take care of the issue (assuming that we have someone else to take care of our issues). That's what the Send To button is for.

DiggerSendTo.png

But, let's face reality, we have to fix our own code by ourselves. Actually, we don't have to -- for simple issues like this, Pex can do it for us. Click on Add Precondition..., and then on the Apply button.

diggeraddprecondition.png

Pex adapts the beginning of our Capitalize method and inserts a check for a null input, similar to the following code snippet:

public static string Capitalize(string value) {
    if (value == null)
        throw new ArgumentNullException("value");
    ...
}

When we run Pex again, instead of seeing a failing NullReferenceException, we now see the acceptable behavior of an ArgumentNullException. The red cross icon has turned into a friendly green checkmark (on top of which sits a little sun and disk icons, whose meaning we'll explain later).

DiggerExpectedException.png

We have seen how Pex can create a table of interesting inputs and outputs, how to interpret the results, and how to fix an issue that Pex found. So far, all the results that Pex generated lived 'in memory', let's see how we can save them in the solution.

Saving a test, and debugging an issue

Some of the inputs that Pex lists in the table look strange. It would be nice if we could debug the code for such an input and step through the code lines. Actually, that's easy with Pex! Select a row, and click on the Capitalize Save Test... button.

DiggerSaveTest.png

Pex shows a dialog that illustrates a sequence of steps that Pex will take: Pex will create a new test project, then perform many more little steps that we'll ignore for the moment, and finally, Pex will create a Unit Test with the test inputs of the currently selected row.

DiggerCodeUpdatePreview.png

What is a Unit Test? A Unit Test is a method that takes no parameters, similar to the Main method of an application. When a Unit Test raises an unexpected exception, it fails, otherwise it passes. There are a number of frameworks for .NET (NUnit, MbUnit, VSTest, ...) that automate the detection, execution, and reporting of such tests. In this tutorial, we use VSTest which comes with the Visual Studio Professional installation. Pex can also work with other test frameworks; check on CodePlex whether a custom extension for your favorite test framework has been written already.

The generated project contains several files. The C# source code of the generated saved test that corresponds to the current row in the table is stored in StringExtensionsTest.Capitalize.g.cs (similarly to designer files, this file is hidden under StringExtensionsTest.cs; unlike the designer files, you should really not edit the hidden file, as Pex may later on delete or regenerate its content).

diggersolutionexplorer.png

We should now see the following C# code of the generated saved test case. The [TestMethod] attribute denotes that the method is a Unit Test, and [PexGeneratedBy(...)] attribute denotes that Pex generated the test.

[TestMethod]
[PexGeneratedBy(typeof(StringExtensionsTest))]
public void Capitalize03()
{
    string s;
    s = this.Capitalize("\0");
    Assert.AreEqual<string>("", s);
}

Go back to the source code of Capitalize, and set a breakpoint on the first line.

diggerbreakpoint.png

We can now press the Debug button in the Pex Exploration Results window under the details of the selected row.

diggerdebug.png

We can now step through the code line by line.

Saving the test suite for VSTest

At this point, we can not only save individual tests, but we can save the entire table as a test suite. This test suite can serve as a regression test suite in the future, or as a fast-running build verification test (BVT) suite, a test suite that can be executed every time a code change is submitted to the source repository.

To save the entire table, select a row in the table, and press Ctrl-A. Then, click on the Save... button to the right of the selected rows.

DiggerSaveAll.png

Again, Pex will show the dialog that details all the individual steps that Pex performs to save the tests as C# code. Press Apply.

We now have an entire test suite that we can execute without Pex. The generated tests are automatically recognized by VSTest, the Unit Test framework built into Visual Studio. Select Test->Windows->Test View.

DiggerTestView.png

Select a test, and select all by pressing Ctrl-A, and press on the Run Selection button in the Test View window to run all tests:

DiggerTestViewRunAll.png

Running the tests at this time will simply reproduce the same results that Pex reported earlier. However, running the tests in the future might discover breaking changes in the program behavior.

A glimpse of Parameterized Unit Tests

We didn't point it out yet, but when we saved the tests, Pex not only created (traditional) Unit Tests that covered all the corner cases, but also created a Parameterized Unit Test (PUT) stub for the Capitalize method:

[TestClass]
[PexClass(typeof(StringExtensions))]
[PexAllowedExceptionFromTypeUnderTest(typeof(ArgumentException))]
public partial class StringExtensionsTest
{
    [PexMethod]
    public string Capitalize(string value)
    {
        // TODO: add assertions to method StringExtensionsTest.Capitalize(String)
        string result = StringExtensions.Capitalize(value);
        return result;
    }
}

This method lives in the file StringExtensionsTest.cs, while the individual (traditional) Unit Tests were saved in StringExtensionsTest.Capitalize.g.cs. The ...g.cs file should never be modified by hand. If we want to customize the way Pex generates tests, or if we want to add checks that verify that the tested code works correctly, then we should always edit the PUT in the top-level file.

For example, here we could adapt the PUT to check that Capitalize does not change the number of letters:

[PexMethod]
public void CapitalizeMaintainsLettersCount(string input)
{  
    string output = StringExtensions.Capitalize(input);
    Assert.AreEqual(
        LettersCount(input),
        LettersCount(output));
}
static int LettersCount(string s)
{
    return Enumerable.Count(s,
        c => char.IsLetter(c) || c == '_');
}

Another PUT could check that Capitalize is idempotent, i.e., if we capitalize an already capitalized string, then we get back the same string:

[PexMethod]
public void CapitalizeIsIdempotent(string input)
{
    string capitalized = StringExtensions.Capitalize(input);
    string capitalizedTwice = StringExtensions.Capitalize(capitalized);
    Assert.AreEqual(capitalized, capitalizedTwice);
}

We can state that the result of Capitalize only contains letters and underscores:

[PexMethod]
public void CapitalizeReturnsOnlyLettersAndUnderscores(string input)
{
    string output = StringExtensions.Capitalize(input);
    PexAssert.TrueForAll(output,
        c => char.IsLetter(c) || c == '_');
}

You get the idea.

Parameterized Unit Tests vs. traditional Unit Tests

What's the big deal with Parameterized Unit Tests (PUTs)? Well, they allow you to separate two concerns that are often lumped together when writing Unit Tests:

  • the description of the intended program behavior,
  • the particular inputs that are needed to cover corner cases in the code.

While you have to write the PUTs, a tool like Pex can take care of generating the test inputs that cover the corner cases. Pex saves these inputs as traditional Unit Tests. As a result, you have to write and maintain less test code: When you change the corner cases in the implementation, you can simply re-generate the individual Unit Tests from the Parameterized Unit Tests with Pex, and you are back to high or full code coverage, or Pex might point out which corner cases cause failures.

You can (and should) start writing PUTs even before writing any code. And, you can mix PUTs with traditional Unit Tests, when it just doesn't make any sense to factor out values as parameters.

Further Reading

Further watching

Further questioning

  • MSDN Forums, all your questions, feedback, bugs, etc...

History

  • 01/28/2009: Added Channel9 link
  • 01/13/2009: Updated links.
  • Initial version.

License

This article, along with any associated source code and files, is licensed under The Microsoft Public License (Ms-PL)

Share

About the Authors

Jonathan de Halleux
Engineer
United States United States
Jonathan de Halleux is Civil Engineer in Applied Mathematics. He finished his PhD in 2004 in the rainy country of Belgium. After 2 years in the Common Language Runtime (i.e. .net), he is now working at Microsoft Research on Pex (http://research.microsoft.com/pex).

Nikolai Tillmann
Software Developer (Senior) Microsoft Research
United States United States
Nikolai Tillmann is a Principal Research Software Design Engineer at Microsoft Research.
 
His main area of research is program specification, analysis, testing, and verification. He is leading the Pex project (http://research.microsoft.com/Pex), a framework for runtime verification and automatic test case generation for .NET applications based on parameterized unit testing and dynamic symbolic execution.
 
Previous projects he worked on include AsmL, an executable modeling language, and the Spec Explorer 2004 model-based testing tool. He contributed to XRT, a concrete/symbolic state exploration engine and software model-checker for .NET code. Spec Explorer 2007 is based on this engine, which is now productized internally by the Protocol Engineering Team at Microsoft to facilitate quality assurance of protocol documentation.
 
He received his M.S. ("Diplom") in Computer Science from the Technical University of Berlin in 2000.

Comments and Discussions

 
GeneralBroken Link PinmemberFabio Franco13-Jan-09 7:19 
GeneralRe: Broken Link PinmemberJonathan de Halleux13-Jan-09 10:32 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web04 | 2.8.140926.1 | Last Updated 29 Jan 2009
Article Copyright 2008 by Jonathan de Halleux, Nikolai Tillmann
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid