White box testing with PEX

• Download source code - 624 KB

Introduction

In this article, we will see problems related to maintenance of Unit test suites, using PEX tp help with the solution, and various PEX concepts like  PEX Factories, PEX Method (PUT), Parameterize Mock and Partial Stubs.

Using PEX as a white box testing tool

Problems

Unit testing is nothing new, it has been there for decades and it has significant history regarding the notion of unit testing. Various software development processes were developed like TDD (Test Driven Development), BDD (Behavior Driven Development) around unit testing. Unit test has many pros and cons and it is beyond the scope of this article. But a basic understanding of Unit testing is required for the remaining article to make sense.

I am going to discuss here about the problem maintaining Unit test cases. When a project is in the developmental stage, zest among the development team is high, they read specs, create test cases, and write code using TDD notions. All good! Team produces thousands of unit test cases. These pile up in Unit test suite silos. Team has a zeal to set up continuous integration where these piles of test cases are executed and checked-in fortnightly, and test reports are sent via automated emails to the relevant stockholders in the team.

Till now, all good! Now, as the deadline approaches, tension runs high in the team, there are lots of fixes in the code in the test cycles, but developers start to cut corners and fail to update test case for code changes. Many unit tests start to fail, code coverage goes down, and so does code quality. Every new line the developer is writing in gust has a potential to introduce a new bug in the system. These bugs are related to boundary conditions, unhandled exceptions, and architecture violation of patterns.

At the end of delivery, unit test suites contain many fail tests, reasons remain best unknown, code which was shiny in the beginning of the project is now dilapidated, in disrepair, and breaking at various places now and accost with unpredictable conditions.

As a result of this, maintenance support increases, integration havocs are created, and hereafter cost shoot ups, which in turn puts a frown on the client face.

Once we understand the problem, there is a quest for automation tools which would find hidden bugs in the code, sneaks into each and every line of code, traverses to branches, and unfolds the areas where code could break, and act as a savior in the midst of code mayhem which last minute pressures and cut corners have created.

Basically, in technical language, we are looking for a white box testing tool, which reads out lines of code and tells us where we have gone wrong. It also addresses the staling test suites issues, where newly written lines of code are automatically explored and checked for bugs without writing additional test cases to cover them.

PEX

To save the day, let me introduce PEX. PEX is a white box testing tool which is still in beta (at the time I was writing this article) and was developed by the Microsoft Research and Development team in Redmond, WA.

I am not going to write a lot about PEX, as lots has been written about it. You can find more information at http://research.microsoft.com/en-us/projects/pex/. I am going to focus on the methodology of using PEX unit test cases to solve the problems we described above.

Goal

1. To explore our code, looking for boundary conditions, crash tests, and unhandled exceptions in an autonomous fashion without much effort needed in writing test cases.
2. Keep unit test suites from getting stale over a period of time, as new lines of code are being written.

Solution

To demonstrate PEX, I have created an over simplified project as shown below. This project has three layers (Entity, DAL, and Insurer). The layer architecture is as given below:

Insurer and DAL both refer to Entity, and Insurer refers to DAL.

Here we are going to test two methods in the Insurer class.

First one is IsPersonEligible, which validates if the Person is eligible or not for insurance based on the business rules as defined in the code:

public virtual bool IsPersonEligible(Person p)
{
    Contract.Requires(p != null);
    List<string> SelectedAreasZipCodes = new List<string> { "400001", "400002" };
    if (p.Age > 17 && p.Age < 45 & SelectedAreasZipCodes.Contains(p.pAddress.ZipCode))
    {
        dal.saveInsurePerson(p);
        return true;
    }
    else
    {
        return false;
    }
}</string></string>

DAL is injected into the Insurer class via the constructor using IOC principles.

public class Insurer
{      
    public Insurer(DAL.IDAL dal)
    {
        this.dal = dal;
    }
}

Now, let us concentrate on the testing of the first method:

public virtual bool IsPersonEligible(Person p);

In order to write a manual unit test, you need to stub the person object to a particular state. For more information on stub and mock, refer to http://martinfowler.com/articles/mocksArentStubs.html.

Three test cases have to be written in order to cover 100% of the test method.

• Unit Test one: Pass Person as null and expect the contract null exception.
• Unit Test two: Stub Person and set Person object to some state, which doesn’t fulfill business rules and expects method to return false.
• Unit Test three: Stub Person and set Person object to some state, which fulfills business rules and expect method to return true.

Now, when developer writes all three test cases, it assumes code coverage of this method is 100% and code is fit to be used. But do you think these test case are sufficient? Do they cover boundary conditions, unhandled exceptions, and hidden bugs?

Ten-ta-ten, there comes PEX to the rescue. For quick run, just right click anywhere inside the method body, and select “Run Pex”.

And see the magic happen. Pex will explore your method and tries to find out all the bugs arising due to boundary conditions, unhandled exceptions, and hidden bugs.

It will, on its own, explore the code under test, create the stubs on Person is such a manner that it traverse to each branch in the code in order to get 100% coverage of your method.

Above is the result you get when you run PEX on the code.

But life is never so easy, we do not write simple objects, our systems are full of complex objects and interactions between them. When you unit test code which involves creation of a complex box, PEX will initially try hard to figure itself about object creation, but for modern day objects like controllers which deal with httpcontext, viewbag, and various other dependencies, PEX fails to create such an object.

In that case, you need to create Pex Factories methods which will help PEX to create an object in a particular state, which can be used to test your subject under test. Here, I have underlines particular state, because when you create a factory method, you instantiate the object in a particular state, which is needed by the code under the state. For example, when you instantiate a controller, the controller holds the httpsession state, and the method you are testing expects a certain value to be stored in the session bag before this method is called. Hence, in the factory, you add the particular value in the session when setting the httpcontext to the controller.

Also, it’s not always possible to set state in the factory. Consider this scenario, where you are sharing the factory across many methods. You will set the state which is generic to all methods, but specifically once, you need to set in a special unit test method which PEX will use to explore. These special methods are called Pex Methods.

Enough of abstract talking, let's see an example.

public virtual bool IsPersonEligible(Person p)

Now, in order to test this method, we need to first instantiate the Insurer class which contains this method. So what is the problem? If you check the constructor of Insurer, there is just one public constructor which takes IDAL (data access layer dependency) as parameter.

PEX is in problem, because PEX doesn’t know how to mock and pass an IDAL object to the Insurer class. PEX factory methods come to the rescue!

We need to create a factory method for the Insurer class, which PEX will use to instantiate the Insurer class whenever it needs to initiate it.

/// <summary>A factory for SampleApp.Insurer instances</summary>
[PexFactoryMethod(typeof(Insurer))]
public static Insurer Create()
{
    DAL.Moles.SIDAL Mdal = new DAL.Moles.SIDAL();          
    Insurer insurer = new Insurer(Mdal);
    return insurer;
}

Here, in factory, I have used Moles for mocking IDAL and pass it to the Insurer object. I won’t discuss Moles there, as it will require a separate set of posts to describe it. But you can find more about Moles at http://research.microsoft.com/en-us/projects/moles/.

Also, we would create a PEX method for IsPersonEligible. Just click under the method and select Create Parameterized Unit Test, follow the wizard, your favorite test framework, than ‘pufs!’, it is done. Pex has created a boiler plate PUT (Parameterize Unit test) for you and you could either use it as it is or customize it to suit your needs for testing. 

/// <summary>Test stub for IsPersonEligible(Person)</summary>
[PexMethod]
public bool IsPersonEligible([PexAssumeUnderTest]Insurer target, Person p)
{
    bool result = target.IsPersonEligible(p);            
    return result;
    // TODO: add assertions to method InsurerPexTest.IsPersonEligible(Insurer, Person)
}

To explore your method, just right click inside PexMethod and click Explore.

Up till now, I have covered two concepts, Pex Factories and Pex Method. Pex Methods are PUT. Now, let us see how these two concepts help to solve our problems.

Once you have written your Pex factories, and Pex methods, you can setup Pex to run in a continuous integrated environment. I.e., whenever a developer checks in code or you can set it fortnightly if the team is big (10+ developers), Pex would run its exploration using Pex Factories and Pex Method and emails you the report with exceptions in your code when ka-put! Aai-la!

To see the point here, no new test cases were written, no effort taken for code review, still boundary conditions, unhandled exceptions, crash values detected. Hence it improves the bug detection induced in code, and fixing those bug improves code quality.

I want you to see my point in action, consider the method under test below:

public virtual bool IsPersonEligible(Person p)
{
    Contract.Requires(p != null);
    List<string> SelectedAreasZipCodes = new List<string> { "400001", "400002" };
    if (p.Age > 17 && p.Age < 45 & SelectedAreasZipCodes.Contains(p.pAddress.ZipCode))
    {
        int zipcode = Convert.ToInt32(p.OtherAddresses[0].ZipCode);
        dal.associateZipCodewithPersonIndex(p);
        dal.saveInsurePerson(p);
        return true;
    }
    else
    {
        return false;
    }
}

The code in bold is added by a zombie developer which will break the elegance. What if there is no address in Person OtherAddresses collection, index 0 will throw index out of bound exception. What if zip code in OtherAddresses is null or not numeric? This will throw a null exception or parse exception. What if zip code is larger than int max value? This will throw an overflow exception. Developer didn’t think about the boundary conditions and exception handling, and PEX will find all that out.

These exceptions maybe be caught with manually written test cases too, but in a manual test case, you need to manually create a Person stub. If the stub contains OtherAddresses, and has a well behaved value set, boundary condition exceptions won’t be caught.

In PEX exploration, PEX generates the Person stub and hence will uncover hidden exceptions.

In order to improve code, the developer needs to put a check for boundary conditions and implement proper exception handling.

In war, a knight needs helpers in the form of soldiers. These are parameterized mocks and partial stubs. To understand this, let us take another method as an example.

The second method in the Insurer class is GetInsureAmount, which retrieves the insured amount of the person from the DAL repository after checking if the person is eligible as per business rules.

Here, DAL is the data access repository which is not implemented as it is beyond the scope of unit testing for the Insurer class.

public float GetInsureAmount(Person p)
{
    float rtnAmount  = 0.0f;
    if (this.IsPersonEligible(p))
    {
        Policy ply = dal.getInsursePersonPolicy(p);
        if (ply.InsureAmount > 0)
            rtnAmount = ply.InsureAmount;
        else
            rtnAmount = 0.1f;
    }
    else
        throw new NotEligibleException("Person " + p.Name + "not eligible for insurance");
    return rtnAmount;
}

If we study the method, this method has two nested branches. In order to pass the first branch (if-else) for positive flow, we need a stub person object in such a state that it passes the condition. IsPersonEligible would return true with that stub.

In order to pass through the second branch, we need to return the policy from dal.getInsursePersonPolicy such that it has InsureAmount greater than zero. Wait, do you see the problem here? dal is a mock layer which we are injecting in the Insurer constructor in the Pex factory for Insurer. The mock layer doesn’t have a body and then will not return any policy when explored with Pex.

Solution? We need to fake the call dal.getInsursePersonPolicy in such a way that it returns a policy with InsureAmount greater than zero.

How do we do it? In the Pex Factory for Insurer,

[PexFactoryMethod(typeof(Insurer))]
public static Insurer Create()
{
    DAL.Moles.SIDAL Mdal = new DAL.Moles.SIDAL();

    Mdal.GetInsursePersonPolicyPerson = ((p) =>
    {
        Policy ply = new Policy();
        ply.PolicyNumber = 123;
        ply.InsureAmount = 100;
            return ply;
    });

    Insurer insurer = new Insurer(Mdal);
    return insurer;
}

The code that is in bold represents the fake call to dal.getInsursePersonPolicy leveraging the Mole framework delegate capabilities.

Just run the Pex exploration with this method and see the outcome. Pex is not able to cover 100% of the code in the method under test. Why? Because we stub Policy which returns InsureAmount as 100 always, and hence Pex was never able to explore the else condition in the second branch.

Solution? We need to tell Pex somehow that once put InsureAmount as 0 for one test case and InsureAmount as 100 for another test case. This way we would be able to cover both conditions of the second branch. But how?

Using PexChoose. PexChoose is the static method in Microsoft.Pex.Framework which sets a value which is required in a symbolic execution of Pex Exploration. I won’t go in deep details of the Pex explore method, using a constraint solver Z3 engine, and how PexChoose will take part in a Constraint solving equation generated by PEX, you can get all these details at http://research.microsoft.com/en-us/projects/pex/.

But in layman's language, it tells PEX to use 0 for one test case, and any value greater than 0 for another test it executes to cover method branches 100%.

There is another advantage of PexChoose. If tomorrow business rules change, and there is a new condition as shown below:

if (ply.InsureAmount > 10)
    rtnAmount = ply.InsureAmount;
else
    rtnAmount = 0.1f;

We don’t need to change your Pex Factories and Pex Method, PEX exploration will figure out the new condition and will generate new Policy stubs based on PexChoose and make sure that it would cover both branches of code.

The new Pex Factory method, with Parameterize Mock, is shown below:

[PexFactoryMethod(typeof(Insurer))]
public static Insurer Create()
{
    DAL.Moles.SIDAL Mdal = new DAL.Moles.SIDAL();

    Mdal.GetInsursePersonPolicyPerson = ((p) =>
    {
        Policy ply = new Policy();
        ply.PolicyNumber = PexChoose.Value<int>("PolicyNumber");
        ply.InsureAmount = PexChoose.Value<float>("InsureAmount");
        PexObserve.Value<Policy>("PolicyReturn", ply);
        return ply;
    });

    Insurer insurer = new Insurer(Mdal);
    return insurer;
}

Here, you have learned a new concept Parameterize Mock and seen how it helps in Pex Exploration. Now I have covered Pex Factories, Pex Method, and Parameterize Mock, there remains one concept which I need to touch: partial stub.

Let us re-visit the second method GetInsureAmount.

public float GetInsureAmount(Person p)
{
    float rtnAmount  = 0.0f;
    if (this.IsPersonEligible(p))
    {
        Policy ply = dal.getInsursePersonPolicy(p);
        if (ply.InsureAmount > 0)
            rtnAmount = ply.InsureAmount;
        else
            rtnAmount = 0.1f;
    }
    else
        throw new NotEligibleException("Person " + p.Name + "not eligible for insurance");
    return rtnAmount;
}

Here you can see that IsPersonEligible is tested again with when running a test on GetInsureAmount. It’s OK and simple in our example, consider another scenario where IsPersonEligible is the complex method which in turn calls another third party DLL. Under the context of unit testing of GetInsureAmount, we don’t want to test IsPersonEligible. Remember, we are doing a Unit test and not integration test. IsPersonEligible testing would be done when doing unit testing on that method.

Here, there is a problem. IsPersonEligible is a public method of the same class Insurer, and I need some notion where I could fake a IsPersonEligible call in a test case, but not a GetInsureAmount call.

One solution would be to decouple the interaction, where I put IsPersonEligible in another class, and do dependency injection in the Insurer class, where I could fake a call to IsPersonEligible. But this is not always possible. There are cases when you can’t change class blueprints.

Here comes the concept of Partial Stubs where I would mock the Insurer class in such a way that I could fake a IsPersonEligible call, but let GetInsureAmount go to a real object.

public static Insurer Create01()
{
    DAL.Moles.SIDAL Mdal = new DAL.Moles.SIDAL();
    Mdal.GetInsursePersonPolicyPerson = ((p) => PexChoose.Value<Policy>("Policy"));
    SInsurer insurer = new SInsurer(Mdal) { CallBase = true };
    insurer.IsPersonEligiblePerson = ((p) => PexChoose.Value<bool>("IsEligible"));
    return insurer;
}

Bolded code stubs the Insurer class, where CallBase is set to true. I.e., whenever any call to any of the methods is called on the SInsurer stub, it will redirect its call to the actual implementation of the Insurer class.

Italicized line is where I have overridden the IsPersonEligible method, and set a parameterize value using PexChoose which would either return true or false basef on the symbolic constraint condition needed by Pex.

Now let us run the Pex exploration with a new partial stub and see the result:

Pex has found the bug in our code:

Policy ply = dal.getInsursePersonPolicy(p);
if (ply.InsureAmount > 0)

Following the above lines, I do not check if ply returned from dal.getInsursePersonPolicy is null or not and directly access the ply.InsureAmount property, and then a NullReferenceException is thrown!

Summary

Wow, I have reached the end of the article and here is the summary:

• Pex is a great candidate for white box testing which helps to find boundary conditions, crash tests, and unhandled exceptions in our code.
• In order to use Pex on a complex object and interaction, we need to use Pex Factories and Pex Method.
• In order to improve Pex Factories or Pex Method to take part in symbolic execution of Pex, we can use Parameterize Mock in them.
• If there is a hard dependency between the methods on the same class, and we need to unit test it, we can leverage the power of the Moles mocking framework from the same Microsoft Research team and could use Partial Stubs

Where do we go now?

• Go to http://research.microsoft.com/en-us/projects/pex/
• Try it out on the web: www.pexforfun.com
• Read documentation at http://research.microsoft.com/en-us/projects/pex/documentation.aspx

References

• http://research.microsoft.com/en-us/projects/pex/
• www.pexforfun.com for code samples
• Pex usage: http://research.microsoft.com/en-us/projects/pex/documentation.aspx
• MSDN Magazine article- Nikhil Sachdeva: http://msdn.microsoft.com/en-us/magazine/ee819140.aspx
• CodeProject: Thomas Weller article: http://www.codeproject.com/Articles/98378/Mocking-the-Unmockable-Using-Microsoft-Moles-with
• Dino Esposito article at http://www.drdobbs.com/testing/pex-microsoft-researchs-unit-test-genera/240009056