Click here to Skip to main content
Click here to Skip to main content

Code Contract Performance Analysis

, 15 Apr 2011 CPOL
Rate this:
Please Sign up or sign in to vote.
A look at the runtime cost of using code contracts

Introduction

I found the article about code contracts in this month's MSDN very exciting. I was unaware of this feature of .NET 4.0 before reading the article, and as the manager of a team of developers who maintain a very large and complex application, I am always interested in techniques that improve code quality and correctness. I won't rehash the interesting details about code contracts in this article. Go read the MSDN article first or this CodeProject introduction. The basics are all there. Then go here to the Microsoft DevLabs page with the download you'll need to run the code below.

My first thought when reading the MSDN article was that the benefits must come at some price, and my initial concern was an impact on performance. Contracts are enforced at run-time by inserting custom code at compile time, and whenever some other process is adding code to mine, I worry about hidden performance costs.

I wrote the small app below to get some metrics and assess how expensive contracts are compared to the other techniques that can be used to validate pre and post execution conditions.

Using the Code

The author of the MSDN article used a trivial calculator function to highlight the benefits of contracts. I'll use the same basic function here:

private static Int32 Add(Int32 x, Int32 y) {
   if (x == y)
      return x * 2;

   return x + y;
}

The extra if statement in there is just to reinforce the difficulty of adding post-condition checking everywhere in your code where you have premature exits. I'm going to leave it here for the analysis.

"If-Then-Throw"

One way to check pre and post conditions is to use explicit if statements to validate your input parameters and output results. All of the pre and post conditions we're adding here are perfectly arbitrary, but will naturally be consistent in all the examples.

private static Int32 IfCheckedAdd(Int32 x, Int32 y) {
   if (x < 0) // Arbitrary pre-condition
      throw new InvalidOperationException("X must be greater than 0");

   if (x == y) {
      if (x * 2 < 0) // Arbitrary post-condition
         throw new InvalidOperationException("Result must be positive");

      return x * 2;
   }

   if (x + y < 0) // Same arbitrary post-condition
      throw new InvalidOperationException("Result must be positive");
            
   return x + y;
}

Debug.Assert

Another way is using Debug.Assert():

private static Int32 AssertCheckedAdd(Int32 x, Int32 y) {
   System.Diagnostics.Debug.Assert(x >= 0);

   if (x == y) {
      System.Diagnostics.Debug.Assert(x * 2 >= 0);

      return x * 2;
   }

   System.Diagnostics.Debug.Assert(x + y >= 0);
   return x + y;
}

Contracts

And the last method to examine is the interesting new one, code contracts:

private static Int32 ContractCheckedAdd(Int32 x, Int32 y) {
   Contract.Requires(x >= 0, "X must be greater than 0");
   Contract.Ensures(Contract.Result<Int32>() >= 0, "Result must be positive");

   if (x == y)
      return x * 2;

   return x + y;
}

One of the benefits to contracts being that you don't have to worry about where you exit. All of your post checking conditions are centralized at the top of the method.

Running the Different Methods

I timed how long it took to execute each of the methods above 100,000,000 times.

private delegate Int32 testMethod(Int32 x, Int32 y);

static void Main(string[] args) {
   const Int32 ITERATIONS = 10000;

   foreach (testMethod tm in new testMethod[] { new testMethod(Add), 
                                    new testMethod(IfCheckedAdd), 
                                    new testMethod(AssertCheckedAdd),
                                    new testMethod(ContractCheckedAdd) }) {
   DateTime start = DateTime.Now;

   for (Int32 i = 0; i < ITERATIONS; i++)
      for (Int32 j = 0; j < ITERATIONS; j++)
         tm(i, j);

      System.Console.WriteLine(tm.Method.Name + " " + 
	(DateTime.Now - start).TotalMilliseconds.ToString());
   }
}

The Setup

I ran the application 5 times to get average times and a feel for the variability of the results. The tests were executed on Windows 7 (32-bit), on a dual core (Intel Core 2 E8400 3.0GHz) CPU with 4Gb of RAM.

I used version 1.4.40314.1 of the code contracts SDK with pre and post contract checking enabled.

I'm not interested in the performance of the different methods of execution when the pre or post conditions are not met, just the overhead of the different validation frameworks.

The Results

Debug

Add If-then-throw Assert Contract
Run 1 (ms) 1934.4 2199.6 2230.8 3213.6
Run 2 1950.0 2184.0 2215.2 3244.8
Run 3 1950.0 2199.6 2246.4 3260.4
Run 4 1950.0 2246.4 2293.2 3369.6
Run 5 1934.4 2184.0 2246.4 3244.8
Average 1940.6 2202.7 2246.4 3266.6
Std Dev 8.54 25.63 29.18 60.01
13.5% 1.98% 45.42%
15.76%
68.33%
Table 1. Results from running code in Visual Studio 2008 built for debug.
  • The "naked" calculator method took on average 1940.6ms to execute.
  • The method with the if-then-throw pre and post condition checking took 2202.7ms to execute (13.5% slower than the unchecked benchmark Add() method).
  • The method that used Debug.Assert() pre and post condition checking took 2246.4ms, (1.98% slower than if-then-throw and 15.76% slower than the unchecked benchmark Add() method).
  • The method that used contracts took on average 3266.6ms and was 45.42% slower than Debug.Assert() and 68.33% slower than the "naked" calculator method.

When the switch to enable contract code insertion is set to false, the ContractCheckedAdd() method took exactly the same amount of time as the unchecked Add() method, as no code was inserted at compile time. You can verify that by looking at the IL of ContractCheckedAdd() in an assembly built with contracts enabled and contracts disabled. ContractCheckedAdd() looks the same as Add() when contracts are not enabled.

Release

Add If-then-throw Assert Contract
Run 1 (ms) 467.8 545.8 467.8 686.2
Run 2 467.9 545.9 467.9 701.9
Run 3 467.9 530.3 467.9 701.9
Run 4 467.9 545.9 467.9 701.9
Run 5 467.9 545.9 467.9 686.3
Average 467.9 542.8 467.9 695.7
Std Dev 0 7.0 0 8.6
16.0%
0%
48.7%
Table 2. Results from running code outside of Visual Studio 2008 built for release.

Conclusion

I hope to avoid maintenance headaches on future projects and would be happy to sacrifice some performance for some assurances about correctness, so I expect to use code contracts in a lot of my future development. I thought it would be good to know some of the costs to weigh against the benefits.

Share and enjoy.

History

  • April 13 2011 - Initial revision
  • April 15 2011 - Added results from running build for Release

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Sean Michael Murphy
Product Manager
Canada Canada
I'm a graduate of the University of Toronto with a degree in zoology. I'm currently a software development manager with a large Canadian financial institution, and a passionate squash player.
 
I am a proud daddy to Alex and Sarah.

Comments and Discussions

 
GeneralMy vote of 3 PinmemberGlobX18-Apr-11 19:52 
GeneralMy Vote of 5 PinmemberScruffyDuck16-Apr-11 22:39 
I have only played around with Code Contracts. However they do appeal to me as a way of ensuring that conditions can be checked for in an orderly manner. You article has prompted me to take a look at Code Contracts again for my work. Thank you.
Jon
ScruffyDuck Software
Microsoft MVP

GeneralMy vote of 5 PinmemberR. Hoffmann15-Apr-11 9:50 
GeneralRequires&lt;T&gt; Pinmemberkornman0015-Apr-11 7:47 
GeneralRe: Requires<T> PinmemberSean Michael Murphy17-Apr-11 18:24 
GeneralOther posters had great responses PinmemberJeremy Hutchinson15-Apr-11 3:31 
GeneralRe: Other posters had great responses PinmemberSean Michael Murphy15-Apr-11 6:58 
GeneralRe: Other posters had great responses PinmemberSean Michael Murphy15-Apr-11 15:18 
GeneralRe: Other posters had great responses PinmemberJeremy Hutchinson18-Apr-11 2:29 
GeneralResults fail PinmemberXetrill15-Apr-11 0:48 
GeneralRe: Results fail PinmemberSean Michael Murphy15-Apr-11 3:15 
QuestionWould it improve in release mode? PinmemberWerner van Deventer14-Apr-11 19:20 
AnswerRe: Would it improve in release mode? PinmemberSean Michael Murphy15-Apr-11 3:17 
GeneralDifferent Results PinmemberMW_Justin14-Apr-11 18:33 
GeneralRe: Different Results PinmemberSean Michael Murphy15-Apr-11 3:23 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web04 | 2.8.141030.1 | Last Updated 15 Apr 2011
Article Copyright 2011 by Sean Michael Murphy
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid