Click here to Skip to main content
Click here to Skip to main content

NPerf, A Performance Benchmark Framework for .NET

, 25 Jan 2004
Rate this:
Please Sign up or sign in to vote.
NPerf is a framework for benchmarking classes and methods, that tastes like NUnit.

Demo

Introduction

This article present NPerf a flexible performance benchmark framework. The framework provides custom attributes that the user uses the tag benchmark classes and methods. If you are familiar with NUnit [1], this is similar to the custom attributes they provide.

The framework uses reflection to gather the benchmark testers, the tested types, runs the tests and output the results. The user just have to write the benchmark methods.

At the end of the article, I illustrate NPerf with some metaphysic .NET question: interface vs. delegates, string concatenation race, fastest dictionary.

QuickStart: Benchmarking IDictionary

Let's start with a small introductory example: benchmarking the [] assignment for the different implementation of IDictionary. To do so, we would like to test the assignment on a growing number of assignment calls.

All the custom attributes are located in the NPerf.Framework namespace, NPerf.Framework.dll assembly.

PerfTester attribute: defining testers

First, you need to create a tester class that will contains method to do the benchmark. This tester method has to be decorated with the PerfTester attribute.

using NPerf.Framework;

[PerfTester(typeof(IDictionary),10)]
public class DictionaryTester
{
   ...
}

The PerfTesterAttribute constructor takes two argument:

  • the Type of the tested class, interface, or struct,
  • the number of test runs. The framework will use this value to call test methods multiple times (explained below).

PerfTest attribute: adding benchmark tests

The PerfTest attribute marks a specific method inside a class that has already been marked with the PerfTester attribute, as a performance test method. The method should take the tested type as parameter, IDictionary here,  and the return type should be void:

[PerfTester(typeof(IDictionary),10)] 
public class DictionaryTester 
{ 
    // explained below
    private int count;
    private Random rnd = new Random();

    [PerfTest] 
    public void ItemAssign(IDictionary dic) 
    {
         for(int i=0;i<this.count;++i) 
             dic[rnd.Next()]=null;
    }
}

PerfSetUp and PerfTearDown Attributes

Often, you will need to set up you tester and tested class before actually starting the benchmark test. In our example, we want to update the number of insertion depending the test repetition number. The PerfSetUp attribute can be used to tag a method that will be called before each test repetition. In our test case, we use this method to update the DictionaryTester.count member:

[PerfTester(typeof(IDictionary),10)] 
public class DictionaryTester 
{
    private int count;
    private Random rnd = new Random();

    [PerfSetUp] 
    public void SetUp(int index, IDictionary dic) 
    {
        this.count = index * 1000;
    }
}

The set-up method must return void and take two arguments:

  • index, current test repetition index. This value can be used to modify the number of elements tested, collection size, etc...
  • dic, the tested class instance

If you need to clean up resources after the tests are run, you can use the PerfTearDown attribute to tag a cleaning method:

[PerfTester(typeof(IDictionary),10)] 
public class DictionaryTester 
{     
    ...

    [PerfTearDown] 
    public void TearDown(IDictionary dic) 
    {
       ...
    }
}

PerfRunDescriptor attribute: giving some information to the framework

In our example, we test the IDictionary object with an increasing number of elements. It would be nice to store this number in the results, and not store just the test index: we would like to store 1000, 2000, .... and not 1, 2, ...

The PerfRunDescriptor attribute can be used to tag a method that returns a double from the test index. This double is typically used for charting the results, as x coordinate.

[PerfTester(typeof(IDictionary),10)] 
public class DictionaryTester 
{     
    [PerfRunDescriptor] 
    public double Count(int index) 
    {
       return index*1000;
    }
}

Full example source.

The full source of the example is as follows:

using System;
using System.Collections;
using NPerf.Framework;

[PerfTester(typeof(IDictionary),10)] 
public class DictionaryTester 
{     
    private int count = 0;
    private Random rnd = new Random();
    [PerfRunDescriptor] 
    public double Count(int index) 
    {
       return index*1000;
    }

    [PerfSetUp]
    public void SetUp(int index, IDictionary dic)
    {
        this.count = (int)Math.Floor(Count(index));
    }

    [PerfTest]
    public void ItemAssign(IDictionary dic)
    {
        for(int i =0;i<this.count;++i)
           dic[rnd.Next()]=null;
    }
}

Compiling and Running

Compile this class to an assembly and copy the NPerf binaries in the output folder: (NPerf.Cons.exe, NPerf.Core.dll, NPerf.Framework.Dll, NPerf.Report.Dll, ScPl.dll).

NPerf.Cons.exe is a console application that dynamically loads the tester assemblies (that you need to specify), the assemblies that contains the tested types (you need to specify), runs the test and output charts using ScPl [2] (ScPl is a chart library under GPL).

The call to NPerf.Cons.exe looks like this:

NPerf.Cons -ta=MyPerf.dll -tdfap=System -tdfap=mscorlib

where

  • ta defines an assembly that contains tester classes (DictionaryTester),
  • tdfap defines an assembly that contains tested type. Moreover, the assembly names are given as partial name and will be loaded by AssemblyLoadWithPartialName.

There are a number of other options that you can get by typing NPerf.Cons -h. Running the command line above will produce the following chart:

Sample screenshot

In the graph, you can see that some type failed the tests (PropertyDescriptorCollection). It is possible to specify to NPerf to avoid those types by passing them in the command line:

NPerf.Cons -ta=MyPerf.dll -tdfap=System -tdfap=mscorlib 
                       -it=PropertyDescriptorCollection

Saving to XML

You can also output the results to XML by adding the -x parameter. Internally, .NET XML serialization is used to render the results to XML.

A few remarks

  • You can add as many test method (PerfTest) as you want in the PerfTester classes,
  • You can define as many tester class as you want,
  • You can load tester/tested types from multiple assemblies

Overview of the Core

The NPerf.Core namespace contains the methods that do the job in the background. I do not plan to explain them in details but I'll discuss some problem I ran into while writing the framework.

Getting the machine properties

Getting the physical properties of the machine was a surprisingly difficult task. It took me a bunch of Google tries to get on the right pages. Anyway, here's the self-explaining code that get the machine properties:

ManagementObjectSearcher query = new 
  ManagementObjectSearcher("SELECT * From Win32_ComputerSystem");
foreach(ManagementObject obj in query.Get())
{
    long ram = long.Parse(obj["TotalPhysicalMemory"].ToString());    
    break;                
}
            
query = new ManagementObjectSearcher("SELECT * From Win32_Processor");
foreach(ManagementObject obj in query.Get())
{
    string cpu =(string)obj["Name"];                
    long cpuFrequency = 
      long.Parse(obj["CurrentClockSpeed"].ToString());
    break;                
}

TypeHelper, easier CustomAttribute support

A type helper static class was added to automate tedious tasks like check for custom attribute, get a custom attribute, etc... The TypeHelper class declaration is as follows:

public sealed class TypeHelper
{
    public static bool HasCustomAttribute(
        Type t,
        Type customAttributeType);
    public static bool HasCustomAttribute(
        MethodInfo t,
        Type customAttributeType)
    public static Object GetFirstCustomAttribute(
        Type t, 
        Type customAttributeType)
    public static Object GetFirstCustomAttribute(
        MethodInfo mi, 
        Type customAttributeType)
    public static MethodInfo GetAttributedMethod(
        Type t, 
        Type customAttributeType)
    public static AttributedMethodCollection GetAttributedMethods(
        Type t, 
        Type customAttributeType);
    public static void CheckSignature(
        MethodInfo mi, 
        Type returnType, 
        params Type[] argumentTypes);
    public static void CheckArguments(
        MethodInfo mi, 
        params Type[] argumentTypes);
}

Benchmark Bonus

In order to illustrate the framework, I have written a few benchmark testers for classic performance questions about .NET. All these benchmarks are provided in the System.Perf project.

IDictionary benchmark

Adding items


Sample screenshot

Sample screenshot

Sample screenshot

String concatenation benchmark

Sample screenshot

Interface vs. Delegate

Sample screenshot

History

  • 26-01-2004, v1.0, Initial release.

References

  1. NUnit, unit test framework for .NET
  2. ScPl, free chart library

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

Jonathan de Halleux
Engineer
United States United States
Jonathan de Halleux is Civil Engineer in Applied Mathematics. He finished his PhD in 2004 in the rainy country of Belgium. After 2 years in the Common Language Runtime (i.e. .net), he is now working at Microsoft Research on Pex (http://research.microsoft.com/pex).

Comments and Discussions

 
GeneralFinding tested types Pinmemberling_aus18-Oct-04 7:29 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web04 | 2.8.140821.2 | Last Updated 26 Jan 2004
Article Copyright 2004 by Jonathan de Halleux
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid