Click here to Skip to main content
15,861,168 members
Articles / Programming Languages / C#
Article

NPerf, A Performance Benchmark Framework for .NET

Rate me:
Please Sign up or sign in to vote.
4.92/5 (44 votes)
25 Jan 20044 min read 229.8K   705   139   49
NPerf is a framework for benchmarking classes and methods, that tastes like NUnit.

Demo

Introduction

This article present NPerf a flexible performance benchmark framework. The framework provides custom attributes that the user uses the tag benchmark classes and methods. If you are familiar with NUnit [1], this is similar to the custom attributes they provide.

The framework uses reflection to gather the benchmark testers, the tested types, runs the tests and output the results. The user just have to write the benchmark methods.

At the end of the article, I illustrate NPerf with some metaphysic .NET question: interface vs. delegates, string concatenation race, fastest dictionary.

QuickStart: Benchmarking IDictionary

Let's start with a small introductory example: benchmarking the [] assignment for the different implementation of IDictionary. To do so, we would like to test the assignment on a growing number of assignment calls.

All the custom attributes are located in the NPerf.Framework namespace, NPerf.Framework.dll assembly.

PerfTester attribute: defining testers

First, you need to create a tester class that will contains method to do the benchmark. This tester method has to be decorated with the PerfTester attribute.

C#
using NPerf.Framework;

[PerfTester(typeof(IDictionary),10)]
public class DictionaryTester
{
   ...
}

The PerfTesterAttribute constructor takes two argument:

  • the Type of the tested class, interface, or struct,
  • the number of test runs. The framework will use this value to call test methods multiple times (explained below).

PerfTest attribute: adding benchmark tests

The PerfTest attribute marks a specific method inside a class that has already been marked with the PerfTester attribute, as a performance test method. The method should take the tested type as parameter, IDictionary here,  and the return type should be void:

C#
[PerfTester(typeof(IDictionary),10)] 
public class DictionaryTester 
{ 
    // explained below
    private int count;
    private Random rnd = new Random();

    [PerfTest] 
    public void ItemAssign(IDictionary dic) 
    {
         for(int i=0;i<this.count;++i) 
             dic[rnd.Next()]=null;
    }
}

PerfSetUp and PerfTearDown Attributes

Often, you will need to set up you tester and tested class before actually starting the benchmark test. In our example, we want to update the number of insertion depending the test repetition number. The PerfSetUp attribute can be used to tag a method that will be called before each test repetition. In our test case, we use this method to update the DictionaryTester.count member:

C#
[PerfTester(typeof(IDictionary),10)] 
public class DictionaryTester 
{
    private int count;
    private Random rnd = new Random();

    [PerfSetUp] 
    public void SetUp(int index, IDictionary dic) 
    {
        this.count = index * 1000;
    }
}

The set-up method must return void and take two arguments:

  • index, current test repetition index. This value can be used to modify the number of elements tested, collection size, etc...
  • dic, the tested class instance

If you need to clean up resources after the tests are run, you can use the PerfTearDown attribute to tag a cleaning method:

C#
[PerfTester(typeof(IDictionary),10)] 
public class DictionaryTester 
{     
    ...

    [PerfTearDown] 
    public void TearDown(IDictionary dic) 
    {
       ...
    }
}

PerfRunDescriptor attribute: giving some information to the framework

In our example, we test the IDictionary object with an increasing number of elements. It would be nice to store this number in the results, and not store just the test index: we would like to store 1000, 2000, .... and not 1, 2, ...

The PerfRunDescriptor attribute can be used to tag a method that returns a double from the test index. This double is typically used for charting the results, as x coordinate.

C#
[PerfTester(typeof(IDictionary),10)] 
public class DictionaryTester 
{     
    [PerfRunDescriptor] 
    public double Count(int index) 
    {
       return index*1000;
    }
}

Full example source.

The full source of the example is as follows:

C#
using System;
using System.Collections;
using NPerf.Framework;

[PerfTester(typeof(IDictionary),10)] 
public class DictionaryTester 
{     
    private int count = 0;
    private Random rnd = new Random();
    [PerfRunDescriptor] 
    public double Count(int index) 
    {
       return index*1000;
    }

    [PerfSetUp]
    public void SetUp(int index, IDictionary dic)
    {
        this.count = (int)Math.Floor(Count(index));
    }

    [PerfTest]
    public void ItemAssign(IDictionary dic)
    {
        for(int i =0;i<this.count;++i)
           dic[rnd.Next()]=null;
    }
}

Compiling and Running

Compile this class to an assembly and copy the NPerf binaries in the output folder: (NPerf.Cons.exe, NPerf.Core.dll, NPerf.Framework.Dll, NPerf.Report.Dll, ScPl.dll).

NPerf.Cons.exe is a console application that dynamically loads the tester assemblies (that you need to specify), the assemblies that contains the tested types (you need to specify), runs the test and output charts using ScPl [2] (ScPl is a chart library under GPL).

The call to NPerf.Cons.exe looks like this:

NPerf.Cons -ta=MyPerf.dll -tdfap=System -tdfap=mscorlib

where

  • ta defines an assembly that contains tester classes (DictionaryTester),
  • tdfap defines an assembly that contains tested type. Moreover, the assembly names are given as partial name and will be loaded by AssemblyLoadWithPartialName.

There are a number of other options that you can get by typing NPerf.Cons -h. Running the command line above will produce the following chart:

Sample screenshot

In the graph, you can see that some type failed the tests (PropertyDescriptorCollection). It is possible to specify to NPerf to avoid those types by passing them in the command line:

NPerf.Cons -ta=MyPerf.dll -tdfap=System -tdfap=mscorlib 
                       -it=PropertyDescriptorCollection

Saving to XML

You can also output the results to XML by adding the -x parameter. Internally, .NET XML serialization is used to render the results to XML.

A few remarks

  • You can add as many test method (PerfTest) as you want in the PerfTester classes,
  • You can define as many tester class as you want,
  • You can load tester/tested types from multiple assemblies

Overview of the Core

The NPerf.Core namespace contains the methods that do the job in the background. I do not plan to explain them in details but I'll discuss some problem I ran into while writing the framework.

Getting the machine properties

Getting the physical properties of the machine was a surprisingly difficult task. It took me a bunch of Google tries to get on the right pages. Anyway, here's the self-explaining code that get the machine properties:

C#
ManagementObjectSearcher query = new 
  ManagementObjectSearcher("SELECT * From Win32_ComputerSystem");
foreach(ManagementObject obj in query.Get())
{
    long ram = long.Parse(obj["TotalPhysicalMemory"].ToString());    
    break;                
}
            
query = new ManagementObjectSearcher("SELECT * From Win32_Processor");
foreach(ManagementObject obj in query.Get())
{
    string cpu =(string)obj["Name"];                
    long cpuFrequency = 
      long.Parse(obj["CurrentClockSpeed"].ToString());
    break;                
}

TypeHelper, easier CustomAttribute support

A type helper static class was added to automate tedious tasks like check for custom attribute, get a custom attribute, etc... The TypeHelper class declaration is as follows:

C#
public sealed class TypeHelper
{
    public static bool HasCustomAttribute(
        Type t,
        Type customAttributeType);
    public static bool HasCustomAttribute(
        MethodInfo t,
        Type customAttributeType)
    public static Object GetFirstCustomAttribute(
        Type t, 
        Type customAttributeType)
    public static Object GetFirstCustomAttribute(
        MethodInfo mi, 
        Type customAttributeType)
    public static MethodInfo GetAttributedMethod(
        Type t, 
        Type customAttributeType)
    public static AttributedMethodCollection GetAttributedMethods(
        Type t, 
        Type customAttributeType);
    public static void CheckSignature(
        MethodInfo mi, 
        Type returnType, 
        params Type[] argumentTypes);
    public static void CheckArguments(
        MethodInfo mi, 
        params Type[] argumentTypes);
}

Benchmark Bonus

In order to illustrate the framework, I have written a few benchmark testers for classic performance questions about .NET. All these benchmarks are provided in the System.Perf project.

IDictionary benchmark

Adding items


Sample screenshot

Sample screenshot

Sample screenshot

String concatenation benchmark

Sample screenshot

Interface vs. Delegate

Sample screenshot

History

  • 26-01-2004, v1.0, Initial release.

References

  1. NUnit, unit test framework for .NET
  2. ScPl, free chart library

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here


Written By
Engineer
United States United States
Jonathan de Halleux is Civil Engineer in Applied Mathematics. He finished his PhD in 2004 in the rainy country of Belgium. After 2 years in the Common Language Runtime (i.e. .net), he is now working at Microsoft Research on Pex (http://research.microsoft.com/pex).

Comments and Discussions

 
GeneralBad Link on Page Pin
Mjlarkin7-Feb-07 10:12
Mjlarkin7-Feb-07 10:12 
GeneralRe: Bad Link on Page Pin
Andrew Phillips17-Sep-08 2:48
Andrew Phillips17-Sep-08 2:48 
NewsLicence Pin
Christian Schweizer23-May-06 23:14
Christian Schweizer23-May-06 23:14 
GeneralCompact Framework Pin
Vasudevan Deepak Kumar14-Mar-06 4:03
Vasudevan Deepak Kumar14-Mar-06 4:03 
GeneralFramework .Net 2.0 Pin
lmclr10-Feb-06 0:53
lmclr10-Feb-06 0:53 
GeneralRe: Framework .Net 2.0 Pin
valentine29227-Aug-06 8:23
valentine29227-Aug-06 8:23 
GeneralRe: Framework .Net 2.0 Pin
Wcohen27-Dec-06 5:30
Wcohen27-Dec-06 5:30 
GeneralExcellent Pin
Desmond McCarter23-Jan-06 2:40
Desmond McCarter23-Jan-06 2:40 
Generaldownload NPerf - site under maintenance Pin
devvvy7-Apr-05 16:53
devvvy7-Apr-05 16:53 
GeneralRe: download NPerf - site under maintenance Pin
guillaume_se10-Nov-05 2:28
guillaume_se10-Nov-05 2:28 
Generalhelp getting started Pin
Member 17413128-Mar-05 11:27
Member 17413128-Mar-05 11:27 
Hi:
looks like it's not able to find the ClassLibrary1.dll, the class to be tested. Can anyone tell me why?

NPerf.Cons -ta=ConsoleApplication1.dll -tdfap=ClassLibrary1.dll

Load tester assembly: ConsoleApplication1.dll
Load tested assembly: ClassLibrary1.dll
System.NullReferenceException: Object reference not set to an instance of an obj
ect.
at NPerf.Core.PerfTester.LoadTestedTypes(Assembly a) in c:\dev tools\nperf_src\releases\unpackaged\nperf-1.1-release\nperf\src\nperf.core\perftester.cs:line 204
at NPerf.Cons.Class1.Main(String[] args) in C:\Dev Tools\nperf_src\Releases\Unpackaged\nperf-1.1-Release\nperf\src\NPerf.Cons\Class1.cs:line 80


Thanks

vk
GeneralFinding tested types Pin
ling_aus18-Oct-04 7:29
ling_aus18-Oct-04 7:29 
GeneralMeasuring the results Pin
steven_pack11-Oct-04 7:14
steven_pack11-Oct-04 7:14 
GeneralExamples Pin
SteveSlo30-Aug-04 4:27
SteveSlo30-Aug-04 4:27 
GeneralPretty darn good Pin
soup19-Jul-04 5:19
soup19-Jul-04 5:19 
QuestionWhy are [PerfTest] methods run twice? Pin
Rjae Easton17-May-04 11:49
Rjae Easton17-May-04 11:49 
AnswerRe: Why are [PerfTest] methods run twice? Pin
Jonathan de Halleux17-May-04 12:07
Jonathan de Halleux17-May-04 12:07 
QuestionSource on dotnetwiki missing code? Pin
panmanphil10-Mar-04 5:17
panmanphil10-Mar-04 5:17 
AnswerRe: Source on dotnetwiki missing code? Pin
Jonathan de Halleux15-Mar-04 20:32
Jonathan de Halleux15-Mar-04 20:32 
GeneralRe: Source on dotnetwiki missing code? Pin
Jonathan de Halleux19-Apr-04 3:26
Jonathan de Halleux19-Apr-04 3:26 
GeneralType comparison problem Pin
Darin Creason5-Feb-04 16:06
sussDarin Creason5-Feb-04 16:06 
GeneralRe: Type comparison problem Pin
Jonathan de Halleux5-Feb-04 21:29
Jonathan de Halleux5-Feb-04 21:29 
GeneralComparisons Pin
Carel Lotz3-Feb-04 20:09
Carel Lotz3-Feb-04 20:09 
GeneralRe: Comparisons Pin
Jonathan de Halleux5-Feb-04 3:24
Jonathan de Halleux5-Feb-04 3:24 
GeneralBlank Graphics Pin
rido28-Jan-04 6:11
rido28-Jan-04 6:11 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.