Introduction
This article present NPerf a flexible performance benchmark framework. The framework provides custom attributes that the user uses the tag benchmark classes and methods. If you are familiar with NUnit [1], this is similar to the custom attributes they provide.
The framework uses reflection to gather the benchmark testers, the tested types, runs the tests and output the results. The user just have to write the benchmark methods.
At the end of the article, I illustrate NPerf with some metaphysic .NET question: interface vs. delegates, string concatenation race, fastest dictionary.
QuickStart: Benchmarking IDictionary
Let's start with a small introductory example: benchmarking the [] assignment for the different implementation of IDictionary. To do so, we would like to test the assignment on a growing number of assignment calls.
All the custom attributes are located in the NPerf.Framework namespace, NPerf.Framework.dll assembly.
PerfTester attribute: defining testers
First, you need to create a tester class that will contains method to do the benchmark. This tester method has to be decorated with the PerfTester attribute.
using NPerf.Framework;
[PerfTester(typeof(IDictionary),10)]
public class DictionaryTester
{
...
}
The PerfTesterAttribute constructor takes two argument:
- the
Type of the tested class, interface, or struct,
- the number of test runs. The framework will use this value to call test methods multiple times (explained below).
PerfTest attribute: adding benchmark tests
The PerfTest attribute marks a specific method inside a class that has already been marked with the PerfTester attribute, as a performance test method. The method should take the tested type as parameter, IDictionary here, and the return type should be void:
[PerfTester(typeof(IDictionary),10)]
public class DictionaryTester
{
private int count;
private Random rnd = new Random();
[PerfTest]
public void ItemAssign(IDictionary dic)
{
for(int i=0;i<this.count;++i)
dic[rnd.Next()]=null;
}
}
PerfSetUp and PerfTearDown Attributes
Often, you will need to set up you tester and tested class before actually starting the benchmark test. In our example, we want to update the number of insertion depending the test repetition number. The PerfSetUp attribute can be used to tag a method that will be called before each test repetition. In our test case, we use this method to update the DictionaryTester.count member:
[PerfTester(typeof(IDictionary),10)]
public class DictionaryTester
{
private int count;
private Random rnd = new Random();
[PerfSetUp]
public void SetUp(int index, IDictionary dic)
{
this.count = index * 1000;
}
}
The set-up method must return void and take two arguments:
index, current test repetition index. This value can be used to modify the number of elements tested, collection size, etc...
dic, the tested class instance
If you need to clean up resources after the tests are run, you can use the PerfTearDown attribute to tag a cleaning method:
[PerfTester(typeof(IDictionary),10)]
public class DictionaryTester
{
...
[PerfTearDown]
public void TearDown(IDictionary dic)
{
...
}
}
PerfRunDescriptor attribute: giving some information to the framework
In our example, we test the IDictionary object with an increasing number of elements. It would be nice to store this number in the results, and not store just the test index: we would like to store 1000, 2000, .... and not 1, 2, ...
The PerfRunDescriptor attribute can be used to tag a method that returns a double from the test index. This double is typically used for charting the results, as x coordinate.
[PerfTester(typeof(IDictionary),10)]
public class DictionaryTester
{
[PerfRunDescriptor]
public double Count(int index)
{
return index*1000;
}
}
Full example source.
The full source of the example is as follows:
using System;
using System.Collections;
using NPerf.Framework;
[PerfTester(typeof(IDictionary),10)]
public class DictionaryTester
{
private int count = 0;
private Random rnd = new Random();
[PerfRunDescriptor]
public double Count(int index)
{
return index*1000;
}
[PerfSetUp]
public void SetUp(int index, IDictionary dic)
{
this.count = (int)Math.Floor(Count(index));
}
[PerfTest]
public void ItemAssign(IDictionary dic)
{
for(int i =0;i<this.count;++i)
dic[rnd.Next()]=null;
}
}
Compiling and Running
Compile this class to an assembly and copy the NPerf binaries in the output folder: (NPerf.Cons.exe, NPerf.Core.dll, NPerf.Framework.Dll, NPerf.Report.Dll, ScPl.dll).
NPerf.Cons.exe is a console application that dynamically loads the tester assemblies (that you need to specify), the assemblies that contains the tested types (you need to specify), runs the test and output charts using ScPl [2] (ScPl is a chart library under GPL).
The call to NPerf.Cons.exe looks like this:
NPerf.Cons -ta=MyPerf.dll -tdfap=System -tdfap=mscorlib
where
ta defines an assembly that contains tester classes (DictionaryTester),
tdfap defines an assembly that contains tested type. Moreover, the assembly names are given as partial name and will be loaded by AssemblyLoadWithPartialName.
There are a number of other options that you can get by typing NPerf.Cons -h. Running the command line above will produce the following chart:
In the graph, you can see that some type failed the tests (PropertyDescriptorCollection). It is possible to specify to NPerf to avoid those types by passing them in the command line:
NPerf.Cons -ta=MyPerf.dll -tdfap=System -tdfap=mscorlib
-it=PropertyDescriptorCollection
Saving to XML
You can also output the results to XML by adding the -x parameter. Internally, .NET XML serialization is used to render the results to XML.
A few remarks
- You can add as many test method (
PerfTest) as you want in the PerfTester classes,
- You can define as many tester class as you want,
- You can load tester/tested types from multiple assemblies
Overview of the Core
The NPerf.Core namespace contains the methods that do the job in the background. I do not plan to explain them in details but I'll discuss some problem I ran into while writing the framework.
Getting the machine properties
Getting the physical properties of the machine was a surprisingly difficult task. It took me a bunch of Google tries to get on the right pages. Anyway, here's the self-explaining code that get the machine properties:
ManagementObjectSearcher query = new
ManagementObjectSearcher("SELECT * From Win32_ComputerSystem");
foreach(ManagementObject obj in query.Get())
{
long ram = long.Parse(obj["TotalPhysicalMemory"].ToString());
break;
}
query = new ManagementObjectSearcher("SELECT * From Win32_Processor");
foreach(ManagementObject obj in query.Get())
{
string cpu =(string)obj["Name"];
long cpuFrequency =
long.Parse(obj["CurrentClockSpeed"].ToString());
break;
}
TypeHelper, easier CustomAttribute support
A type helper static class was added to automate tedious tasks like check for custom attribute, get a custom attribute, etc... The TypeHelper class declaration is as follows:
public sealed class TypeHelper
{
public static bool HasCustomAttribute(
Type t,
Type customAttributeType);
public static bool HasCustomAttribute(
MethodInfo t,
Type customAttributeType)
public static Object GetFirstCustomAttribute(
Type t,
Type customAttributeType)
public static Object GetFirstCustomAttribute(
MethodInfo mi,
Type customAttributeType)
public static MethodInfo GetAttributedMethod(
Type t,
Type customAttributeType)
public static AttributedMethodCollection GetAttributedMethods(
Type t,
Type customAttributeType);
public static void CheckSignature(
MethodInfo mi,
Type returnType,
params Type[] argumentTypes);
public static void CheckArguments(
MethodInfo mi,
params Type[] argumentTypes);
}
Benchmark Bonus
In order to illustrate the framework, I have written a few benchmark testers for classic performance questions about .NET. All these benchmarks are provided in the System.Perf project.
IDictionary benchmark
String concatenation benchmark
Interface vs. Delegate
History
- 26-01-2004, v1.0, Initial release.
References
- NUnit, unit test framework for .NET
- ScPl, free chart library
|
|
 |
 | Bad Link on Page
Mjlarkin | 11:12 7 Feb '07 |
|
|
Your link to scpl is bad:
http://netcontrols.org/nplot/wiki/index.php?n=Main.Scpl
is the correct URL
|
|
|
|
|
|
|
Sept 2008: SCPL link is still wrong.
Andrew Phillips
http://www.hexedit.com
andrew @ hexedit.com
|
|
|
|
|
 | Licence
Christian Schweizer | 0:14 24 May '06 |
|
|
Hi there!
I just had a look on your project and it looks very interesting. Im currently thinking about integration NPerf in our own project for benchmarks as implementing an own benchmark framework takes too much time for us...
My question to you is about licensing.
Which license has your project? And is there still developing going on on NPerf?
Thanks a lot in advance for your answer! By the way, have you considered to publish your project on sourceforge.net e.g.? It seems as there is nothing like NPerf on sourceforge...
Best regards
Christian Schweizer
|
|
|
|
|
| Compact Framework
Vasudevan Deepak Kumar | 5:03 14 Mar '06 |
|
|
|
| Framework .Net 2.0
lmclr | 1:53 10 Feb '06 |
|
|
Hello,
Vey good tool, but does NPerf work with .Net 2.0 ? 
Thanks
|
|
|
|
|
|
|
I am also wondering this. I have created a project I would like to test using Visual Studio 2005 (.Net 2.0). It is not working, and I suspect that it is because it is not compatabile with 2.0. Here is the console output I get when running from console (myPerf is a dll built with 2.0)
Nperf.Cons -ta=MyPerf.dll -tdfap=System -tdfap=mscorlib
Load tester assembly: MyPerf.dll
System.BadImageFormatException: The format of the file 'MyPerf.dll' is invalid.
File name: "MyPerf.dll"
at System.Reflection.Assembly.nLoad(AssemblyName fileName, String codeBase, B
oolean isStringized, Evidence assemblySecurity, Boolean throwOnFileNotFound, Ass
embly locationHint, StackCrawlMark& stackMark)
at System.Reflection.Assembly.InternalLoad(AssemblyName assemblyRef, Boolean
stringized, Evidence assemblySecurity, StackCrawlMark& stackMark)
at System.Reflection.Assembly.LoadFrom(String assemblyFile, Evidence security
Evidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm)
at NPerf.Cons.Class1.Main(String[] args)
|
|
|
|
|
|
|
It compiled fine under .NET 2.0 and I had no troubles using it. I have to mention that I don't have .NET 1.0 nor do I have .NET 1.1 installed on my machine.
|
|
|
|
|
| Excellent
Desmond McCarter | 3:40 23 Jan '06 |
|
|
This is really good work: something that should have been thought of years ago.
Just wondering though - does NPerf have any in-built multi-threading capabilities? i.e. is there a way through the attributes I can muti-thread method calls rather than run them sequencially (without any effort from my end)?
Again, very good job.
Des.
Rgds,
Des.
|
|
|
|
|
| download NPerf - site under maintenance
norm | 17:53 7 Apr '05 |
|
|
hey, broken link!
Norman Fung
|
|
|
|
|
|
|
site alway down, 8 month later...
obviously, there's no pilot on the plane anymore.
but you can still use the source files mnetionned on the top of this article.
It compile fine, even under the 2.0 version of the framework
|
|
|
|
|
| help getting started
vk2101 | 12:27 8 Mar '05 |
|
|
Hi:
looks like it's not able to find the ClassLibrary1.dll, the class to be tested. Can anyone tell me why?
NPerf.Cons -ta=ConsoleApplication1.dll -tdfap=ClassLibrary1.dll
Load tester assembly: ConsoleApplication1.dll
Load tested assembly: ClassLibrary1.dll
System.NullReferenceException: Object reference not set to an instance of an obj
ect.
at NPerf.Core.PerfTester.LoadTestedTypes(Assembly a) in c:\dev tools\nperf_src\releases\unpackaged\nperf-1.1-release\nperf\src\nperf.core\perftester.cs:line 204
at NPerf.Cons.Class1.Main(String[] args) in C:\Dev Tools\nperf_src\Releases\Unpackaged\nperf-1.1-Release\nperf\src\NPerf.Cons\Class1.cs:line 80
Thanks
vk
|
|
|
|
|
| Finding tested types
ling_aus | 8:29 18 Oct '04 |
|
|
I am very new to NPerf, so please bare with me if my question doesn't make any sense.
Currently to run a test assembly using nperf.cons, I need to specify the name of the test assembly, plus the names of the assemblies that contain the types my test assembly tests.
Since all the assemblies are referenced from the test assembly, can NPerf load all the referenced assemblies at start time and save user the work to enter the assembly names?
Ling
|
|
|
|
|
| Measuring the results
steven_pack | 8:14 11 Oct '04 |
|
|
The y axis shows log(s). Is this log10?
That is, if a test comes out as -2, that is 10^-2=.01 of a second. I.e. 10ms?
|
|
|
|
|
| Examples
SteveSlo | 5:27 30 Aug '04 |
|
|
Could someone point me in the direction of some additional examples (I learn best by example)? I had no problem with the demo, however, when I try to create my own tests, I don't get very far. In particular, I'm trying to create a test to evaluate database access performance.
|
|
|
|
|
| Pretty darn good
soup | 6:19 19 Jul '04 |
|
|
Hi there,
This is an excellent idea, something I think many software projects could benefit from. 5 stars!
Small article correction though: the source code download link was about 750k, not 49k!
Simon
|
|
|
|
|
| Why are [PerfTest] methods run twice?
Rjae Easton | 12:49 17 May '04 |
|
|
Is this a bug? PerfTester.RunTests calls RunTest twice where (testIndex == 0). Given a [PerfTest] that inserts db records, this causes 2 writes where 1 is desired.
I have not figured a way to circumvent this. Do you have advise or comment on this situation?
Thank you.
P.S. I am a big fan of NPerf!
Rjae Easton
p: +1.508.898.4691
c: +1.508.369.7339
e: reaston@applanet.com
aim: M1ngSheng
msn: rjae_easton@hotmail.com
Y!: m1ngsheng
|
|
|
|
|
|
|
Test is run twice at the beginning to make sure JIT does not corrupts all the results. I have not found better solution for this.
Rjae Easton wrote:
P.S. I am a big fan of NPerf!
Thanks 
Jonathan de Halleux - My Blog - www.dotnetwiki.org -
MbUnit - QuickGraph - NCollection
|
|
|
|
|
| Source on dotnetwiki missing code?
Philip Nelson | 6:17 10 Mar '04 |
|
|
I just downloaded the binary from dotnetwiki and it's missing the NPerf.Framework.dll. No problem, I downloaded the source, and it won't compile. It seems a couple of what appears to be delegates are not defined in the source anywhere.
PerfTestEventHandler
PerfTestRun
Is there more updated source somewhere?
Thanks
Philip Nelson
|
|
|
|
|
|
|
|
|
|
|
| Type comparison problem
Darin Creason | 17:06 5 Feb '04 |
|
|
Hello everyone. Thanks for taking the time to write this and sharing it, Jonathan. I have been working with your library today and found what I think is a problem in the type comparison routine.
Taken from PerfTester.cs:
public void LoadTestedTypes(Assembly a)
{
if (a==null)
throw new ArgumentNullException("a");
if (this.testedType.IsInterface)
{
foreach(Type t in a.GetExportedTypes())
{
if (
t.GetInterface(this.testedType.ToString())!=null
&& !t.IsAbstract
)
this.testedTypes.Add(t);
}
}
else
{
foreach(Type t in a.GetExportedTypes())
{
if ( t.IsInstanceOfType(TestedType)
&& !t.IsAbstract
)
this.testedTypes.Add(t);
}
}
}
The check for exportable types when the testedType is an interface seems to work okay. The check when it is not doesn't work and I think I know why. The call to IsInstanceOfType is defined by Microsoft as returning "true if the current Type is in the inheritance hierarchy of the object represented by the parameter, or if the current Type is an interface that the object supports. false if neither of these conditions is the case, or if the object is a null reference."
Full definition: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemtypeclassisinstanceoftypetopic.asp
The kind of object being passed to IsInstanceOfType is an object of type Type not an object of the type you actually want to compare. In order to use this method you will need to create an instance of the type you want to compare and pass that in.
However, I think the method that you really want to use in this case is IsAssignableFrom. This method takes an object of type Type instead of an actual instance. Furthermore, this method also returns:
"true if the parameter and the current Type represent the same type, or if the current Type is in the inheritance hierarchy of the parameter, or if the current Type is an interface that the parameter supports. false if none of these conditions are the case, or if the parameter is a null reference."
I think this covers all of the cases that you might be concerned with for your framework. I have included an example to illustrate this.
using System;
using System.Reflection;
namespace CheckAssignability
{
public interface IFoo
{
}
public class FooBase : IFoo
{
}
public class Foo : FooBase
{
}
public class Bar
{
}
class Class1
{
[STAThread]
static void Main(string[] args)
{
Foo f = new Foo();
FooBase fb = new FooBase();
Type fooType = typeof(Foo);
Type fooBaseType = typeof(FooBase);
Type ifooType = typeof(IFoo);
Type barType = typeof(Bar);
Console.WriteLine(String.Format("A Foo is {0} assignable from a Foo.", fooType.IsAssignableFrom(fooType) ? "" : "not"));
Console.WriteLine(String.Format("A FooBase is {0} assignable from a Foo.", fooBaseType.IsAssignableFrom(fooType) ? "" : "not"));
Console.WriteLine(String.Format("An IFoo is {0} assignable from a Foo.", ifooType.IsAssignableFrom(fooType) ? "" : "not"));
Console.WriteLine(String.Format("An IFoo is {0} assignable from a FooBase.", ifooType.IsAssignableFrom(fooBaseType) ? "" : "not"));
Console.WriteLine(String.Format("An IFoo is {0} assignable from a Bar.", ifooType.IsAssignableFrom(barType) ? "" : "not"));
}
}
}
Cheers,
Darin
|
|
|
|
|
|
|
Thanks, in fact, the loadtypes method looks much simpler now. I will integrate that in the next version.
public void LoadTestedTypes(Assembly a)
{
foreach(Type t in a.GetExportedTypes())
{
if (this.testedType.IsAssignableFrom(t) &&!t.IsAbstract )
{
this.testedTypes.Add(t);
}
}
}
Jonathan de Halleux.
www.dotnetwiki.org
|
|
|
|
|
| Comparisons
Carel Lotz | 21:09 3 Feb '04 |
|
|
Hi
Thanks for the excellent implementation. I have been looking for something of the kind for quite a while. I was wondering whether it is possible to add some kind of comparison feature which would enable you to automatically compare different runs against each other. Much in the same way that you run Unit tests to ensure that your code still performs some basic functions correctly, I am looking to run some kind of performance unit tests to ensure that the code is still performing at least as fast as a certain base-line of acceptable performance.
I see that you can store the results in XML and I gather from that you would be able to write some kind of tool to compare results of different runs to see whether performance has increased/decreased? Is there a better way to do this kind of comparison? Can it be built into the framework?
Thanks
Carel Lotz
|
|
|
|
|
|
|
Carel Lotz wrote:
I see that you can store the results in XML and I gather from that you would be able to write some kind of tool to compare results of different runs to see whether performance has increased/decreased? Is there a better way to do this kind of comparison? Can it be built into the framework?
I've added this to my "to think about" list. I'll let you know.
Jonathan de Halleux.
www.dotnetwiki.org
|
|
|
|
|
| Blank Graphics
rido | 7:11 28 Jan '04 |
|
|
Runnning the demos, I only get empy charts.
¿why?
|
|
|
|
|
|
|
