NTime - Performance unit testing tool

Introduction

This article presents a unique tool to run repeatable performance tests in an application developed by programmers. The NTime tool is very similar to NUnit tool - another unit testing tool, so the users of NUnit will find an almost similar GUI and functionality.

Background

This is the first release of the tool that performs repeatable tasks to help managers, architects, developers and testers test an application against its performance. This version may not be very consistent and it may report errors under different situations. However, the tool was tested under many situations.

Tool usage

This section provides information for the application management team that wishes to use NTime tool to keep applications working at the desired performance level.

Managers

Program managers are welcome to specify application performance boundaries at the beginning of the application development process. They may also note which modules and functions of the application need to work and at which time frames. The performance specifications written by them may be helpful for developers when designing the application performance. Further, the NTime tool tests their performance specifications against real, developed applications. The tests performed by NTime tool may be reproduced at different time periods to see whether the application is giving the expected performance. But the main advantage of using this tool is that it enables managers to create configuration test files for developers and testers to give them a starting point on how to optimize the application.

Developers

Developers use NTime tool to test whether their application works in accordance with the performance specifications designed by managers or by themselves. They need to run the NTime tool when they aren't sure whether their rewritten code still works well. The NTime tool can also be made to remain open on the desktop even when the programmers develop, compile or debug their applications - every application that is built will be reloaded in NTime automatically and shown with the updated tests.

Testers

Testers are actively in the performance test stage. They run repeatable tests after every application's alpha release. Testers will review hundreds or thousands of tests and register them to the bug tracing applications.

Architecture

NTime architecture is similar to that of NUnit tool. This section shows the features found in NTime tool.

Simultaneous compile and test time

NTime tool may still be running even when the compilation process of the application that we want to test is going on. After the application has been built, NTime will reflect all the changes, you just need to run the tests and wait for the performance test results whether they are accepted or rejected.

Test timings

Actually, NTime's timings are based on high frequency timers that rely on different onboard chipsets' architecture. This difference generates more or less precision in the performed tests. The disadvantages, while profiling the tests, include system processes and CPU load. Even after closing the heavy loaded applications, there may be other bad things that can occur - like, NTime needs some CPU and threads to perform complex tests and therefore you can indicate about 10 microseconds overhead, so if you want to test high frequency running functions, then you need to write inside of the test code a "for" loop statement i.e. 100 loops, and then test the result for the specified time multiplied by 100.

SDK documentation

NTime tool is supported by two documentation files. The NTime Framework SDK will be your starting point to test your applications against performance tests by adding .NET attributes into your code.

NTime framework SDK

NTime Framework SDK is the often used SDK documentation. This help file shows you the available .NET attributes that you can use in your testing applications. Many attributes are similar to that of NUnit attributes, so if you've used NUnit tool before, then working with NTime will be quite easy.

NTime GUI SDK

After the installation of NTime, you will find the NTime source documentation. This will help you to add some functionality to NTime, but remember every change that you make to this tool is only for your private and non-commercial use. Read License.rtf file for further information about NTime licensing.

Further development

This NTime version is a GUI only application. In future, I plan to develop a console application to provide batch script usability. For further improvements, you may send comments to adamslosarski@tlen.pl with your suggestions to help me make the application work better. Bugs found in this release will be fixed in the next version, so the earlier found errors will be corrected as soon as possible. The source code of NTime will be soon available on the sourceforge.net website, so you can track the latest versions.

Using the code

Below, you can see a sample code which uses all the attributes available with NTime framework. See the NTime framework SDK documentation for more information:

using System;
using NTime.Framework;

namespace NTime.Tests
{
    /// <SUMMARY>
    /// This is a tested class.
    /// </SUMMARY>
    [TimerFixture]
    public class TestClass
    {
        [TimerFixtureSetUp]
        public void GlobalSetUp()
        {
            // initialize one-time initialization data in
            // this testfixture.
        }

        [TimerFixtureTearDown]
        public void GlobalTearDown()
        {
            // release one-time initialized data in
            // this testfixture.
        }

        [TimerSetUp]
        public void LocalSetUp()
        {
            // initialize data for every test found
            // in this testfixture class
        }

        [TimerTearDown]
        public void LocalTearDown()
        {
            // release data for every finished test
            // found in this testfixture class
        }

        [TimerHitCountTest(300, Threads = 3,
          Unit = TimePeriod.Second)]
        public void WebRequest()
        {
            // invoke some code from real application
            // to test its functions against specified
            // performance.
            // in example we will keep our fictional
            // web server's response at 10 milliseconds
            if(System.Threading.Thread.CurrentThread.Name == 
                                                  "NTimeThread_0")
              System.Threading.Thread.Sleep(5);
            else if(System.Threading.Thread.CurrentThread.Name == 
                                                   NTimeThread_1")
              System.Threading.Thread.Sleep(8);
            else
              System.Threading.Thread.Sleep(10);
        }

        [TimerDurationTest(20, Unit = TimePeriod.Millisecond)]
        public void GameScreenFlicking()
        {
            // we want to calc game AI, 3d engine and
            // other stuff to keep it running at 50Hz
            // vertical screen sync.
            System.Threading.Thread.Sleep(5);
        }

        [TimerDurationTest(10, Unit = TimePeriod.Millisecond)]
        [TimerIgnore("This test is disabled.")]
        public void SuperSortAlgorithm()
        {
            // code here will not be profiled until TimerIgnore
            // flag was removed.
        }

        [TimerCounterTest("Process", "% Processor Time", 
          Threads = 1, InstanceName = "NTimeGUI", 
          MinimumValue = 0, MaximumValue = 50)]
        public void ProcessorUsage()
        {
            // we want to see whether our application is
            // optimized to work with minimal CPU usage.
            for(int i = 0; i < 150; i++)
            {
                System.Threading.Thread.Sleep(30);
            }
        }
    }
}

Points of interest

This tool contains some .NET technologies used inside the code, including application domains and .NET remoting - these technologies were introduced to allow simultaneous compile and run test functionality, so that the assemblies may be loaded into separate application domains which run shadowed assemblies, and so the original ones may be overwritten by the compiler. Also, the assemblies can be unloaded from the application domain anytime.

The second thing is the file monitoring which tracks file changes and refreshes the project test tree to reflect changes in the assemblies when a new DLL is deployed.

History

2006-03-28
- Bug fixes:
  1. Tested methods were launched five times to average time, but the LocalSetup method was launched only once. Now it is launching both during every test method call.
- Improvements:
  1. Added named threads to detect which thread is executing the test method. Look at the example method 'WebRequest'.
2005-04-05
- Bug fixes:
  1. Tested assemblies could not read the application configuration settings.
  2. NTimeSetup.exe installer failed when the source files were being selected for installation.
- Improvements:
  1. New option dialog box allows you to specify the application configuration settings file.
  2. NTime.Framework.dll was promoted to GAC to allow for better shared reference to it.
2004-12-08
- Updated NTime application with bug fixes and an additional console mode.
2004-05-14
- Added source code and documentation download separately.
2004-05-11
- First release of NTime tool.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here