Monitoring Distributed Service Performance in .NET

• Download source files - 2.91 Kb
• Download demo project - 11.8 Kb

Introduction

The software application I am maintaining consists of 28 (yes, Twenty Eight) types of Windows services running on multiple machines. Each service might have a single instance (running on a single machine), multiple instances running on a single machine, or multiple instances running on multiple machines.

To make things more interesting, the application is running in a separate production environment to which I have only limited access. With so many services running on the back-end, I am faced with three questions:

• What is service X doing right now?
• Why is performing action Y slow?
• Do we need to buy more hardware to improve performance?

How can I monitor service performance in real-time and predict future performance as the application scales? The solution – instrument all services with Performance Counters.

What to measure

Deciding what to measure is not entirely simple. If you have a very simple application, you may be in luck, but when you have 28 different services doing different things, you have to analyze all your services and decide on what would be the best metrics to use. An important thing to do at this stage is to consult your operations team – get input from the people responsible for the production environment. The operations team (IT OPS in my case) would be the ones looking at the performance data. The operations team is responsible to alert me, the developer, if services are not behaving ‘as they should’. Get input from your operations team and your performance measurement will be much more productive.

In my case, after analyzing the services and discussing performance monitoring with the operations team, we decided to measure the following per-service information:

• Total number of requests made
• Requests per-second rate
• Total number of requests completed
• Requests completed per-second rate
• Average time to complete a request
• Number of active threads
• Number of items processed per-second

With the above information available for each service, the operations team felt they could easily do the following:

• Monitor service performance (with Number of items processed per-second being the most important counter and Average time to complete a request, the supporting counter)
• Measure service load (with the Requests made per-second and Number of active threads as the principal counters)

The question now became, how do I instrument the services to monitor performance?

Instrumenting for performance

There are several ways to instrument an application for performance monitoring:

• Measure performance in code and write the results to log files – requires adding application specific code to each and every service, uses resources to write to files, and forces me to sift through thousands of log-file lines to gather information.
• Measure performance in code and send the results to a central performance monitoring service – loads our message bus with performance monitoring data, requires me to develop a new performance monitoring service and GUI.
• Measure performance using Performance Counters and then view and analyze the information using Performance Monitor – combines built-in performance monitoring with an existing monitoring and analysis tool.

Once I made the decision to use Performance Counters, the rest was easy.

I decided to create a base class encapsulating performance monitoring and derive all the services from the base class. The decision to use a base class was almost a no-brainer as all the services were already being derived from System.ServiceProcess.ServiceBase.

Useful Performance Counter Types

The .NET Framework contains many types of performance counters. You can search for Performance Counter on MSDN and find all the types and the different methods used to calculate the counter value. Also, see the documentation for the PerformanceCounterType class.

In my experience, the following counter types are the most useful:

• PerformanceCounterType.NumberOfItems32 and its sibling PerformanceCounterType.NumberOfItems64 – when measuring quantities of items.
• PerformanceCounterType.RateOfCountsPerSecond32 – when measuring the per-second rate of item production or consumption.
• PerformanceCounterType.AverageTimer32 with PerformanceCounterType.AverageBase – measure the average time it takes to perform an action.

Creating Performance Counters

When creating Performance Counters, the first step is to decide on a Performance Category (using the PerformanceCounterCategory class). The Performance Category groups all your counters under one heading in Performance Monitor.

The second step is to decide how to define your Performance Counters – each counter defined may have multiple instances. In my case, I have decided to create a performance counter for each performance measurement and create an instance for each service running. The result of my decision is that I have one ItemsProcessPerSecond counter but I have a counter instance for each service instance currently running in my system.

Important fact – Performance Counters are globally visible to all processes on a specific machine, and once created, persist until specifically removed. Once you have created Performance Counters, they are always available to you and you do not need to re-create them.

And the specifics:

Check to see whether your performance counter category already exists using PerformanceCounterCategory.Exists(). If the category exists, you can either remove it with PerformanceCounterCategory.Remove() or skip the creation of the performance counters as everything should be in place.

Once you have a category defined, create a CounterCreationDataCollection. This collection holds all your performance counter definitions. Note that the order of adding counters to the collection is important.

A special type of counter is the average counter. An average counter measures a value over time and displays the average of the last two measurements. Associated with each average counter is a base counter that tracks the number of samples involved. When creating an average counter, the average base (PerformanceCounterType.AverageBase in the case of AverageTimer32) must immediately follow the average counter in the collection.

When your CounterCreationDataCollection is complete, create the performance counters by calling SetupServicePerformanceCounters().

// Does the category exists?
if (!PerformanceCounterCategory.Exists(PERF_CNT_CATEGORY)) 
{
    // Allways attempt to create the category
    CounterCreationDataCollection CCDC = new CounterCreationDataCollection();

    // Add the standard counters

    // Total requests made
    CounterCreationData totalRequestsMade = 
        new CounterCreationData(PERF_CNT_TOTAL_RQSTS_MADE,
        PERF_CNT_TOTAL_RQSTS_MADE_DESC, PerformanceCounterType.NumberOfItems32);
    CCDC.Add(totalRequestsMade);

    // Requests per seond
    CounterCreationData requestsPerSecond = 
        new CounterCreationData(PERF_CNT_RQSTS_PER_SCND,
        PERF_CNT_RQSTS_PER_SCND_DESC, 
        PerformanceCounterType.RateOfCountsPerSecond32);
    CCDC.Add(requestsPerSecond);

    ...

    // Items process per second
    CounterCreationData itemsPerSecond = 
        new CounterCreationData(PERF_CNT_ITEMS_PER_SCND,
        PERF_CNT_ITEMS_PER_SCND_DESC, 
        PerformanceCounterType.RateOfCountsPerSecond32);
    CCDC.Add(itemsPerSecond);

    // Average request completion time
    CounterCreationData averageRequestCompletionTime = 
        new CounterCreationData(PERF_CNT_AVG_RQST_CMPLT_TIME,
        PERF_CNT_AVG_RQST_CMPLT_TIME_DESC, 
        PerformanceCounterType.AverageTimer32);
    CounterCreationData averageRequestCompletionTimeBase = 
        new CounterCreationData(PERF_CNT_AVG_RQST_CMPLT_TIME_BASE,
        PERF_CNT_AVG_RQST_CMPLT_TIME__BASE_DESC, PerformanceCounterType.AverageBase);
    CCDC.Add(averageRequestCompletionTime);
    CCDC.Add(averageRequestCompletionTimeBase);

    ...

    // Create the category.
    PerformanceCounterCategory.Create(PERF_CNT_CATEGORY,
        PERF_CNT_CATEGORY_DESC, CCDC);
}

Finally, create a PerformanceCounterCategory using PerformanceCounterCategory.Create().

Now that you have the performance counters defined, it is time to create the actual performance counters – use the PerformanceCounter class to create new instances of the performance counters. When creating an instance of a performance counter, pass in the name of the instance. I use the name of the service (and the service instance if multiple services are running on the same machine) as the counter instance name.

// Create and initialize
m_totalRequestsMade = new PerformanceCounter(PERF_CNT_CATEGORY,
    PERF_CNT_TOTAL_RQSTS_MADE, m_serviceInstanceName, false);
m_totalRequestsMade.RawValue = 0;

m_requestsPerSecond = new PerformanceCounter(PERF_CNT_CATEGORY,
    PERF_CNT_RQSTS_PER_SCND, m_serviceInstanceName, false);
m_requestsPerSecond.RawValue = 0;

...

Using Performance Counters

The PerformanceCounter class supplies the following properties/methods to set the value of performance counters:

• PerformanceCounter.RawValue property can be used to assign an initial value to a counter. It only makes sense to assign value to counters. RawValue can also be used when monitoring numbers that fluctuate wildly, where Increment() and Decrement() are not useful. I use RawValue to set initial values and to set values for things such as queue lengths which I sample periodically.

• PerformanceCounter.Increment() and PerformanceCounter.IncrementBy() are used to increment the value of a counter instance.
• PerformanceCounter.Decrement() and PerformanceCounter.DecrementBy() are used to decrement the value of a counter instance.

/// <summary>
/// Call when a single item has been processed
/// </summary>
public void IncrementItemsProcessed()
{
    m_itemsPerSecond.Increment();
}

Removing Performance Counters

When you are done with a performance counter instance (when your service is stopping), make a call to the RemoveInstance() method.

/// <summary>
/// Remove the performance counter instances
/// A call to RemoveServicePerformanceCounters is made
/// to allow the removal of service-specific performance counters
/// </summary>
protected void RemovePerformanceCounters()
{
    m_totalRequestsMade.RemoveInstance();
    m_requestsPerSecond.RemoveInstance();

    ...

    m_numRunningThreads.RemoveInstance();
}

Monitoring Performance

Monitor performance using PerfMon (the Windows performance monitoring tool, PerfMon.exe) or any other WMI (Windows Management Interface) compatible monitoring tool.

PerfMon allows you to monitor Performance Counters on multiple machines and visually compare the values of many Performance Counter instances.

To view Performance Counters in PerfMon, choose ‘Add counter’ (the plus sign button), choose your category from the list of available performance categories, and choose which performance counters (and instances) to monitor.

PerfMon graphs the chosen performance counters so you can follow the counter values in real time. In addition, you can add counters such as CPU load, free memory and disk utilization, and track your service performance compared to the additional counters.

If you need to monitor multiple servers, you can add counters from remote servers (in addition to the local box) by choosing the server in the PerfMon server selector.

The MonitoredService class

MonitoredService is an abstract base class for a monitored service. The MonitoredService class implements the creation and instantiation of Performance Counters. In addition, MonitoredService provides convenient methods to set the value of the Performance Counters.

The MonitoredService class defines the following performance counters:

• TotalRequestsMade – of type NumberOfItems32.
• RequestsPerSecond – of type RateOfCountsPerSecond32.
• TotalRequestsCompleted – of type NumberOfItems32.
• ItemsPerSecond – of type RateOfCountsPerSecond32.
• NumThreads – of type NumberOfItems32.
• AverageRequestCompletionTime – of type AverageTimer32.

Using the MonitoredService class

In order to use the class, derive your service from MonitoredService (instead of deriving from System.ServiceProcess.ServiceBase) and implement the following methods:

• ListenToMessages()
• StopListeningToMessages()

/// <summary>
/// Initialize and start listening to messages
/// </summary>
protected abstract void ListenToMessages();

/// <summary>
/// Stop listening to messages (possibly temporarily)
/// </summary>
protected abstract void StopListeningToMessages();

(The above two methods are needed in order to implement OnStart(), OnPause(), OnContinue(), and OnStop()).

You may override OnStart(), OnPause(), OnContinue(), and OnStop() but please make sure to call the base class version to make sure performance counters are handled correctly.

To set the performance counters, the following methods are implemented in MonitoredService:

• DateTime IncRequestsMade() – increments the number of requests made to the service and returns the current time. Save the returned value for the call to IncRequestsCompleted().
• void IncRequestsCompleted(DateTime) – increments the number of requests completed and calculates the request completion time based on the start time passed in to the method.
• void IncrementItemsProcessed() – overloaded, can either increment the number of items processed by one or by the specified integer.
• void IncTreadCount() and void DecThreadCount() – increment and decrement the number of running threads. Call IncThreadCount() when creating a new thread and call DecThreadCount() when disposing off a thread.

MonitoredService Example

The supplied example (MonitorServiceExample.zip) is a Visual Studio .NET project containing the following:

• MonitoredService base class
• MonitoredServiceExample – a service that does some random work. MonitoredServiceExample is based on MonitoredService and updates the following Performance Counters:
  • ItemsPerSecond
  • NumThreads
• InstallService.bat – a batch file that installs two copies of MonitoredServiceExample, one instance is named MonitorServiceExample1 and the other MonitorServiceExample2.
• UninstallService.bat – a batch file that removes (uninstalls) the two services above.

Using the Example

Unzip and build the example in Debug mode (just because the install scripts are preset to look for the executable in bin/Debug).

Double click the InstallService.bat – the services will be installed. For each service, you will be prompted for the user credentials to use when running the service. You can modify the batch file (set the USER and PASSWORD variables) if you do not want to be prompted for the credentials information.

In Administrative Tools, Services, start the two services (MonitoredServiceExample1 and MonitoredServiceExample2).

Start PerfMon (Administrative Tools, Performance) and select the ‘MyMonitoredServices’ category. Choose the NumRunningThreads and ItemsProcessedPerSecond counters and select ‘All instances’. Click ‘Add’ and then ‘Close’.

You can now follow the progress of the two services. After about a minute, the services will be done and the performance indicators will go to zero. You can then stop and restart the services to see the graphs active again.

Enhancements

The MonitoredService class can be enhanced by adding the WaitForThreads() method to the OnPause() and OnStop() methods. See the article WaitForThreads.

Versions

Version 1.0 is the initial version. The code for this article was created specifically for publication. The actual code I use contains logging and error checking. Leaving all the logging and error checking in the code clutters the code. The article version of the code contains only the minimum required to illustrate the article points.