The software application I am maintaining consists of 28 (yes, Twenty Eight) types of Windows services running on multiple machines. Each service might have a single instance (running on a single machine), multiple instances running on a single machine, or multiple instances running on multiple machines.
To make things more interesting, the application is running in a separate production environment to which I have only limited access. With so many services running on the back-end, I am faced with three questions:
- What is service X doing right now?
- Why is performing action Y slow?
- Do we need to buy more hardware to improve performance?
How can I monitor service performance in real-time and predict future performance as the application scales? The solution – instrument all services with Performance Counters.
What to measure
Deciding what to measure is not entirely simple. If you have a very simple application, you may be in luck, but when you have 28 different services doing different things, you have to analyze all your services and decide on what would be the best metrics to use. An important thing to do at this stage is to consult your operations team – get input from the people responsible for the production environment. The operations team (IT OPS in my case) would be the ones looking at the performance data. The operations team is responsible to alert me, the developer, if services are not behaving ‘as they should’. Get input from your operations team and your performance measurement will be much more productive.
In my case, after analyzing the services and discussing performance monitoring with the operations team, we decided to measure the following per-service information:
- Total number of requests made
- Requests per-second rate
- Total number of requests completed
- Requests completed per-second rate
- Average time to complete a request
- Number of active threads
- Number of items processed per-second
With the above information available for each service, the operations team felt they could easily do the following:
- Monitor service performance (with Number of items processed per-second being the most important counter and Average time to complete a request, the supporting counter)
- Measure service load (with the Requests made per-second and Number of active threads as the principal counters)
The question now became, how do I instrument the services to monitor performance?
Instrumenting for performance
There are several ways to instrument an application for performance monitoring:
- Measure performance in code and write the results to log files – requires adding application specific code to each and every service, uses resources to write to files, and forces me to sift through thousands of log-file lines to gather information.
- Measure performance in code and send the results to a central performance monitoring service – loads our message bus with performance monitoring data, requires me to develop a new performance monitoring service and GUI.
- Measure performance using Performance Counters and then view and analyze the information using Performance Monitor – combines built-in performance monitoring with an existing monitoring and analysis tool.
Once I made the decision to use Performance Counters, the rest was easy.
I decided to create a base class encapsulating performance monitoring and derive all the services from the base class. The decision to use a base class was almost a no-brainer as all the services were already being derived from
Useful Performance Counter Types
The .NET Framework contains many types of performance counters. You can search for Performance Counter on MSDN and find all the types and the different methods used to calculate the counter value. Also, see the documentation for the
In my experience, the following counter types are the most useful:
PerformanceCounterType.NumberOfItems32 and its sibling
PerformanceCounterType.NumberOfItems64 – when measuring quantities of items.
PerformanceCounterType.RateOfCountsPerSecond32 – when measuring the per-second rate of item production or consumption.
PerformanceCounterType.AverageBase – measure the average time it takes to perform an action.
Creating Performance Counters
When creating Performance Counters, the first step is to decide on a Performance Category (using the
PerformanceCounterCategory class). The Performance Category groups all your counters under one heading in Performance Monitor.
The second step is to decide how to define your Performance Counters – each counter defined may have multiple instances. In my case, I have decided to create a performance counter for each performance measurement and create an instance for each service running. The result of my decision is that I have one
ItemsProcessPerSecond counter but I have a counter instance for each service instance currently running in my system.
Important fact – Performance Counters are globally visible to all processes on a specific machine, and once created, persist until specifically removed. Once you have created Performance Counters, they are always available to you and you do not need to re-create them.
And the specifics:
Check to see whether your performance counter category already exists using
PerformanceCounterCategory.Exists(). If the category exists, you can either remove it with
PerformanceCounterCategory.Remove() or skip the creation of the performance counters as everything should be in place.
Once you have a category defined, create a
CounterCreationDataCollection. This collection holds all your performance counter definitions. Note that the order of adding counters to the collection is important.
A special type of counter is the average counter. An average counter measures a value over time and displays the average of the last two measurements. Associated with each average counter is a base counter that tracks the number of samples involved. When creating an average counter, the average base (
PerformanceCounterType.AverageBase in the case of
AverageTimer32) must immediately follow the average counter in the collection.
CounterCreationDataCollection is complete, create the performance counters by calling
CounterCreationDataCollection CCDC = new CounterCreationDataCollection();
CounterCreationData totalRequestsMade =
CounterCreationData requestsPerSecond =
CounterCreationData itemsPerSecond =
CounterCreationData averageRequestCompletionTime =
CounterCreationData averageRequestCompletionTimeBase =
Finally, create a
Now that you have the performance counters defined, it is time to create the actual performance counters – use the
PerformanceCounter class to create new instances of the performance counters. When creating an instance of a performance counter, pass in the name of the instance. I use the name of the service (and the service instance if multiple services are running on the same machine) as the counter instance name.
m_totalRequestsMade = new PerformanceCounter(PERF_CNT_CATEGORY,
PERF_CNT_TOTAL_RQSTS_MADE, m_serviceInstanceName, false);
m_totalRequestsMade.RawValue = 0;
m_requestsPerSecond = new PerformanceCounter(PERF_CNT_CATEGORY,
PERF_CNT_RQSTS_PER_SCND, m_serviceInstanceName, false);
m_requestsPerSecond.RawValue = 0;
Using Performance Counters
PerformanceCounter class supplies the following properties/methods to set the value of performance counters:
PerformanceCounter.RawValue property can be used to assign an initial value to a counter. It only makes sense to assign value to counters.
RawValue can also be used when monitoring numbers that fluctuate wildly, where
Decrement() are not useful. I use
RawValue to set initial values and to set values for things such as queue lengths which I sample periodically.
PerformanceCounter.IncrementBy() are used to increment the value of a counter instance.
PerformanceCounter.DecrementBy() are used to decrement the value of a counter instance.
public void IncrementItemsProcessed()
Removing Performance Counters
When you are done with a performance counter instance (when your service is stopping), make a call to the
protected void RemovePerformanceCounters()
Monitor performance using PerfMon (the Windows performance monitoring tool, PerfMon.exe) or any other WMI (Windows Management Interface) compatible monitoring tool.
PerfMon allows you to monitor Performance Counters on multiple machines and visually compare the values of many Performance Counter instances.
To view Performance Counters in PerfMon, choose ‘Add counter’ (the plus sign button), choose your category from the list of available performance categories, and choose which performance counters (and instances) to monitor.
PerfMon graphs the chosen performance counters so you can follow the counter values in real time. In addition, you can add counters such as CPU load, free memory and disk utilization, and track your service performance compared to the additional counters.
If you need to monitor multiple servers, you can add counters from remote servers (in addition to the local box) by choosing the server in the PerfMon server selector.
The MonitoredService class
MonitoredService is an abstract base class for a monitored service. The
MonitoredService class implements the creation and instantiation of Performance Counters. In addition,
MonitoredService provides convenient methods to set the value of the Performance Counters.
MonitoredService class defines the following performance counters:
TotalRequestsMade – of type
RequestsPerSecond – of type
TotalRequestsCompleted – of type
ItemsPerSecond – of type
NumThreads – of type
AverageRequestCompletionTime – of type
Using the MonitoredService class
In order to use the class, derive your service from
MonitoredService (instead of deriving from
System.ServiceProcess.ServiceBase) and implement the following methods:
protected abstract void ListenToMessages();
protected abstract void StopListeningToMessages();
(The above two methods are needed in order to implement
You may override
OnStop() but please make sure to call the base class version to make sure performance counters are handled correctly.
To set the performance counters, the following methods are implemented in
DateTime IncRequestsMade() – increments the number of requests made to the service and returns the current time. Save the returned value for the call to
void IncRequestsCompleted(DateTime) – increments the number of requests completed and calculates the request completion time based on the start time passed in to the method.
void IncrementItemsProcessed() – overloaded, can either increment the number of items processed by one or by the specified integer.
void IncTreadCount() and
void DecThreadCount() – increment and decrement the number of running threads. Call
IncThreadCount() when creating a new thread and call
DecThreadCount() when disposing off a thread.
The supplied example (MonitorServiceExample.zip) is a Visual Studio .NET project containing the following:
MonitoredService base class
MonitoredServiceExample – a service that does some random work. MonitoredServiceExample is based on
MonitoredService and updates the following Performance Counters:
- InstallService.bat – a batch file that installs two copies of MonitoredServiceExample, one instance is named MonitorServiceExample1 and the other MonitorServiceExample2.
- UninstallService.bat – a batch file that removes (uninstalls) the two services above.
Using the Example
Unzip and build the example in Debug mode (just because the install scripts are preset to look for the executable in bin/Debug).
Double click the InstallService.bat – the services will be installed. For each service, you will be prompted for the user credentials to use when running the service. You can modify the batch file (set the
PASSWORD variables) if you do not want to be prompted for the credentials information.
In Administrative Tools, Services, start the two services (MonitoredServiceExample1 and MonitoredServiceExample2).
Start PerfMon (Administrative Tools, Performance) and select the ‘MyMonitoredServices’ category. Choose the NumRunningThreads and ItemsProcessedPerSecond counters and select ‘All instances’. Click ‘Add’ and then ‘Close’.
You can now follow the progress of the two services. After about a minute, the services will be done and the performance indicators will go to zero. You can then stop and restart the services to see the graphs active again.
MonitoredService class can be enhanced by adding the
WaitForThreads() method to the
OnStop() methods. See the article WaitForThreads.
Version 1.0 is the initial version. The code for this article was created specifically for publication. The actual code I use contains logging and error checking. Leaving all the logging and error checking in the code clutters the code. The article version of the code contains only the minimum required to illustrate the article points.