Click here to Skip to main content
Click here to Skip to main content

Process Dump triggered on Custom Performance Counter Threshold

, 13 May 2012 CPOL
Rate this:
Please Sign up or sign in to vote.
Generating process dump files based on custom performance counter thresholds

Introduction

I am a big fan of using right tool for generating process dumps depending upon  what type of scenario you are debugging and what type of constraints you are facing. I have blogged about the case for ADPlus, DebugDiag & ProcDump . These are not the only tools available for creating dump files and there are other means of generating process dumps as well. In general these tools provide enough flexibility and options for generating dump file for widely needed problem categories like Application Crashes, hangs, memory issues, slow performance etc. However sometimes there are situations where you need more flexibility on what event should trigger dump file generation. This article describes how this could be done using Performance Counters, Windows Task Scheduler and ADPlus.

Background

This article assumes you are familiar with basics of production debugging and have some basic know how of dump files. If not, you can watch my video on this topic here.

Performance Counters

Performance Counters provide useful information about how well operation system or any given application is performing. This information could be used to monitor overall health and reliability of application and also to troubleshoot performance bottlenecks etc. You can also create custom counters and use these to keep track of your application health and stability.

In this article, I will demonstrate how you can automate generation of dump files based on threshold defined on these custom performance counters.

Creating Custom Counters

There are several ways of creating customer counters. Covering all possible methods of creating these custom counters is out of scope for this article, however, I will only discuss steps involved in creating a custom counter using Server Explorer. If you need to learn more about creating custom counters, the article found here could be a good start.

Using Server Explorer is probably simplest way of creating custom performance counter. Just launch Server Explorer from using Visual Studio, right click on Performance Counter node and select Create New Category option as shown in Figure 1.

Figure 1: Performance Counter context menu

You will get Performance Counter Builder dialog where you can enter information about Category and also to add individual Performance counters withint this category as shown in figure 2.

Figure 2: Performance Counter Builder Dialog

Let's assume that we have start using and publishing this counter in application. Also assume that once in a while TransactionsCount is getting quite high in a production environment and we are not been able to re-produce this problem in house. This situation obviously will make debugging and troubleshooting this problem quite tricky. We will have to adapt Production Debugging techniques, which involves taking a process dump in production environment only when the problem is happening. Challenge is of course how to get the process dump as this problem only happens intermittently. This is where we can use Peformance Counter's Data collector to trigger a creating a task to generate process dump only when the process reaches a pre-defined threshold for our custom performance counter.

Listing below show the code that will cause the TransactionsCount to go high.

private void SimulateHighTransactionsCount()
{
 for (int i = 0; i < 100; i++)
 {
  IncrementTransactionsCount();
  System.Threading.Thread.Sleep(1000 * 2);
 }
}
 
private void IncrementTransactionsCount()
{
 using (PerformanceCounter TransactionCount = new PerformanceCounter())
 {
  TransactionCount.CategoryName = "MyApplicationStats";
  TransactionCount.CounterName = "TransactionsCount";
  TransactionCount.ReadOnly = false;
  TransactionCount.Increment();
 }
}

Once the code above get execute, you can see custom performance counter go high as shown in Figure 3 below.

Figure 3: Performance Monitor

Setting up Data-Collector

In order to get a process dump file in the scenario described above, we will first setup a user-defined Data Collector using Performance Monitor. Let's launch the PerfMon, right click on option for User Defined Data Collector Sets and select New -> Data Collector Set option as shown in figure 4 below.

Figure 4: Performance Monitor

At this point, you will get a dialog where you can name your Data Collector. Make sure you select "Create manually" option.

Figure 5: Create new Data Collector dialog

Next dialog will provide you option for adding your performance counter for this Data Collector. Hit Add button here.

Figure 6: Add Performance Counter in Data Collector



Hitting Add button provides you with a list of performance counter. Select and add our custom performance counter from this dialog as shown in figure 7 below.

Figure 7: Choose custom performance counter

Next dialog provides you ability to set the threshold. As shown in figure 8, we want to use 25 as TransactionsCount threshold for generating the dump file.

Figure 8: Defined threshold

On next dialog, just hit "Save and close".

Figure 9: Save your work.

Now there are couple of adjustments that we would like to do in this newly created Data Collector. So click on the properties for this data collector, and you will get a dialog with three tabs, Alerts, Alert Action and Alert Task. First thing we want to do is adjust sample interval in the Alerts tab. For this example, I am adjusting it to 10 seconds, however, for a production environment this will not be an appropriate value. You may not want to monitor your counter this frequently and be better off using a higher value.

Figure 10: Use an appropriate sample interval

Next thing you want to do here is on the Alert Task Tab. You would like to setup the name of the task that will be executed whenever this threshold get hit. The name that you use here has to match exactly with the name that will be used in next step where we will create a task using Windows Task Scheduler.

Figure 11: Setup the task that you like to execute when threshold is reached.

Windows Task Scheduler

In order to configure the task that will be executed each time our performance counter threshold is reached, we will launch Windows Task Scheduler and Create a task.

Figure 12: Create Task.

Next we will setup the Action. In our case, it will be commands that will get executed.

Figure 13: Setup Action

The contents of CreateDump.cmd as follows.

CD C:\"Program Files"\"Debugging Tools for Windows"
ADPlus -hang -pn CustomPerformanceCounter.exe -o C:\Test
Logman -stop TransactionsSpike
Pause

We are simply running ADPlus in hang mode. This mean that as soon as ADPlus command is run it will attach to the process name CustomPerformanceCounter (configured using -pn switch) and will generate a process dump file in C:\Test folder. Note that we are stopping the TransactionsSpike data collector right after ADPlus is run, otherwise it will continue to keep running the task which in turn will continue to write the dump file. Given the potentially big size of dump files, this is not something you want to have in your production environment.

Our setup is all complete now. We just have to go back and start the data-collector in the Performance monitor as by default it is created in the the stop mode. Now as soon as you start the data collector and the threshold is hit, you should be able see the dump file. Once you have the dump file, you can use any tool like WinDbg or DebugDiag to analyze dump file and find the root cause of the problem.

Summary

In this article, I demonstrated how you can automate process dump file generation based upon custom performance counter threshold using a combination of ADPlus, Performance counters and Windows Task Manager.  I hope you find this technique useful.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Kamran Bilgrami
Architect
Canada Canada
Kamran Bilgrami is a seasoned software developer with background in designing mission critical applications for carrier grade telecom networks. More recently he is involved in design & development of real-time biometric based security solutions. His areas of interest include .NET, software security, mathematical modeling and patterns.
 
He blogs regularly at http://WindowsDebugging.Wordpress.com
Follow on   Twitter

Comments and Discussions

 
GeneralExcellent article PinmemberTahaFarid19-Nov-12 16:29 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web04 | 2.8.141216.1 | Last Updated 13 May 2012
Article Copyright 2012 by Kamran Bilgrami
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid