Click here to Skip to main content
13,458,736 members
Click here to Skip to main content
Add your own
alternative version


10 bookmarked
Posted 16 Apr 2009

Using a Profiler to Speed Application Performance

, 16 Apr 2009
Ryan Garaygay found some significant slow-down in his data processing time and needed to regain tighter control of the Business Intelligence process. He recounts how he managed to increase the performance of his application by more than 5 times and shares the lessons he learned.

Editorial Note

This article is in the Product Showcase section for our sponsors at CodeProject. These articles are intended to provide you with information on products and services that we consider useful and of value to developers.

Try out ANTS Profiler for yourself: Download a free trial of ANTS Profiler here

As part of our company’s business intelligence, marketing effectiveness platform (including a high-scale contact relationship management application) is a system which uses SSIS (among other things) to extract, transform and load huge amounts of data from a wide range of different sources. Once it’s all in a central SQL Server database, a background C# batch processing application works through the data using highly complex grouping and clustering algorithms to analyze and make the data consistent. Somewhat similar to SSIS fuzzy grouping but we had extra considerations and needed more control on the process.


When this was taking weeks rather than days we knew we had a problem and that was when I got more involved in the project. It’s fine if this task takes up to a week to process all the data (currently around 9 million rows for example in one of our clients), but it was now taking this long to process less than half the data!

Bringing in a Profiler

I initially started inserting timing code throughout the code, but this is obviously very tedious, and can be inaccurate. Worse, you depend on guesswork to work out where to effectively insert code. I quickly realized I needed proper non-intrusive profiling.

A quick Google search showed up a shortlist: dotTrace from JetBrains and ANTS Profiler from Red Gate. Although I love JetBrains’ Resharper, I found ANTS Profiler was more usable at taking me down to which methods and which lines were taking the time plus being able to look at those lines in the same window. That is without going back and forth from Visual Studio

The analysis proved to be an incremental process. Each time, I would work down from the method level timing and find the top bottleneck. At first, the bottlenecks turned out to be data access, so I used SQL Profiler and Database Tuning Advisor to work out the necessary DB and query changes.

5X Performance Gains (Or More)

This still left performance problems in the C# code itself. Assuming that the quickest way to get a major improvement was to change the core algorithms, I tried rewriting some of the fuzzy grouping algorithms – this made the code a lot more complicated, but turned out to make almost no improvement to performance. As ever, before you optimize, always measure. So I reverted the code, and went back to ANTS Profiler again.

Surprisingly, some of the major problems turned out to be the simple things – regexp and String performance, and basic collection types. A couple of changes in the string handling*, and using hashtables instead of lists (Hashtable is faster than SortedDictionary, SortedList or List), and we now get through the full 9 million rows in 2-3 days – at least 5 times faster!


ANTS Profiler results before optimization.


ANTS Profiler results after optimization.

Lessons Learned

In retrospect, the lessons are fairly obvious:

  • Use a profiler to measure before you optimize – it really sucks trying to optimize something only to find out that it is not the bottleneck.
  • Use the appropriate DB tuning tools.
  • Keep measuring – it can be the simple things that are causing the problem.

* <string>.SubString(...) does some considerable lifting, so if you want to check if the first two chars in a string are equal to some other string then you should consider using <string>.StartsWith instead or avoid the SubString if you can.


Code before optimization.


Code after optimization.

Note: Although it is generally not recommended to place a dictionary as hard-coded values, the stakeholders in this project have specific reasons for doing so.

Also profiler screenshots provided was made only against a subset of the data

Try out ANTS Profiler for yourself: Download a free trial of ANTS Profiler here


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

You may also be interested in...


Comments and Discussions

General[Message Deleted] Pin
Talking Clipboard20-Apr-09 22:03
memberTalking Clipboard20-Apr-09 22:03 
GeneralRe: [My vote of 1] Amazing poor article Pin
deanpm21-Apr-09 13:05
memberdeanpm21-Apr-09 13:05 
GeneralRe: [My vote of 1] Amazing poor article Pin
Savara20-May-09 6:41
memberSavara20-May-09 6:41 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.180323.1 | Last Updated 16 Apr 2009
Article Copyright 2009 by Red Gate
Everything else Copyright © CodeProject, 1999-2018
Layout: fixed | fluid