Click here to Skip to main content
Click here to Skip to main content
Go to top

Windows Azure Diagnostics – Performance Counters in Action

, 27 Dec 2011
Rate this:
Please Sign up or sign in to vote.
Manage and monitor performance counters in Azure

Introduction

If you are familiar with performance counter in Windows, you probably know that there are hundreds of performance counters divided into logical categories, all serving the purpose of monitoring the operation system, application or service in Windows.

Windows operation system provides a great handy application (which I am sure you must have used or heard of) named perfmon.exe (Start->Run->perfmon).

You can still use perfmon in your Windows Azure environment to monitor a role instance (by remote desktop a.k.a. RDP to the role instance. Additional information is available here and here.)

Yet, monitoring an entire Azure application, composed of multiple role instances is extremely cumbersome, not to mention that the data is not saved locally (not by default).

With that motivation in mind, Microsoft created a mechanism called Windows Azure Diagnostics that automates the gathering and persistence of diagnostic data from the role instances (in the essence the virtual machines) into an Azure Storage for a later and a deeper investigation. Azure diagnostics retrieves many other types of diagnostics data like trace logs, crash dumps window event logs, etc. In this discussion, I will focus on performance counters which can share huge amount of critical information on how your Azure application functions. Practically, performance counters can help you isolate performance problem and most importantly save money by indicating if the Azure resources are optimally exploited.

How Windows Azure Diagnostics Works

Every role VM has a built-in diagnostics process (DiagnosticsAgent.exe) that is responsible for periodically collecting the diagnostic data, caching it to a local file and eventually storing to a predefined Azure Storage. Note that the diagnostics process can also be manually triggered.

Specifically, the diagnostics data for the performance counters are persisted to a designated Azure table named WADPerformanceCountersTable. Other diagnostics data such as trace logs, event logs, etc. are also stored in designated tables like: WadLogsTable, WADDiagnosticInfrastructureLogsTable, etc. (Additional information can be found here).

Every role instance has a configuration file located in the azure storage under directory (blob directory) called wad-control-container. The configuration file primarily defines the performance counters to be collected and associated collection rate.

The following displays the location of the diagnostic configuration file. Access wad-control-container using Cloud Storage Manager for Azure:

Capture1.PNG
The configuration file uses a standard XML format and can be manually modified (although not recommended).

Using the code

In this article, we will review:

  1. How to configure performance counters from the Role code (running in the Azure application) and remotely from an external program.
  2. How to access the data from the Azure storage table.
  3. Quick analysis of performance counters.

The entire code is available for download below.

First let's cover the API we going to use in order to enable monitoring for our role instances.

Windows Azure Diagnostics API

(definition taken from the MSDN)

To enable diagnostics monitoring in your role, you must first import the diagnostic module in the ServiceDefinition.csdef configuration file. Visual Studio provides an easy way to enable the diagnostics via the role properties window in the cloud solution (more information is available here).

Configuring Performance Counters from within a Role Code

The following code retrieves the role instance configuration (keep in mind that the diagnostics configuration is managed per role instance), sets a new performance counter to monitor and starts the diagnostics monitor with the modified configuration.

public class WebRole : RoleEntryPoint
{
    /// <summary>
    /// Entry point before starting the Role
    /// </summary>
    public override bool OnStart()
    {
        // Get the Role instance Diagnostics configuration. 
        var config = DiagnosticMonitor.GetDefaultInitialConfiguration();

        // Define interval for persisting the performance counter data to azure storage.
        config.PerformanceCounters.ScheduledTransferPeriod = TimeSpan.FromMinutes(5);
        // CPU Utilization Counter
        PerformanceCounterConfiguration cpuUtilizationCounter = 
                new PerformanceCounterConfiguration()
        {
            CounterSpecifier = @"\Processor(_Total)\% Processor Time",
            //define the sample internal for the specific performance counter
            SampleRate = TimeSpan.FromSeconds(1)
        };
        if (!config.PerformanceCounters.DataSources.Contains(cpuUtilizationCounter, 
                    new PerformanceCounterComparer()))
        {
            config.PerformanceCounters.DataSources.Add(cpuUtilizationCounter);
        }
        // Start diagnostic monitor with the new configuration.
        DiagnosticMonitor.Start
    ("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", config);
        return base.OnStart();
    }

    /// <summary>
    /// Performance Counter Comparer
    /// </summary>
    private class PerformanceCounterComparer : 
        IEqualityComparer<PerformanceCounterConfiguration>
    {
        public bool Equals(PerformanceCounterConfiguration a, 
                PerformanceCounterConfiguration b)
        {
            //Check whether the compared objects reference the same data.
            if (Object.ReferenceEquals(a, b)) return true;
            //Check whether any of the compared objects is null.
            if (Object.ReferenceEquals(a, null) || Object.ReferenceEquals(b, null))
                return false;
            // Check if the counter specifier and the sampling rate are the same.
            return (a.CounterSpecifier == b.CounterSpecifier && 
                a.SampleRate == b.SampleRate);
        }
        public int GetHashCode(PerformanceCounterConfiguration counter)
        {
            //Check whether the object is null
            if (Object.ReferenceEquals(counter, null)) return 0;
            //Get hash code for the CounterSpecifier field if it is not null.
            int hashCounterSpecifier = counter.CounterSpecifier == 
                           null ? 0 : counter.CounterSpecifier.GetHashCode();
            //Calculate the hash code for the counter.
            return hashCounterSpecifier ^ counter.SampleRate.GetHashCode();
        }
    }
}

Configuring Performance Counters Remotely

Configuring performance counters can also be performed externally to the role code, by using Microsoft.WindowsAzure.Diagnostics.Management.

const string storageAccoutName = "Storage-Name-Here"; 
const string privateKey = "Storge-Private-Key-Here";
const string deploymentId = "Deployment-Id-Here";
var storageAccount = CloudStorageAccount.Parse(String.Format(
    "DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", 
    storageAccoutName, privateKey));

// Get the diagnostics manager for the entire storage account.
var diagnosticManager = storageAccount.CreateDeploymentDiagnosticManager(deploymentId);

// Get diagnostics manager for the specific role instance.
RoleInstanceDiagnosticManager roleDiagManager = 
  diagnosticManager.GetRoleInstanceDiagnosticManager("WebRole1", "WebRole1_IN_0");

//Modify current configuration
var currentConfiguariton = roleDiagManager.GetCurrentConfiguration();
currentConfiguariton.PerformanceCounters.ScheduledTransferPeriod = 
                        TimeSpan.FromMinutes(5);
currentConfiguariton.PerformanceCounters.DataSources.Add
            (new PerformanceCounterConfiguration()
{
    CounterSpecifier = @"\Processor(_Total)\% Processor Time",
    SampleRate = TimeSpan.FromSeconds(1)
});

//Commit the changes
roleDiagManager.SetCurrentConfiguration(currentConfiguariton); 

Retrieving the Performance Counter Data

Now that we've configured the counters we want to monitor, let's access the recorded data from the Azure Table (WADPerformanceCountersTable) and display it.

I've created the PerformanceDataContext class that derives from TableServiceContext which is part of the ADO extensions that Microsoft provides in order to connect to Azure Tables. You can use a LINQ query in order to retrieve the data.

/// <summary>
/// Query helper for retrieving data from the WADPerformanceCountersTable
/// </summary>
public class QueryExecuter
{
    /// <summary>
    /// Cloud storage account client
    /// </summary>
    private CloudStorageAccount accountStorage;
    /// <summary>
    /// Default Constructor - Use development storage emulator.
    /// </summary>
    public QueryExecuter()
    {
        accountStorage = CloudStorageAccount.DevelopmentStorageAccount;
    }
    /// <summary>
    /// Constructor
    /// </summary>
    /// <param name="accountName">Azure storage name</param>
    /// <param name="privateKey">Azure storage private key</param>
    public QueryExecuter(string accountName, string privateKey)
    {
        accountStorage = CloudStorageAccount.Parse(String.Format(
          "DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", account
    }
    /// <summary>
    /// Retrieve Performance counter data
    /// </summary>
    /// <param name="counterFullName">Counter specifier full name</param>
    /// <param name="deploymentid">Deployment id</param>
    /// <param name="roleName">Role name</param>
    /// <param name="roleInstanceName">Role instance name</param>
    /// <param name="startPeriod">Start sample date time</param>
    /// <param name="endPeriod">End sample date time</param>
    /// <returns></returns>
    public List<PerformanceData> QueryPerformanceCounter(string counterFullName, 
        string deploymentid, string roleName, 
        string roleInstanceName, DateTime startPeriod, DateTime endPeriod)
    {
        PerformanceDataContext context = new PerformanceDataContext(
          accountStorage.TableEndpoint.ToString(), accountStorage.Credentials);
        var data = context.PerfData;
        CloudTableQuery<PerformanceData> query = null;
        query = (from d in data
                 where d.PartitionKey.CompareTo("0" + startPeriod.Ticks) >= 0
                                        && d.PartitionKey.CompareTo
                    ("0" + endPeriod.Ticks) <= 0
                                         && d.CounterName == counterFullName
                                             && d.EventTickCount >= startPeriod.Ticks
                                                 && d.EventTickCount <= endPeriod.Ticks
                                                      && d.DeploymentId == deploymentid
                                                         && d.Role == roleName
                                                             && d.RoleInstance == 
                            roleInstanceName
                 select d).AsTableServiceQuery<PerformanceData>();
        List<PerformanceData> selectedData = new List<PerformanceData>();
        try
        {
            selectedData = query.Execute().ToList<PerformanceData>();
        }
        catch
        {
        }
        return selectedData;
    }
}

For this demonstration, I've created a Chart in WinForm that is populated with data retrieved from the diagnostic table.

The graph created displays the CPU utilization of specific role instance for the last 2 hours.

Capture2.PNG

Points of Interest

Quest Software develops a very handy and easy to use tool called Spotlight on Azure. We provide an in depth monitoring capabilities of the entire Azure subscription starting from the role instance and above, data aggregation, historical display, alerting mechanism and custom user defined counters for deep analysis.

Capture3.PNG

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Shay Yannay
Software Developer Quest Software
Israel Israel
Shay Yannay is a Senior Software Developer and Windows Azure Domain Expert.
He is experienced with designing and developing highly scalable, distributed, 24x7 availability complex system. Shay also specializes in performance management & diagnostics of multi-tier applications.
He is passionate about the cloud technologies and trends, specifically with Microsoft Azure.
He currently works for Dell as an Azure Specialist.
 
Shay holds a B.Sc in Communication Systems Engineering from the Ben-Gurion university.
 
Personal Blog: http://shayyannay.wordpress.com
Follow on   Twitter

Comments and Discussions

 
QuestionCounter aggregation Pinmemberbitstr3am7-Apr-14 21:10 
GeneralMy vote of 5 PinmvpKanasz Robert28-Sep-12 5:36 
QuestionSource Code Pinmembersahil.gupta56211-Sep-12 2:37 
SuggestionGood article - one more point to add PinmemberHobby Jose10-Sep-12 21:15 
GeneralMy vote of 5 PinmemberOJ23-Jan-12 4:19 
Very well written. Example code is an excellent way to get started on Azure.
QuestionExcellent PinmemberOJ23-Jan-12 4:17 
AnswerRe: Excellent PinmemberShay Yannay30-Jan-12 20:50 
GeneralMy vote of 5 PinmemberNavaVa30-Dec-11 5:29 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.140916.1 | Last Updated 27 Dec 2011
Article Copyright 2011 by Shay Yannay
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid