Windows Azure Diagnostics – Performance Counters in Action

• Download source code - 26.46 KB

Introduction

If you are familiar with performance counter in Windows, you probably know that there are hundreds of performance counters divided into logical categories, all serving the purpose of monitoring the operation system, application or service in Windows.

Windows operation system provides a great handy application (which I am sure you must have used or heard of) named perfmon.exe (Start->Run->perfmon).

You can still use perfmon in your Windows Azure environment to monitor a role instance (by remote desktop a.k.a. RDP to the role instance. Additional information is available here and here.)

Yet, monitoring an entire Azure application, composed of multiple role instances is extremely cumbersome, not to mention that the data is not saved locally (not by default).

With that motivation in mind, Microsoft created a mechanism called Windows Azure Diagnostics that automates the gathering and persistence of diagnostic data from the role instances (in the essence the virtual machines) into an Azure Storage for a later and a deeper investigation. Azure diagnostics retrieves many other types of diagnostics data like trace logs, crash dumps window event logs, etc. In this discussion, I will focus on performance counters which can share huge amount of critical information on how your Azure application functions. Practically, performance counters can help you isolate performance problem and most importantly save money by indicating if the Azure resources are optimally exploited.

How Windows Azure Diagnostics Works

Every role VM has a built-in diagnostics process (DiagnosticsAgent.exe) that is responsible for periodically collecting the diagnostic data, caching it to a local file and eventually storing to a predefined Azure Storage. Note that the diagnostics process can also be manually triggered.

Specifically, the diagnostics data for the performance counters are persisted to a designated Azure table named WADPerformanceCountersTable. Other diagnostics data such as trace logs, event logs, etc. are also stored in designated tables like: WadLogsTable, WADDiagnosticInfrastructureLogsTable, etc. (Additional information can be found here).

Every role instance has a configuration file located in the azure storage under directory (blob directory) called wad-control-container. The configuration file primarily defines the performance counters to be collected and associated collection rate.

The following displays the location of the diagnostic configuration file. Access wad-control-container using Cloud Storage Manager for Azure:

Using the code

In this article, we will review:

1. How to configure performance counters from the Role code (running in the Azure application) and remotely from an external program.
2. How to access the data from the Azure storage table.
3. Quick analysis of performance counters.

The entire code is available for download below.

First let's cover the API we going to use in order to enable monitoring for our role instances.

Windows Azure Diagnostics API

(definition taken from the MSDN)

• Microsoft.WindowsAzure.Diagnostics – allows you to collect logs and diagnostic information from code running within your role.
• Microsoft.WindowsAzure.Diagnostics.Management - allows you to collect logs and diagnostic information remotely.

To enable diagnostics monitoring in your role, you must first import the diagnostic module in the ServiceDefinition.csdef configuration file. Visual Studio provides an easy way to enable the diagnostics via the role properties window in the cloud solution (more information is available here).

Configuring Performance Counters from within a Role Code

The following code retrieves the role instance configuration (keep in mind that the diagnostics configuration is managed per role instance), sets a new performance counter to monitor and starts the diagnostics monitor with the modified configuration.

public class WebRole : RoleEntryPoint
{
    /// <summary>
    /// Entry point before starting the Role
    /// </summary>
    public override bool OnStart()
    {
        // Get the Role instance Diagnostics configuration. 
        var config = DiagnosticMonitor.GetDefaultInitialConfiguration();

        // Define interval for persisting the performance counter data to azure storage.
        config.PerformanceCounters.ScheduledTransferPeriod = TimeSpan.FromMinutes(5);
        // CPU Utilization Counter
        PerformanceCounterConfiguration cpuUtilizationCounter = 
                new PerformanceCounterConfiguration()
        {
            CounterSpecifier = @"\Processor(_Total)\% Processor Time",
            //define the sample internal for the specific performance counter
            SampleRate = TimeSpan.FromSeconds(1)
        };
        if (!config.PerformanceCounters.DataSources.Contains(cpuUtilizationCounter, 
                    new PerformanceCounterComparer()))
        {
            config.PerformanceCounters.DataSources.Add(cpuUtilizationCounter);
        }
        // Start diagnostic monitor with the new configuration.
        DiagnosticMonitor.Start
    ("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", config);
        return base.OnStart();
    }

    /// <summary>
    /// Performance Counter Comparer
    /// </summary>
    private class PerformanceCounterComparer : 
        IEqualityComparer<PerformanceCounterConfiguration>
    {
        public bool Equals(PerformanceCounterConfiguration a, 
                PerformanceCounterConfiguration b)
        {
            //Check whether the compared objects reference the same data.
            if (Object.ReferenceEquals(a, b)) return true;
            //Check whether any of the compared objects is null.
            if (Object.ReferenceEquals(a, null) || Object.ReferenceEquals(b, null))
                return false;
            // Check if the counter specifier and the sampling rate are the same.
            return (a.CounterSpecifier == b.CounterSpecifier && 
                a.SampleRate == b.SampleRate);
        }
        public int GetHashCode(PerformanceCounterConfiguration counter)
        {
            //Check whether the object is null
            if (Object.ReferenceEquals(counter, null)) return 0;
            //Get hash code for the CounterSpecifier field if it is not null.
            int hashCounterSpecifier = counter.CounterSpecifier == 
                           null ? 0 : counter.CounterSpecifier.GetHashCode();
            //Calculate the hash code for the counter.
            return hashCounterSpecifier ^ counter.SampleRate.GetHashCode();
        }
    }
}

Configuring Performance Counters Remotely

Configuring performance counters can also be performed externally to the role code, by using Microsoft.WindowsAzure.Diagnostics.Management.

const string storageAccoutName = "Storage-Name-Here"; 
const string privateKey = "Storge-Private-Key-Here";
const string deploymentId = "Deployment-Id-Here";
var storageAccount = CloudStorageAccount.Parse(String.Format(
    "DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", 
    storageAccoutName, privateKey));

// Get the diagnostics manager for the entire storage account.
var diagnosticManager = storageAccount.CreateDeploymentDiagnosticManager(deploymentId);

// Get diagnostics manager for the specific role instance.
RoleInstanceDiagnosticManager roleDiagManager = 
  diagnosticManager.GetRoleInstanceDiagnosticManager("WebRole1", "WebRole1_IN_0");

//Modify current configuration
var currentConfiguariton = roleDiagManager.GetCurrentConfiguration();
currentConfiguariton.PerformanceCounters.ScheduledTransferPeriod = 
                        TimeSpan.FromMinutes(5);
currentConfiguariton.PerformanceCounters.DataSources.Add
            (new PerformanceCounterConfiguration()
{
    CounterSpecifier = @"\Processor(_Total)\% Processor Time",
    SampleRate = TimeSpan.FromSeconds(1)
});

//Commit the changes
roleDiagManager.SetCurrentConfiguration(currentConfiguariton); 

Retrieving the Performance Counter Data

Now that we've configured the counters we want to monitor, let's access the recorded data from the Azure Table (WADPerformanceCountersTable) and display it.

I've created the PerformanceDataContext class that derives from TableServiceContext which is part of the ADO extensions that Microsoft provides in order to connect to Azure Tables. You can use a LINQ query in order to retrieve the data.

/// <summary>
/// Query helper for retrieving data from the WADPerformanceCountersTable
/// </summary>
public class QueryExecuter
{
    /// <summary>
    /// Cloud storage account client
    /// </summary>
    private CloudStorageAccount accountStorage;
    /// <summary>
    /// Default Constructor - Use development storage emulator.
    /// </summary>
    public QueryExecuter()
    {
        accountStorage = CloudStorageAccount.DevelopmentStorageAccount;
    }
    /// <summary>
    /// Constructor
    /// </summary>
    /// <param name="accountName">Azure storage name</param>
    /// <param name="privateKey">Azure storage private key</param>
    public QueryExecuter(string accountName, string privateKey)
    {
        accountStorage = CloudStorageAccount.Parse(String.Format(
          "DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", account
    }
    /// <summary>
    /// Retrieve Performance counter data
    /// </summary>
    /// <param name="counterFullName">Counter specifier full name</param>
    /// <param name="deploymentid">Deployment id</param>
    /// <param name="roleName">Role name</param>
    /// <param name="roleInstanceName">Role instance name</param>
    /// <param name="startPeriod">Start sample date time</param>
    /// <param name="endPeriod">End sample date time</param>
    /// <returns></returns>
    public List<PerformanceData> QueryPerformanceCounter(string counterFullName, 
        string deploymentid, string roleName, 
        string roleInstanceName, DateTime startPeriod, DateTime endPeriod)
    {
        PerformanceDataContext context = new PerformanceDataContext(
          accountStorage.TableEndpoint.ToString(), accountStorage.Credentials);
        var data = context.PerfData;
        CloudTableQuery<PerformanceData> query = null;
        query = (from d in data
                 where d.PartitionKey.CompareTo("0" + startPeriod.Ticks) >= 0
                                        && d.PartitionKey.CompareTo
                    ("0" + endPeriod.Ticks) <= 0
                                         && d.CounterName == counterFullName
                                             && d.EventTickCount >= startPeriod.Ticks
                                                 && d.EventTickCount <= endPeriod.Ticks
                                                      && d.DeploymentId == deploymentid
                                                         && d.Role == roleName
                                                             && d.RoleInstance == 
                            roleInstanceName
                 select d).AsTableServiceQuery<PerformanceData>();
        List<PerformanceData> selectedData = new List<PerformanceData>();
        try
        {
            selectedData = query.Execute().ToList<PerformanceData>();
        }
        catch
        {
        }
        return selectedData;
    }
}

For this demonstration, I've created a Chart in WinForm that is populated with data retrieved from the diagnostic table.

The graph created displays the CPU utilization of specific role instance for the last 2 hours.

Points of Interest

Quest Software develops a very handy and easy to use tool called Spotlight on Azure. We provide an in depth monitoring capabilities of the entire Azure subscription starting from the role instance and above, data aggregation, historical display, alerting mechanism and custom user defined counters for deep analysis.