It is needless to say that Tracing and Instrumentation are a very important facet of any application. The term Instrumentation refers to the ability to monitor or measure the level of a product's performance and to diagnose errors. In terms of programming this may relate to code tracing, debugging techniques, performance counters, event logs and so on (MSDN). The .NET framework includes the
Trace and the
Debug classes which provide the above mentioned aspects of instrumentation and have been very valuable to application development. Unfortunately, these classes have various shortcomings, which does not make them viable for certain situations or types of applications. To tackle the issues of conventional instrumentation techniques, MS has come up with an comprehensive framework called the Enterprise Instrumentation Framework (let's call it EIF in short). In this article, I shall elaborate what EIF is, and how the shortcomings of the
Debug classes are overcome using the EIF.
Before going forward, I would like to recommend two must read articles regarding Trace and Debug classes and their drawbacks.
What is EIF ?
The Enterprise Instrumentation Framework provides a very simple, yet extensible framework for instrumenting applications. Most importantly, the EIF provides a "Unified API" which helps integrate with already existing frameworks like WMI, Event Log etc. Also, as the name suggests, this framework is suitable for enterprise scale applications. That said, this framework not only enables instrumenting applications on one machine, but even those, which are distributed across several machines.
Let us first understand what EIF comprises of, and then, we can do a comparative analysis with existing instrumentation techniques. The following section describes the basic elements of EIF:
Event sources represent sources from which events occur. Typically, these may be represented by a particular application itself or specific parts or components of an application. The EIF provides three types of Event Sources:
|SoftwareElement||This type is used to control instrumentation behavior at a granular level. What developers can do is, split the application into a number of sources such that event raising can be independently controlled.|
|Application||Events that are raised without specifying an event source use the Application event source. In this case, Enterprise Instrumentation internally defines a SoftwareElement event source to represent the instrumented application. This event source is given the reserved name of Application.|
|Request||According to me, Request tracing is the best feature provided by the EIF. Using Request tracing, we can instrument an application even if it spans process or machine boundaries. To achieve this, developers would create a request event source and specify the beginning and end points of the request in the application code, which causes Enterprise Instrumentation to automatically manage the context of that request. I would discuss more about this in the later sections.|
Event sinks basically govern the destination store for the event data. Enterprise Instrumentation provides three standard event sinks. You can use Standard event sinks or Custom Event sinks. The standard event sinks provided are WMIEventSink, TraceEventSink and the LogEventSink
|This event sink writes events to a to a Windows event trace log file if a trace session is active. This tracing mechanism is suitable for the higher-frequency eventing, which can generate hundreds or even thousands of events per second in a running application.|
|This event sink outputs the event to the Windows event log and is suitable for lower-frequency events such as errors, warnings, or high-level audits.|
This event sink outputs the event to Windows Management Instrumentation (WMI). This is the slowest of the standard eventing mechanisms on Windows 2000 systems, and therefore should be used primarily for infrequent or high-visibility events.
Custom Event Sinks
A developer can leverage preexisting log formats and message routing mechanisms, such as Microsoft Message Queue (MSMQ) to create a custom sink. A developer can use either of the following two methods to implement custom event sinks:
Event schema defines the set of events which an application can raise. Implementation-wise, the event schema represents a set of classes that the EIF provides. Again, there are two event schemas: Standard and Internal. Standard events are those which a developer explicitly raises and include events like errors, audits, administrative events, and diagnostic trace events. Internal events are fired automatically by the EIF itself (mainly related to Request Tracing).
We can also extend the standard event schema by creating events of our own. The standard event classes provide certain base classes which can be inherited to create custom event classes.
Given below are some simple examples of using
Event classes to raise events. The event schema provides an exhaustive set of events and methods which one can use. There is no point detailing all these here, the EIF documentation is more than apt for that.
TraceMessageEvent.Raise("A simple trace event")
AuditMessageEvent.Raise("A simple audit event")
We have seen what Event Source, Event Sink and Event Schema means in the previous section. In this section, we shall see how we relate these in an application through configuration. Each application should maintain a configuration file for storing EIF settings and this, by default is called EnterpriseInstrumentation.config. This file can be created programmatically using the Configuration API or by manually editing it. Given below are the typical configuration sections:
In this section, we document what types of event classes are grouped into named categories. This helps to logically group certain event classes in an application. For example, creating a category by name All Events for the type
System.Object would include all events (since all classes are implicitly derived from
System.Object) in that category. Here's an example:
<eventCategory name="All Events"
description="A category that contains all events.">
<event type="System.Object" />
This section documents all event sinks used by the application and their corresponding parameters. The example shown below declares a
TraceEventSink by name traceSink and session name has been configured as
description="Outputs events to the Windows Event Trace."
<parameter name="sessionName" value="TraceSession" />
This section lists all the Event sources in the application. The example given below documents a request event source by name
<eventSource name="MyRequest" type="request"
description="A request event source." />
This section binds a
Event Category to one or more
Event Sinks. The configuration shown below associates All Events event category to the
TraceEventSink and the
LogEventSinks respectively. That is, any event raised will automatically be logged to the Windows Event trace log and the Event log.
description="A default filter for the Software Element event sources.">
<eventCategoryRef name="All Events">
This section binds an event source to a filter. In the example given below, the Application event source is bound to the
<filterRef name="defaultSoftwareElementFilter" />
Here's the big picture: An application creates
Event sources, maps event classes to categories and uses one or more
Event sinks. Using filters, an application is configured to route certain categories of events to certain event sinks and Event sources are mapped to these filters using filter bindings. The figure shown below roughly illustrates these relationships.
The application uses classes of the event schema to raise certain events and based on the configuration, the event data would find its way into one or more event sinks. Event sinks can finally be queried for data, which can be used to analyze the application performance or behavior.
EIF to the rescue
This section has to be the most interesting one of this article. Here, I shall discuss various shortcomings of conventional instrumentation techniques (Trace and Debug classes) and how EIF aids in overcoming those. I shall also enunciate some of the features provided by EIF, which I found very interesting and useful.
Firstly, the Trace and Debug classes are not inheritable. Therefore, to implement a custom trace functionality, we need to adopt containment as a technique. Here, by containment I mean, wrap the Trace class in your class implementation and add your functionality over it. This implementation can be seen in Marc's article. The EIF event schema, on the other hand is completely extensible. The framework provides a lot of Base classes which can easily be extended. For example, some of the provided base event classes are
TraceEvent and so on (The
BaseEvent sits at the top of the hierarchy).
The second problem with the trace class is that, you cannot associate different trace messages with different trace listeners. For example, (as mentioned in Vagil's article) you cannot assign different severity thresholds to trace listeners. So, the trace output is same for all listeners. In EIF, you can categorize different event classes into categories and define filters to determine which event sink the messages are routed to. For example, let's consider the
ExternalErrorMessageEvent classes to be categorized into a category called Errors and we can have these events route to the Event log. On the other hand, we can have
AuditEvent classes categorized into a category, say Audit, which can be configured to route events to the Trace log. Therefore, by configuring appropriate filters, we can bifurcate different event data to go to different sinks, something that the trace class and trace listeners definitely lacked.
The third and major drawback with
Trace and related classes is the difficulty/inability to do distributed request tracing. By distributed request tracing, I mean the ability to trace an application that may span several processes or several machine boundaries for that matter. If you recollect, the TraceListener collection added (through the Trace.Listeners.Add method) to one process/appdomain is not available directly to another process/appdomain. Therefore, by default, the trace context does not flow from one process to another. To achieve some amount of correlation between trace messages written in different processes, we might end up writing custom trace listeners that write trace output to common stores like SQL Server or MSMQ and use Trace categories to correlate trace messages. Even this method is not completely fool proof in itself. Probably, we may have to use a complex remoting infrastructure for this. Bottom line is, we don't have distributed request tracing "out of the box". Enter EIF, this feature is inherently provided and it is a remoting infrastructure that is used; and the request information is stored in the
LogicalCallContext class, which flows along the path of execution, across process and machine boundaries.
Thus far, we saw some features of EIF that were a solution to shortcomings of the Trace class. Now, I shall take you through some features which I think are really cool and helpful. Foremost among them, are certain information fields populated by the event classes. Usually, when we write trace statements in code, we inevitably, end up writing code to capture some information like thread identity, operation execution time, ProcessID etc. The EIF event classes provide a wealth of such information without any coding effort from our side. Each event class provides certain useful fields which are grouped into named property groups. Certain properties are populated by default. For example, a group called
PopulateRequestInfo is populated by all events by default. This group includes information like
RequestSequenceNumber ( which can be used to track the order of requests),
RootRequestInstance (to track the root request) and many more. Certain property groups, on the other hand, are not populated by default. An example for the same is
PopulateComPlusInfo, which contains COM+ related information like
TransactionID and so on. EIF provides us an extra level of control by providing us an option to decide which property groups are populated and which are not, for a particular event source. In the configuration snippet given below,
PopulateWindowsSecurityInfo is enabled for the SecureRequestTrace event source.
<eventSource name="SecureRequestTrace" type="request"
<eventSourceParameter name="PopulateWindowsSecurityInfo" value="true" />
Another good feature is that of the project installer. Immediately following the imports directives, add the following new empty installer class that inherits from
ProjectInstaller. Define it with the
RunInstaller attribute set to true.
public class MyProjectInstaller
After compiling the application, run the installutil.exe on the application executable. Voila! You would see that the EnterpriseInstrumentaion.config is automatically created with all configuration information already documented.
Finally, I was really impressed by the Trace Session Manager. The trace session manager runs as a windows service and is primarily responsible for maintaining trace sessions. Each trace session includes a trace log file and all the event classes which have the event sink configured as the trace log sink end up writing trace data to this log file. (To facilitate this, we just have to make sure that the sessionName parameter of the of the trace log event sink in the EnterpriseInstrumentation.Config matches the name of the configured trace session in the TraceSessions.config). I have already mentioned one advantage of trace logs: they are suitable for high frequency eventing. The other is more of a personal preference. I really prefer analyzing trace data from a trace log file using
TraceLogReader API over analyzing the same from an event log. The latter, I thought, requires more plumbing).
The EIF is definitely a great framework for instrumentation, especially for enterprise scale applications and is a definitive improvement over the Trace and Debug classes. The EIF provides easy to use API and also seamlessly integrates with VS.NET. Applications instrumented using EIF are easy to configure and also do not incur much performance degradation as a result of instrumenting the code. Therefore, I would like to conclude, asserting that, there is more than one compelling reason for a development team to adopt EIF as their chief instrumentation strategy in a .NET application.
Unfortunately, in my opinion, the EIF, though being a rich framework is not so well advertised by MS. EIF is only available through MSDN Universal subscriber downloads. The EIF documentation provided with the EIF framework is one of the very few good resources I came across. The other resource is the MSDN TV on EIF and is available here.
Even to this day, I feel EIF still remains to be an unearthed treasure in the MS technology stack!