Yesterday I encountered a weird issue where my NServiceBus performance counters did not appear to be working correctly.
To put this situation into context, having developed an integration project over the last few months using NServiceBus, the time came to perform some load/stress testing on the various endpoints. I dutifully wrote a little command line app which simulated heavy load, dumping hundreds of thousands of messages into the queue and monitoring the host process as it worked its way through the backlog.
Everything worked wonderfully except for one thing, the Critical Time and SLA Violation Countdown performance counters did not budge from their default values, not a flicker. The perf counters were installed correctly as per the documentation and my host was loading the performance counters profile correctly, but yet no matter what I tried, the Critical Time counter was always zero and the SLA Violation Countdown counter was always at max value. After several hours of head scratching, fruitless Googling and studying of the NServiceBus source code, it dawned on me that I had made one crucial oversight…
While writing the stress test simulator app, I had deemed it “overkill” to bring in a dependency on NServiceBus itself, so I decided to write the messages to the queue using the MSMQ api.
XNamespace xmlns = @"http://tempuri.net/MyMessagesNamespace";
var body = new XDocument(
new XDeclaration("1.0", "utf-8", "yes"),
new XElement(xmlns + "Messages",
new XElement(xmlns + "MyCommand",
new XElement("MyCommandData", "foo")
)
)
);
var message = new Message(body, StringMessageFormatter.UTF8);
myMessageQueue.Send(message, MessageQueueTransactionType.Single);
Rather than using the XML serializer I replicated the XML message content manually. The StringMessageFormatter.UTF8 object is an IMessageFormatter implementation I wrote which puts the XML string directly into the message body without running another serializer on it - unbelievably this is not something that’s supported by the System.Messaging namespace by default.
When NServiceBus itself sends messages (via IBus.Send()), it adds some important meta-data to the messages extension header. One of those headers is "NServiceBus.TimeSent". NServiceBus uses this timestamp value to calculate the Critical Time, and therefore the SLA Violation Countdown performance counters. While NServiceBus was more than happy to consume my messages, because they were sent directly via MSMQ they did not include this timestamp in the header, thus the metrics were not calculated.
Resolving the problem was relatively simple. In addition to manually replicating the XML body of the message, I also needed to manually replicate the NServiceBus meta-data in the messages extension.
var extension = new XDocument(
new XDeclaration("1.0", "utf-8", "yes"),
new XElement("ArrayOfHeaderInfo",
new XAttribute(XNamespace.Xmlns + "xsd",
"http://www.w3.org/2001/XMLSchema"),
new XAttribute(XNamespace.Xmlns + "xsi",
"http://www.w3.org/2001/XmlSchema-instance"),
new XElement("HeaderInfo",
new XElement("Key", "NServiceBus.TimeSent"),
new XElement("Value",
DateTime.UtcNow.ToString("yyyy-MM-dd HH:mm:ss:ffffff Z"))
)
)
)
);
message.Extension = Encoding.UTF8.GetBytes(extension.ToString());
Upon re-running the stress test I was able to observe the Critical Time performance counter creep up as the tool bulk loaded messages into the queue, and shortly after I was able to see the SLA Violation Countdown counter start to come down as expected. Problem solved!
