Deadlock Detection in Existing Code

Introduction

Deadlocks are common problems in multi-threaded programming. When it comes to multithreading development, the most common problem developers are facing is critical sections. It is not uncommon to use more than a single lock, but if one does not pay attention to the order of the locks, or to the context in which they are being called (e.g., from within a callback), deadlocks will form. (There are many reasons for deadlocks to occur other than the obvious critical section, e.g., two threads that are waiting for each other to signal an event, but we will not discuss them here).

As with anything that is related to threads, timing is everything. The most problematic deadlocks are those which occur rarely, they have this amazing nature of occurring at your client's site...

What if we could make the rare case the normal case? Recall that the reason a deadlock does not occur has to do with the fact that two threads that might deadlock happen not to be at the problematic places at the same time. So, all we have to do is record their "visits" at the problematic places, then we need to verify that the locks order is always the same, and if not, output the stack trace and notify the developer that we found a mismatch in the locks order.

The attached ZIP file contains a DLL that does exactly that. The DLL hacks all common (Enter, Exit, TryEnter methods, but it can be extended easily to support others, if used) monitor calls (including, of course, the .NET lock keyword) and keeps track of the locks order. Once it finds a problem in the order, it creates two stack traces, and directs you to a sample of the problematic locks (after fixing the error, repeat the test and see if there is another problem with the detected locks in another flow).

Note that there is no need for the deadlock to really occur; rather, it is only important that suspected flows (or all flows) will be performed at least once.

Using the code

Add the file incslock.cs to your project
Add a reference to the slockimp.dll
Compile your component and execute it

Analyzing the stacks (slockimp based)

Once a problem is detected, the console (if one exists) will output the last lock conflicts with the n^th of the stack
Two files are being created in the working directory: first_xxx.txt and now_yyy (xxx and yyy represent numbers)
Go to the "now" file – find the last lock (prior to the last four calls that are inner to the DLL)

This is the lock that when locked caused the problem

In order to find the other problematic lock, you can either:

Spot the n^th lock from the beginning of the stack (not counting locks that were recursively locked, and locks that were locked and released)
Find the last lock in the "first" file

Go over the "first" file stack, and find the place in which the lock from 3.a was locked

This is the second version of the implementation, which now supports more complex scenarios like the dining philosophers (thanks to Sergey's question below). The stack files are numbered as follows: 0_xxx.txt, 1_xxx.txt, and so on...

0_xxx points to the lock that when locked caused the problem. Other files point to other locks that created a kind of circular waiting.

Points of interest

Notice that you do not need to change a single existing line of code. Rather, you add two files to your project. The trick here is to cause the compiler to use our reference for implementing monitor calls rather then .NET's. (In the DLL itself, the locks are being locked and released properly, so your program should work fine critical-section wise). This trick is similar to replacing a header file in C.

Notice that the DLL is not meant for production, since it affects performance. Also, notice that the DLL allows recursive locks of the same lock. The DLL, however, will notify about possible deadlocks even if the waiting time is not infinite. (Despite the fact this is a false alarm, it indicates bad behavior, since such a behavior might influence performance and will deadlock if the time would be set to infinite).

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

eransha

Team Leader

Israel

Member

Article Top

Poor

Excellent

Add a reason or comment to your vote: x
Votes of 3 or less require a comment

Comments and Discussions

You must Sign In to use this message board. (secure sign-in)

Search this forum
	Profile popups Noise Layout Per page

slockimp.dll

ComplexityChaos

9:03 19 Nov '09

no stack trace files are being generated

PMD

4:48 17 Jun '09

Re: no stack trace files are being generated

9:43 17 Jun '09

11:01 1 Apr '09

9:46 17 Jun '09

Simple enhancement to imitate lock() statement

cwienands

17:45 15 Feb '09

Re: Simple enhancement to imitate lock() statement

eransha

8:53 23 Feb '09

what's the "trick"

jdstuart

22:09 25 Nov '08

Re: what's the "trick"

3:40 29 Nov '08

11:21 20 Dec '07

9:44 24 Dec '07

No, dining philosophers did not get any help [modified]

SAKryukov

6:42 20 Sep '07

Sorry, as I did suspect, you slockimp2 code does not detect indirect deadlock of dining philosophers, either. This is too bad, really, that I cannot see your source code, but -- frankly -- I simply don't know good criteria to detect this condition in any essentially universal way.

Please see my sample code below. I tried to make it as simple as I can. It can work as a console application just one source file.

First, I made sure I'm using your monitor, not the default System.Threading.Monitor, so I renamed it to Debug.Threading (included in my sample below). I also used slockimp.imp directly (2 lines below, commented out). (By the way your trick with replacement of the class using same namespace does not always work: in my VS2005 solution make issues a warning with notification saying the default System.Threading.Monitor was used, so I renamed the namespace.)

I used only two philosophers, which is enough to show the issue, and a simplest form of the Fork code. Anyway, the deadlock is clearly demonstrated, as well as the inability of your slockimp2 to give it any help.

Could you please analyze this with my sample and give us your ideas?
Thank you.

////////////// CODE SAMPLE

using System;

namespace Debug.Threading {
      using System.Threading;
      class Monitor {
            public static void Enter(object obj) {
                  slockimp.imp.DoLock(obj);
            } //Enter
            public static void Exit(object obj) {
                  slockimp.imp.DoUnlock(obj);
            } //Exit
            public static bool TryEnter(object obj) {
                  return slockimp.imp.DoTryEnter(obj);
            } //TryEnter
            public static bool TryEnter(object obj, int millisecondsTimeout) {
                  return slockimp.imp.DoTryEnter(obj, millisecondsTimeout);
            } //TryEnter
            public static bool TryEnter(object obj, TimeSpan timeout) {
                  return slockimp.imp.DoTryEnter(obj, timeout);
            } //TryEnter
      } //Monitor
} //namespace Debug.Threading

namespace DiningPhilosophers {
      using System.Text;
      using System.Threading;

      internal class Player {
            internal Player(string name) { this.Name = name; }
            internal void ShowNamedMessage(string format) {
                  Console.WriteLine(string.Format(format, this.Name));
            } //ShowNamedMessage
            string Name;
      } //Player

      internal class Fork : Player {
            internal Fork(string name) : base(name) {}
            internal void PickUp() {
                  Debug.Threading.Monitor.Enter(this);
                  //slockimp.imp.DoLock(this);
                  ShowNamedMessage("Fork {0} is taken");
                  Thread.Sleep(1000); //SA!!! this is done to ensure deadlock in a very first iteration
            } //PickUp
            internal void PutDown() {
                  Debug.Threading.Monitor.Exit(this);
                  //slockimp.imp.DoUnlock(this);
                  ShowNamedMessage("Fork {0} is put down");
            } //PutDown
      } //class Fork

      internal class Philosopher : Player {
            internal Philosopher(string name, Fork left, Fork right) : base(name) {
                  this.Left = left;
                  this.Right = right;
                  Thread = new Thread(delegate() {
                        while (true) {
                              Think();
                              Left.PickUp();
                              Right.PickUp();
                              Eat();
                              Left.PutDown();
                              Right.PutDown();
                        } //loop
                  });
            } //Philosopher
            internal void Start() { this.Thread.Start(); }
            void Eat() {
                  ShowNamedMessage("Philosopher {0} is eating");
            } //Eat
            void Think() {
                  ShowNamedMessage("Philosopher {0} is thinking");
            } //Think
            Thread Thread;
            Fork Left, Right;
      } //class Philosopher

      class Program {
            static void Main(string[] args) {
                  Console.WriteLine("Process started");
                  Fork fA = new Fork("A");
                  Fork fB = new Fork("B");
                  Philosopher pA = new Philosopher("A", fA, fB);
                  Philosopher pB = new Philosopher("B", fB, fA);
                  pA.Start(); pB.Start();
                  Console.ReadKey(true);
            } //Main
      } //class Program

} //namespace DiningPhilosophers

////////////// END OF CODE SAMPLE

Thank you very much.

Sergey A Kryukov

-- Modified Monday, July 4, 2011 3:49 PM