Click here to Skip to main content
13,091,363 members (55,089 online)
Click here to Skip to main content
Add your own
alternative version

Stats

160.2K views
1K downloads
39 bookmarked
Posted 29 Nov 2002

A Fast Mostly User Mode Inter-Process Mutex

, 26 Feb 2003
Rate this:
Please Sign up or sign in to vote.
Win32 Mutexes are slow and cumbersome, this is a fast and nifty alternative!

Results from test program using our custom mutex class

Introduction

I recently was working on a project where several resources needed to be shared among many processes. Easy! I just made use of a Win32 Mutex. This seemed fine at first, but after running test scenarios where the various processes could be locking and releasing the mutex hundreds of thousands of times simultaneously, performance became an issue. I attributed the poor performance to the Win32 mutex (for reasons I get into soon) and thus decided to write my own mutex class. This article describes the mutex class I made (actually, it's a DLL, which is a requirement, but I'll get into that later), how it works, and how it stacks up against the Win32 mutex.

The Problem with Win32 Mutexes

Win32 Mutexes are notoriously slow. Every time a mutex is created the OS must switch from "user mode" to "kernel mode", which in itself can be responsible for hundreds of CPU instructions. If you're locking a shared resource every so often, it's really not a big deal. In that case (especially if there is a network connection involved), the rest of your code is probably using much more time. However if you are constantly hitting the resource as your main information source (such as my proprietary database file example above), all this extra work Win32 mutexes have to do can be a real time hog.

The Solution

I solved the problem by writing my own mutex. It uses a combination of inter-process shared memory, Win32 critical sections (which only switch to "kernel mode" where there is a contention), and the Windows <CODE lang=c++>InterlockedExchange function. The kind of inter-process shared memory I'm using is the kind where you create a data segment in a binary (executable or DLL), then flag it as shared memory. The only memory I share in this data segment are two small variables (a <CODE lang=c++>LONG and <CODE lang=c++>DWORD). I rejected using memory mapped files and other forms of inter-process communication because I didn't want the performance hit I would incur by using those methods. Choosing the shared memory data segment option comes with a price: a DLL must be created to host our mutex (or you can incorporate it into an existing DLL if you don't mind such a thing). Once you've created a DLL project, statements such as those listed below need to be added to one of the files (preferably a .cpp) in the project: <PRE lang=c++>#pragma data_seg(".USRMUTEX_SHARED") volatile DWORD g_dwDSMutexOwner = 0; volatile LONG g_lLocker = 0; #pragma data_seg() #pragma comment(linker, "/section:.USRMUTEX_SHARED,rws")

The mutex object will be implemented as a C++ class I'll refer to as "the mutex class". The class and it's methods will be exports of the DLL. The mutex class essentially has only a constructor and destructor for its methods along with three member variables, all three of which are static. The first static member variable is an <CODE lang=c++>unsigned long whose purpose is to keep track of the number of recurring times that the same thread of the same process gained exclusive access to the mutex (another way of saying "exclusive access" would be "a lock"; what I meant by "recurring" is "locked a second or third time or so on before releasing all of it's locks"). The second static member variable holds the ID of the process that created the instance of our mutex class we're manipulating. The third static member variable is a Win32 critical section. We initialize our first two static member variables to zero and our third using <CODE lang=c++>InitializeCriticalSection.

Now that we've taken care of our static member variables, lets get our class prototype mocked up. Remember that we've only got a default constructor and destructor. Before you get started with the implementation, it might be a good idea if you make the default copy constructor and default copy operator of the class unreachable (put the method prototypes for those methods in a "private" section of the class but don't actually implement them). Now that we've got the class prototype, lets implement the methods. The first bit of work we want to do in the constructor is ask if we've ever set the process ID of the current process into the second static member variable we mentioned above. If that second static member variable is still set to 0, then we haven't, so get the current process ID using <CODE lang=c++>GetCurrentProcessId and set it into this variable. Now lock our critical section static member variable using <CODE lang=c++>EnterCriticalSection. By doing this we're doing two things:

  • We're ensuring that the current thread in the current process will be the only thread available to gain exclusive access to the mutex.
  • By entering the critical section we've already got half of the way to gaining exclusive access to the mutex.
This is considered "half-way" because in order to gain access to some synchronization object on an inter-process level, you must know two things:
  • which process owns it and
  • which thread in that process owns it.
By entering the critical section, we have already been able to answer the second part of that question. Now we've got to answer that first part. The first thing we have to do to figure that out is ensure that we have exclusive access to the shared memory variables. One of the shared memory variables exists for this very purpose (<CODE lang=c++>g_lLocker). Using that variable, we create a loop that is only broken when an the result of an <CODE lang=c++>InterlockedExchange call given that variable (<CODE lang=c++>&g_lLocker) and 1 is 0. Here is an example of how the code might look: <PRE lang=c++> while (::InterlockedExchange(&g_lLocker, 1) != 0) { Sleep(0); }

This says: "If I set that variable to 1 but the value of that variable was 0 before I set it, then the shared memory is safe for me to use. By setting it to 1, I'm letting others know they need to "spin" (loop until the condition is true) until I'm done with it".

The trick in this is we must make sure we set that shared memory variable back to 0 when we're finished so spinning threads can gain excusive use. So, when finished we set it back to 0 using the <CODE lang=c++>InterlockedExchange function as demonstrated below:

<PRE lang=c++> ::InterlockedExchange(&g_lLocker, 0);

Once we have exclusive access to the shared memory, we compare the other shared memory variable (<CODE lang=c++>g_dwDSMutexOwner) against both 0 and that static memory variable that holds our process ID. If that shared <CODE lang=c++>DWORD equals either of those values then we copy the current process ID from that static member variable into our shared <CODE lang=c++>DWORD. Regardless if the comparison passes or fails, we need to set the <CODE lang=c++>g_lLocker variable back to 0 (using <CODE lang=c++>InterlockedExchange as described above) to give other anxious threads a chance to compare. Once the compare passes and the resulting copy occurs, the mutex will then be considered "locked" by the calling thread of that calling thread's process. At that time all we've got left to do is increment the static member variable that keeps track of the number of times the same thread in the same process has locked the mutex. If that initial comparison had failed above instead of passing (that shared <CODE lang=c++>DWORD was not 0 or the current process id), then we call <CODE lang=c++>Sleep(1) here (even though using 0 gives up the rest of the current threads time slice, I've learned that using something other that 0 plays nicer). Next we test the comparison again by looping back to the point where we get exclusive access to the shared memory using the <CODE lang=c++>g_lLocker variable, but not as far back as where we entered the critical section (at this time we still haven't left the critical section). Keep this loop going forever until we get a passing comparison. The very fact that the comparison fails means that some other thread either in the same or in another process already owns the mutex.

The destructor is actually much more simple. First we decrement that static member variable that keeps track of the number of times the same thread in the same process has locked the mutex. Next we check to see if that variable is zero; if so we need to get exclusive access to the shared memory variables (using g_lLocker) employing the same method described above in the constructor. Once we have exclusive access, set our shared <CODE lang=c++>DWORD (<CODE lang=c++>g_dwDSMutexOwner) to 0. This completely unlocks the current process's lock on the mutex and allows other threads to then gain access. Now release our exclusive access to the shared memory (setting <CODE lang=c++>g_lLocker to 0 using <CODE lang=c++>InterlockedExchange as described above). Lastly, regardless of whether or not our first static member variable is now zero, leave the critical section we entered in the constructor (and stayed entered). That last step allows other threads in the current process to attempt to gain exclusive access to the mutex.

The CUsrMutex Class

Here is the class definition for the class we've just created (I've called it <CODE lang=c++>CUsrMutex). The full class implementation is available in the zip file attached to this article.

<PRE lang=c++>//This is a class we made for CUsrMutex to consume, it makes the win32 // critical sections easier to use class CCritSec { protected: CCritSec(const CCritSec& src); const CCritSec& operator=(const CCritSec& src); public: CCritSec(); virtual ~CCritSec(); operator LPCRITICAL_SECTION(); bool Lock(); bool Unlock(); protected: mutable CRITICAL_SECTION m_sect; }; //This class makes for easy locking of our g_lLocker shared memory variable class CUsrMutexLockerWkr { private: CUsrMutexLockerWkr(const CUsrMutexLockerWkr& src); const CUsrMutexLockerWkr& operator=(const CUsrMutexLockerWkr& src); public: CUsrMutexLockerWkr(); ~CUsrMutexLockerWkr(); }; class CUsrMutex { protected: CUsrMutex(const CUsrMutex& src); const CUsrMutex& operator=(const CUsrMutex& src); public: CUsrMutex(); ~CUsrMutex(); protected: static unsigned long m_ulLockCnt; static DWORD m_dwProcessId; static CCritSec m_dscs; };

Here is the implementation of these three classes:

<PRE lang=c++>#pragma data_seg(".USRMUTEX_SHARED") volatile DWORD g_dwDSMutexOwner = 0; volatile LONG g_lLocker = 0; #pragma data_seg() #pragma comment(linker, "/section:.USRMUTEX_SHARED,rws") unsigned long CUsrMutex::m_ulLockCnt(0); DWORD CUsrMutex::m_dwProcessId(0); CCritSec CUsrMutex::m_dscs; CCritSec::CCritSec() : m_sect() { ::InitializeCriticalSection(&m_sect); } CCritSec::~CCritSec() { ::DeleteCriticalSection(&m_sect); } CCritSec::operator LPCRITICAL_SECTION() { return (LPCRITICAL_SECTION)&m_sect; } bool CCritSec::Lock() { ::EnterCriticalSection(&m_sect); return TRUE; } bool CCritSec::Unlock() { ::LeaveCriticalSection(&m_sect); return TRUE; } CUsrMutexLockerWkr::CUsrMutexLockerWkr() { while (::InterlockedExchange(&g_lLocker, 1) != 0) { Sleep(0); } } CUsrMutexLockerWkr::~CUsrMutexLockerWkr() { ::InterlockedExchange(&g_lLocker, 0); } CUsrMutex::CUsrMutex() { if (m_dwProcessId == 0) { assert(sizeof(LONG) == sizeof(DWORD)); m_dwProcessId = ::GetCurrentProcessId(); assert(m_dwProcessId != 0); } m_dscs.Lock(); for (;;) { { CUsrMutexLockerWkr locker; if ((g_dwDSMutexOwner == 0) || (g_dwDSMutexOwner == m_dwProcessId)) { g_dwDSMutexOwner = m_dwProcessId; break; } } Sleep(1); } ++m_ulLockCnt; } CUsrMutex::~CUsrMutex() { if (--m_ulLockCnt == 0) { CUsrMutexLockerWkr locker; assert(g_dwDSMutexOwner == m_dwProcessId); g_dwDSMutexOwner = 0; } m_dscs.Unlock(); }

Proving Performance

Once I got the class I described above implemented, I wanted to test it to see if it was really faster than win32 mutex. I ran lots of tests, and my class always had the edge. The test where it really shined is where 6 separate processes were competing for the mutex. These 6 processes first attempted to lock and unlock a Win32 mutex 100,000 times, then attempted to lock and unlock my mutex 100,000 times. The total times were similar for each process, each something like the following numbers:

Win32 mutex: Took 6800 milliseconds to lock and unlock the mutex
My mutex: Took 240 milliseconds to lock and unlock the mutex

Along with this article a zip archive is included that includes my implementation of the mutex, a sample application, and several performance and comparison test applications.<?p>

Limitations

This design has two major limitations: 1) It's required to live in a DLL (I suppose you could put it directly into an executable if that met your specific needs), and 2) You only get one mutex you can share. The second limitation is only a limitation of my specific implementation. You can make changes in your implementation so you would have access to more than one of these mutexes.

Conclusion

This solution met my needs and vastly improved the performance of our product. Although this implementation of a custom mutex class might not meet your needs, I imagine (and hope) that this has been a good resource to help you get started in building your own custom synchronization objects. Good luck!

History

11/23/2002 - Initial revision.
02/18/2003 - Fixed possibility of race condition by adding g_lLocker.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

Olan Patrick Barnes
Web Developer
United States United States
Olan Patrick Barnes is a Senior Software Engineer at Open Text Corporation. He is currently working towards a Masters of Computer Science at Eastern Michigan University.

You may also be interested in...

Comments and Discussions

 
QuestionCute, but not practicle Pin
TheToeJoe5-Oct-11 2:32
memberTheToeJoe5-Oct-11 2:32 
GeneralSimilar solution from Microsoft Pin
VVVovka29-Sep-06 1:54
memberVVVovka29-Sep-06 1:54 
GeneralSleep(0) or Sleep(1) Pin
studio_ukc28-Sep-06 8:43
memberstudio_ukc28-Sep-06 8:43 
QuestionReal-time App NOT using a "Spinlock" Pin
cmenge4-Sep-06 14:25
membercmenge4-Sep-06 14:25 
AnswerRe: Real-time App NOT using a "Spinlock" Pin
studio_ukc28-Sep-06 8:40
memberstudio_ukc28-Sep-06 8:40 
GeneralLimitations of CriticalSection with terminate thread Pin
john56655410-Feb-06 9:06
memberjohn56655410-Feb-06 9:06 
GeneralThis Mutex Can Not be used under multi-session Pin
gyb99831-Aug-05 22:48
membergyb99831-Aug-05 22:48 
GeneralHook EnumProcesses From PSAPI.DLL Pin
Anonymous30-Jun-05 0:56
sussAnonymous30-Jun-05 0:56 
GeneralSounds a bit like Metered Section Pin
staceyw4-Jun-04 12:47
memberstaceyw4-Jun-04 12:47 
GeneralI appreciate u auther Pin
dharani3-Jun-04 1:08
memberdharani3-Jun-04 1:08 
GeneralSleep(0) won't allow to run lower priority thread Pin
ladislav Hruska1-Mar-03 16:25
memberladislav Hruska1-Mar-03 16:25 
GeneralRe: Sleep(0) won't allow to run lower priority thread Pin
Olan Patrick Barnes2-Mar-03 14:47
memberOlan Patrick Barnes2-Mar-03 14:47 
Generalsenceless improvement on SMP or hyperthread systems Pin
Heiko Rabe1-Mar-03 9:15
sussHeiko Rabe1-Mar-03 9:15 
GeneralRe: senceless improvement on SMP or hyperthread systems Pin
Olan Patrick Barnes2-Mar-03 14:51
memberOlan Patrick Barnes2-Mar-03 14:51 
GeneralRead the MSDN sample Pin
Dudi Avramov27-Feb-03 4:39
memberDudi Avramov27-Feb-03 4:39 
GeneralPoor coding = unfair test Pin
MrBern25-Feb-03 10:31
sussMrBern25-Feb-03 10:31 
GeneralRe: Poor coding = unfair test Pin
Olan Patrick Barnes2-Mar-03 14:52
memberOlan Patrick Barnes2-Mar-03 14:52 
GeneralArticle updated Pin
Olan Patrick Barnes19-Feb-03 6:48
memberOlan Patrick Barnes19-Feb-03 6:48 
GeneralNot mutex and not safe Pin
Felix Gartsman1-Dec-02 9:06
memberFelix Gartsman1-Dec-02 9:06 
GeneralRe: Not mutex and not safe Pin
Olan Patrick Barnes1-Dec-02 18:18
memberOlan Patrick Barnes1-Dec-02 18:18 
GeneralRe: Not mutex and not safe Pin
Felix Gartsman2-Dec-02 4:52
memberFelix Gartsman2-Dec-02 4:52 
GeneralRe: Not mutex and not safe Pin
Olan Patrick Barnes2-Dec-02 5:27
memberOlan Patrick Barnes2-Dec-02 5:27 
GeneralRe: Not mutex and not safe Pin
Felix Gartsman2-Dec-02 8:13
memberFelix Gartsman2-Dec-02 8:13 
GeneralRe: Not mutex and not safe Pin
Olan Patrick Barnes2-Dec-02 10:48
memberOlan Patrick Barnes2-Dec-02 10:48 
GeneralSleep(1) = Sleep(ALOT MORE) Pin
Anonymous30-Dec-02 14:38
sussAnonymous30-Dec-02 14:38 
GeneralRe: Not mutex and not safe Pin
Anonymous2-Dec-02 7:59
sussAnonymous2-Dec-02 7:59 
GeneralRe: Not mutex and not safe Pin
Olan Patrick Barnes2-Dec-02 10:42
memberOlan Patrick Barnes2-Dec-02 10:42 
GeneralYou need.. Pin
Jörgen Sigvardsson30-Nov-02 13:09
memberJörgen Sigvardsson30-Nov-02 13:09 
GeneralRe: You need.. Pin
Olan Patrick Barnes1-Dec-02 18:28
memberOlan Patrick Barnes1-Dec-02 18:28 
GeneralCOptex Pin
Tim Smith30-Nov-02 7:17
memberTim Smith30-Nov-02 7:17 
GeneralRe: COptex Pin
Thomas Freudenberg1-Dec-02 9:55
memberThomas Freudenberg1-Dec-02 9:55 
GeneralRe: COptex Pin
Olan Patrick Barnes1-Dec-02 18:33
memberOlan Patrick Barnes1-Dec-02 18:33 
GeneralRe: COptex Pin
Tim Smith19-Feb-03 14:12
memberTim Smith19-Feb-03 14:12 
GeneralRe: COptex Pin
Olan Patrick Barnes20-Feb-03 4:52
memberOlan Patrick Barnes20-Feb-03 4:52 
GeneralRe: COptex Pin
Olan Patrick Barnes1-Dec-02 18:31
memberOlan Patrick Barnes1-Dec-02 18:31 
GeneralRe: COptex Pin
Olan Patrick Barnes1-Dec-02 18:49
memberOlan Patrick Barnes1-Dec-02 18:49 
GeneralRe: COptex Pin
Tim Smith19-Feb-03 14:06
memberTim Smith19-Feb-03 14:06 
GeneralRe: COptex Pin
Olan Patrick Barnes20-Feb-03 4:56
memberOlan Patrick Barnes20-Feb-03 4:56 
GeneralRe: Assert fired and cause for concern. Pin
Paolo Messina30-Nov-02 7:26
memberPaolo Messina30-Nov-02 7:26 
GeneralRe: Assert fired and cause for concern. Pin
Olan Patrick Barnes1-Dec-02 18:38
memberOlan Patrick Barnes1-Dec-02 18:38 
GeneralRe: Assert fired and cause for concern. Pin
Olan Patrick Barnes1-Dec-02 18:40
memberOlan Patrick Barnes1-Dec-02 18:40 
GeneralExcellent! Pin
Marc Clifton30-Nov-02 4:48
memberMarc Clifton30-Nov-02 4:48 
GeneralRe: Excellent! Pin
Olan Patrick Barnes1-Dec-02 18:42
memberOlan Patrick Barnes1-Dec-02 18:42 
GeneralRe: Excellent! Pin
Marc Clifton2-Dec-02 0:21
memberMarc Clifton2-Dec-02 0:21 
GeneralRe: Excellent!?????? Pin
Anonymous2-Dec-02 7:53
sussAnonymous2-Dec-02 7:53 
GeneralRe: Excellent!?????? Pin
Olan Patrick Barnes2-Dec-02 10:41
memberOlan Patrick Barnes2-Dec-02 10:41 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.170813.1 | Last Updated 27 Feb 2003
Article Copyright 2002 by Olan Patrick Barnes
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid