Click here to Skip to main content
11,578,241 members (61,503 online)
Click here to Skip to main content

MSMQ Backed FIFO Queue

, 1 Dec 2010 CPOL 14.7K 384 7
Rate this:
Please Sign up or sign in to vote.
A standard in-memory generic .NET queue that, when the configurable overflow limit is reached, will start using MSMQ as a backend to mitigate the issues with memory usage.

Introduction

I recently ran into a problem where I needed to store a collection of objects in a queue for quick access. This turned out to be a problem when I started storing a large number of these objects, which themselves had a decent size, in memory. I was seeing OutOfMemoryExceptions and other odd behavior resulting from the overloading of my system's memory.

The OverFlowQueue<T> is a direct result of needing a FIFO queue that would not blow my system's memory away when flooded with objects. Using MSMQ as a back-end to the standard .NET generic queue successfully mitigates the memory issue with acceptable performance decreases.

That being said, I am sure that there are improvements I can make to this class, and I am welcoming all feedback!

Using the Code

The sample project included here has several classes in it, of which I am only going to discuss OverflowQueue<T>, but feel free to use the other classes which contain methods to create/modify MSMQ queues, retrieve typed app.config values, and other utility methods.

The OverflowQueue<T> class itself must be created using a serializable type. This is mandated by the binary formatter that is being used to pack and unpack items from MSMQ.

OverflowQueue<T> has one constructor that takes two parameters:

public OverflowQueue(string queueName, bool useBackgroundMsmqPull)
{...}

The first parameter, queueName, is the name of the MSMQ queue that should be created and used for the overflow from the in-memory queue.

The second parameter, useBackgroundMsmqPull, defines whether or not the operation of pulling from MSMQ to repopulate the in-memory queue is done on the calling thread (on the call to Dequeue()) or on a background thread that runs every second. For performance reasons, I have this value set to true so that the pull from MSMQ happens on a secondary thread and does not hold up my dequeue operation. The pull from MSMQ looks like this:

void PullFromMSMQ()
{
    // We've been putting all new messages into MSMQ... 
    // Now is the time to get them out and put them back into memory.
    while (Interlocked.Read(ref currentMSMQSize) > 0 
        && Interlocked.Read(ref currentQueueSize) < maxInternalQueueSize)
    {
        Message message = overflowMSMQ.Receive();

        // decrement the MSMQ size
        Interlocked.Decrement(ref currentMSMQSize);

        T item = message.Body as T;

        PushToMemoryQueue(item);
    }

    if (Interlocked.Read(ref currentMSMQSize) <= 0)
    {
        // lock to prevent incorrect count when turning off msmq pushing
        lock (msmqLock) 
        {
            if (Interlocked.Read(ref currentMSMQSize) <= 0)
                pushingToMSMQ = false;
        }
    }
}

So, you can see why I am using a background thread to pull from MSMQ... My application has a good amount of processing on the dequeue, so it works better for me to have the background thread pull from MSMQ, but this might not be the same for everyone.

The rest of the OverflowQueue<T> class is fairly easy to use; there are the following functions that allow you to enqueue or dequeue messages:

public void Enqueue(T item) {...}
public T Dequeue() {...} 

Both of these functions will transparently use the overflow MSMQ queue to enqueue or dequeue messages, which you can see if you download the source code.

Points of Interest

My goal in creating this class was to create a utility that transparently incorporated the MSMQ back-end for my queue. All functionality dealing with the creation and use of the underlying MSMQ is encapsulated in the OverflowQueue<T> class itself.

There are only two prerequisites to using this class:

  1. The user must have MSMQ installed
  2. The type <T> in OverflowQueue<T> must be serializable

History

This is the first revision of the OverflowQueue<T> class.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Aron Weiler
Technical Lead CareFusion
United States United States
I just looooove software.

Check out my technical blog here: The Fyslexic Duck. You can find most of what I've put on CodeProject there, plus some additional technical articles.

You may also be interested in...

Comments and Discussions

 
QuestionRemoteQueuePathUri Pin
Marc Leger12-Jul-11 12:15
memberMarc Leger12-Jul-11 12:15 
AnswerRe: RemoteQueuePathUri Pin
Aron Weiler12-Jul-11 13:04
memberAron Weiler12-Jul-11 13:04 
GeneralRe: RemoteQueuePathUri Pin
Marc Leger13-Jul-11 7:49
memberMarc Leger13-Jul-11 7:49 
GeneralRe: RemoteQueuePathUri Pin
Aron Weiler13-Jul-11 18:11
memberAron Weiler13-Jul-11 18:11 
Questionwant to read msmq message fast continously Please help me Pin
Amit kumar pathak28-Mar-11 2:45
memberAmit kumar pathak28-Mar-11 2:45 
AnswerRe: want to read msmq message fast continously Please help me Pin
Aron Weiler28-Mar-11 10:48
memberAron Weiler28-Mar-11 10:48 
GeneralFor this code and more... Pin
Aron Weiler25-Feb-11 7:01
memberAron Weiler25-Feb-11 7:01 
Generalnice Pin
Pranay Rana30-Dec-10 16:53
memberPranay Rana30-Dec-10 16:53 
GeneralNice integration of new and old technologies! Pin
xzz01951-Dec-10 20:25
memberxzz01951-Dec-10 20:25 
GeneralRe: Nice integration of new and old technologies! Pin
Aron Weiler3-Dec-10 11:01
memberAron Weiler3-Dec-10 11:01 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.150603.1 | Last Updated 1 Dec 2010
Article Copyright 2010 by Aron Weiler
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid