SAFMQ Store and Forward Message Queue

mattbattey

4.74/5 (12 votes)

Jan 16, 2006

4 min read

85561

1846

An OpenSource cross-compilable/cross-platform message queue server like MSMQ or MQSeries.

Introduction

The SAFMQ server provides asynchronous messaging. Message Publishers send or enqueue a message with the SAFMQ server. The SAFMQ server stores that message until the point at which the message can be forwarded on to the client. Message Publishers are assured that the messages are delivered. That's how SAFMQ got its name.

Timely Messaging

SAFMQ provides the ability to perform timely message delivery. So, if a Message Publisher wants a message to be read by a Message Subscriber in a certain amount of time, or not read at all, then the Message Publisher can prescribe a Time-To-Live for the message it publishes. SAFMQ will notify the Message Publisher about messages which have outlived their Time-To-Live, or a Message Publisher can choose to ignore the event.

Round-Trip, PsudoSynchronous Messaging

Round-Trip, or PsudoSynchronous Messaging is when a Message Publisher acts as a Message Subscriber after sending a "query" message. A Message Publisher may want to receive information back from the Message Subscriber. Thus, after the first message is sent by a Message Publisher and is received by a Message Subscriber, the original Message Publisher and Message Subscriber switch rolls.

SAFMQ provides a special messaging context element for Round-Trip/PsudoSynchronous Messaging. It's called a Receipt ID. Whenever a message is enqueued in a SAFMQ server, it is given a Universally Unique Identifier or UUID for short. When a round-trip message event is taking place, the original Message Subscriber publishes a message with a Receipt ID identical to the Message ID assigned to the message sent by the original Message Publisher. Then the original Message Publisher (now a subscriber) waits for a message with a Receipt ID equal to the Message ID the original Message Publisher sent.

Batch Processing

Not every task is best handled real-time. Sometimes, there are real money benefits to sending transactions to a trading partner in a large group or batch. SAFMQ can be an intermediary between real-time systems and a back-end batch processor. The real-time system knows that the messages will be delivered, and the batch processor can let data queue up until it is ready to send all the data. The batch system can even respond via SAFMQ, and a real-time system can instantly see the results.

Background

SAFMQ started as an in-house project at the Prescription Benefit Manager (PBM) I currently work for. In our line of business, we receive prescription drug claims from pharmacies, and are given approximately 10 seconds or less to process the claim, determine the price of the drug being purchased, determine the discount the cardholder's group has at that pharmacy, determine the patient's copy, how much funds he or she has left in his or her account, and send all that information back to the pharmacy, all while the customer is waiting at the counter. It's all a little like processing credit cards for purchases. That being said, we need a way to look at claims as they happen, real-time, but the database the production claim adjudicator is using can't be accessed without the chance of slowing down the claim processing. To fix the problem, we built SAFMQ and publish claims (in XML format) to the message queue real-time, and another system reads the claim details and writes them to a data warehouse type database, enabling our customer service agents the ability to lookup claim results real-time.

Using the code

Here's an example showing the number of messages that can be enqueued with the server per second:

/*
This demo program creates a queue named 
"test", writes 10000 messages,
then determines how ling it took to write 
the messages to the server and outputs the results.
*/

#include <windows.h>
#include <iostream>
#include "lib/MessageQueue.h"
#include "lib/MQFactory.h"
#include "lib/MQConnection.h"

using namespace safmq;
using namespace std;

#define MCOUNT 10000
int main(int argc, char* argv[]) {
    try {
        ErrorCode     ec;
        // Build a connection to the server
        MQConnection* con = 
          MQFactory::BuildConnection("safmq://localhost","admin","");
        // remove the queue just in case it was already there
        con->DeleteQueue("test");
        // create the queue named "test"
        con->CreateQueue("test");

        // Open an instance of the queue
        MessageQueue *queue = new MessageQueue(con,"test",false);

        // Get the start time for our process
        DWORD start = GetTickCount();

        for (int x=0; x<MCOUNT; x++) { {
            QueueMessage msg;
            // Set up the message
            msg.setLabel("Test Message");
            msg.setBodyType(BT_TEXT);
            *msg.getBufferStream() << "Test test test" << endl;

            // Sed the message to the server
            ec = queue->Enqueue(msg);
            if (ec != EC_NOERROR) {
               cout << "Error Code: " << ec << endl;
               break;
            }
            // output some status dots
            if (x%100 == 0)
                putchar('.');
        } }
        putchar('\n');

        // Get the end time
        DWORD end = GetTickCount();

        con->DeleteQueue("test");
        delete queue;
        delete con;

        // output the stats on our run.
        double duration = (end-start)/1000.0;
        cout << "Durration: " << duration << endl;
        cout << "Rate:style='mso-spacerun:yes'>      " 
             << (double)MCOUNT / duration 
             << " Messages per second." << endl;

    } catch (ErrorCode e) {
        cout << "Error Code: " << e << endl;
    } catch (exception& e) {
        cout << e.what() << endl;
    }
    return 0;
}

Points of Interest

Some of the things I tried to use when designing the SAFMQ client API was using the I/O stream pattern. If you dig around in the code, you'll find a directory containing classes to implement a std::basic_iostream derived class that wraps socket communication. You might find, there is some repetition between the code written for SAFMQ and MFC. That's because the code is actually cross-compilable, and can be compiled on Linux and other UNIX based systems.

Additionally, there're classes for performing SSL communication either directly or via the I/O stream pattern. This way, you can chain together the type of underlying transport mechanism, and use class construction to determine whether you use SSL or not. You'll see this in action in the MQFactory class, which is part of the client API.

History

SAFMQ's home page is located at SourceForge, you'll find the documentation for the API setting up the server, and the Java interface, there.