Click here to Skip to main content
14,031,046 members
Click here to Skip to main content
Add your own
alternative version

Stats

72.2K views
1.8K downloads
32 bookmarked
Posted 16 Jan 2006
Licenced

SAFMQ Store and Forward Message Queue

, 16 Jan 2006
Rate this:
Please Sign up or sign in to vote.
An OpenSource cross-compilable/cross-platform message queue server like MSMQ or MQSeries.

Introduction

The SAFMQ server provides asynchronous messaging. Message Publishers send or enqueue a message with the SAFMQ server. The SAFMQ server stores that message until the point at which the message can be forwarded on to the client. Message Publishers are assured that the messages are delivered. That's how SAFMQ got its name.

Timely Messaging

SAFMQ provides the ability to perform timely message delivery. So, if a Message Publisher wants a message to be read by a Message Subscriber in a certain amount of time, or not read at all, then the Message Publisher can prescribe a Time-To-Live for the message it publishes. SAFMQ will notify the Message Publisher about messages which have outlived their Time-To-Live, or a Message Publisher can choose to ignore the event.

Round-Trip, PsudoSynchronous Messaging

Round-Trip, or PsudoSynchronous Messaging is when a Message Publisher acts as a Message Subscriber after sending a "query" message. A Message Publisher may want to receive information back from the Message Subscriber. Thus, after the first message is sent by a Message Publisher and is received by a Message Subscriber, the original Message Publisher and Message Subscriber switch rolls.

SAFMQ provides a special messaging context element for Round-Trip/PsudoSynchronous Messaging. It's called a Receipt ID. Whenever a message is enqueued in a SAFMQ server, it is given a Universally Unique Identifier or UUID for short. When a round-trip message event is taking place, the original Message Subscriber publishes a message with a Receipt ID identical to the Message ID assigned to the message sent by the original Message Publisher. Then the original Message Publisher (now a subscriber) waits for a message with a Receipt ID equal to the Message ID the original Message Publisher sent.

Batch Processing

Not every task is best handled real-time. Sometimes, there are real money benefits to sending transactions to a trading partner in a large group or batch. SAFMQ can be an intermediary between real-time systems and a back-end batch processor. The real-time system knows that the messages will be delivered, and the batch processor can let data queue up until it is ready to send all the data. The batch system can even respond via SAFMQ, and a real-time system can instantly see the results.

Background

SAFMQ started as an in-house project at the Prescription Benefit Manager (PBM) I currently work for. In our line of business, we receive prescription drug claims from pharmacies, and are given approximately 10 seconds or less to process the claim, determine the price of the drug being purchased, determine the discount the cardholder's group has at that pharmacy, determine the patient's copy, how much funds he or she has left in his or her account, and send all that information back to the pharmacy, all while the customer is waiting at the counter. It's all a little like processing credit cards for purchases. That being said, we need a way to look at claims as they happen, real-time, but the database the production claim adjudicator is using can't be accessed without the chance of slowing down the claim processing. To fix the problem, we built SAFMQ and publish claims (in XML format) to the message queue real-time, and another system reads the claim details and writes them to a data warehouse type database, enabling our customer service agents the ability to lookup claim results real-time.

Using the code

Here's an example showing the number of messages that can be enqueued with the server per second:

/*
This demo program creates a queue named 
"test", writes 10000 messages,
then determines how ling it took to write 
the messages to the server and outputs the results.
*/

#include <windows.h>
#include <iostream>
#include "lib/MessageQueue.h"
#include "lib/MQFactory.h"
#include "lib/MQConnection.h"

using namespace safmq;
using namespace std;

#define MCOUNT 10000
int main(int argc, char* argv[]) {
    try {
        ErrorCode     ec;
        // Build a connection to the server
        MQConnection* con = 
          MQFactory::BuildConnection("safmq://localhost","admin","");
        // remove the queue just in case it was already there
        con->DeleteQueue("test");
        // create the queue named "test"
        con->CreateQueue("test");

        // Open an instance of the queue
        MessageQueue *queue = new MessageQueue(con,"test",false);

        // Get the start time for our process
        DWORD start = GetTickCount();

        for (int x=0; x<MCOUNT; x++) { {
            QueueMessage msg;
            // Set up the message
            msg.setLabel("Test Message");
            msg.setBodyType(BT_TEXT);
            *msg.getBufferStream() << "Test test test" << endl;

            // Sed the message to the server
            ec = queue->Enqueue(msg);
            if (ec != EC_NOERROR) {
               cout << "Error Code: " << ec << endl;
               break;
            }
            // output some status dots
            if (x%100 == 0)
                putchar('.');
        } }
        putchar('\n');

        // Get the end time
        DWORD end = GetTickCount();

        con->DeleteQueue("test");
        delete queue;
        delete con;

        // output the stats on our run.
        double duration = (end-start)/1000.0;
        cout << "Durration: " << duration << endl;
        cout << "Rate:style='mso-spacerun:yes'>      " 
             << (double)MCOUNT / duration 
             << " Messages per second." << endl;

    } catch (ErrorCode e) {
        cout << "Error Code: " << e << endl;
    } catch (exception& e) {
        cout << e.what() << endl;
    }
    return 0;
}

Points of Interest

Some of the things I tried to use when designing the SAFMQ client API was using the I/O stream pattern. If you dig around in the code, you'll find a directory containing classes to implement a std::basic_iostream derived class that wraps socket communication. You might find, there is some repetition between the code written for SAFMQ and MFC. That's because the code is actually cross-compilable, and can be compiled on Linux and other UNIX based systems.

Additionally, there're classes for performing SSL communication either directly or via the I/O stream pattern. This way, you can chain together the type of underlying transport mechanism, and use class construction to determine whether you use SSL or not. You'll see this in action in the MQFactory class, which is part of the client API.

History

SAFMQ's home page is located at SourceForge, you'll find the documentation for the API setting up the server, and the Java interface, there.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

mattbattey
Architect
United States United States
A programmer for 20 years and professionaly employed for 12, I am currently Cheif Engineer for Pharmacy Chare Professionals, Inc., located in Omaha, NE.

My experience is in the area of OO Design, Application, and programming, technical team leadership, RDBMS applications, ISAM applications, Image Processing, Mathematical image generation, Client-Server business applications, eBusiness applications, XML & EDI B2B communications, Java application development, C/C++ application development, CFML/ASP/VB development, on systems like Win2K/NT/98/95, Linux, Irix, Solaris, and MacOS.

You may also be interested in...

Comments and Discussions

 
GeneralStatus of this code Pin
Robert Mecklenburg17-Oct-06 7:00
memberRobert Mecklenburg17-Oct-06 7:00 
GeneralRe: Status of this code Pin
mattbattey20-Nov-06 14:13
membermattbattey20-Nov-06 14:13 
GeneralRe: Status of this code Pin
jmansion3-Jul-07 19:14
memberjmansion3-Jul-07 19:14 
General/var/safmq/queues Pin
jm.alkema26-Aug-06 2:41
memberjm.alkema26-Aug-06 2:41 
GeneralE-Mail Bounced Pin
Neville Franks13-Jul-06 12:21
memberNeville Franks13-Jul-06 12:21 
GeneralRe: E-Mail Bounced Pin
BarryHolleran30-Apr-12 14:32
memberBarryHolleran30-Apr-12 14:32 
GeneralRe: E-Mail Bounced Pin
Neville Franks2-May-12 19:27
memberNeville Franks2-May-12 19:27 
GeneralAdd JMS support to the Java package Pin
Ronny AERTS6-Mar-06 5:40
memberRonny AERTS6-Mar-06 5:40 
GeneralRe: Add JMS support to the Java package Pin
mattbattey20-Nov-06 14:15
membermattbattey20-Nov-06 14:15 
Generalconfiguration file in xml format Pin
Ronny AERTS6-Mar-06 5:37
memberRonny AERTS6-Mar-06 5:37 
GeneralInstalling SAFMQ on a windows machine as a service Pin
Ronny AERTS6-Mar-06 4:37
memberRonny AERTS6-Mar-06 4:37 
GeneralRe: Installing SAFMQ on a windows machine as a service Pin
mattbattey6-Mar-06 4:49
membermattbattey6-Mar-06 4:49 
GeneralInteresting++ Pin
Neville Franks17-Jan-06 0:01
memberNeville Franks17-Jan-06 0:01 
GeneralRe: Interesting++ Pin
mattbattey17-Jan-06 6:31
membermattbattey17-Jan-06 6:31 
I’ll answer your last question first: I didn’t know about MQ4CPP or XmlBlaster when I started work on SAFMQ... I did a search for “message queue” on freshmeat.net, sourceforge.net, and codeproject.com, and I wasn’t able to find what I was looking for, which was a store and forward message oriented communication mechanism, specifically to reduce the load on a high volume database server and move the data to a data warehouse in real-time, with the requirement that the publisher and subscriber need not be operating at the same time. Even when I published this article, I hadn’t heard of MQ4CPP or XmlBlaster, thanks for pointing them out to me!

One difference between SAFMQ and both MQ4CPP and XmlBlaster is the software license. I’ve chosen to use the Apache 2.0 license, whereas MQ4CPP and XmlBlaster have chosen the LGPL (GNU) license. Under Apache 2.0, commercial products may included SAFMQ in object form with or without modification and sell their product for profit at a price greater than the distribution media, as long as attribution is given to SAFMQ as being licensed under Apache 2.0 and that the copyright owner is “Matthew Battey,” and if modifications have been made to SAFMQ they must be noted. However, under LGPL, the software must remain free of cost, save the cost of distribution media, and depending on how one reads the LGPL, your code that uses an LGPL library must also be provided to the general public in an open-source form. Now, I am not an expert on LGPL licensing, and if I have misinterpreted its language, please correct me.

If I were to compare SAFMQ to XmlBlaster, the first note I would make is about architecture. The SAFMQ server is written in C & C++, and uses standard libraries. One exception is that optionally it can use OpenSSL for encrypted communication, but this is not required (OpenSSL is also under an Apache style license, btw). I don’t have a great under standing of XmlBlaster, but it looks like you need a CORBA orb to achieve high performance from it. Notably, the performance listed at XmlBlaster’s website is on an older machine (an AMD-K7 600 MHz) they received 672 messages per second. In tests under Linux with a 3.4 GHz PIV, with 768 MB, we received > 5900 messages per second using the C++ API and > 5200 messages per second using the Java API. These aren’t apples to apples comparisons, so you may need to experiment on your own. SAFMQ doesn’t require the use of XML, although it is possible to pass XML documents in the message buffer. To XmlBlaster’s benefit, it has a very wide range of transport protocols and client API libraries. XmlBlaster seems to be mature, and has been around a while, and has a very wide feature set, and has a great website detailing its features and benefits.

On MQ4CPP, in the short time I had to look at it, I was able to under stand less than I did about XmlBlaster. However, from my initial reading of code snippets, I saw what looked as though MQ4CPP did not have a central server. Instead, one would implement a server via use of MQ4CPP’s framework, instantiate it, and then clients could attach and deliver messages. Storing and Forwarding message can be accomplished by coding this feature into the “server” MessageQueue object. Also it appears that MQ4CPP is a C++ only suite. Also it appears the MQ4CPP supports encryption, but at the message level, instead of at the protocol level. Whereas SAFMQ supports SSL for communications, so the entire data exchange is encrypted, via both C++ or Java API.

I don’t claim to be an expert on either MQ4CPP or XmlBlaster, so please point out omissions, errors, and ignorance on my part.

GeneralRe: Interesting++ Pin
Neville Franks17-Jan-06 19:24
memberNeville Franks17-Jan-06 19:24 
GeneralRe: Interesting++ Pin
mattbattey18-Jan-06 4:43
membermattbattey18-Jan-06 4:43 
GeneralRe: Interesting++ Pin
rpompeo6817-Jan-06 23:29
memberrpompeo6817-Jan-06 23:29 
GeneralRe: Interesting++ Pin
mattbattey18-Jan-06 4:53
membermattbattey18-Jan-06 4:53 
GeneralInteresting - Nice work .. Pin
Garth J Lancaster16-Jan-06 22:52
professionalGarth J Lancaster16-Jan-06 22:52 
GeneralRe: Interesting - Nice work .. Pin
mattbattey17-Jan-06 4:59
membermattbattey17-Jan-06 4:59 
GeneralRe: Interesting - Nice work .. Pin
Garth J Lancaster17-Jan-06 10:15
professionalGarth J Lancaster17-Jan-06 10:15 
GeneralRe: Interesting - Nice work .. Pin
mattbattey17-Jan-06 11:26
membermattbattey17-Jan-06 11:26 
GeneralRe: Interesting - Nice work .. Pin
Garth J Lancaster17-Jan-06 11:37
professionalGarth J Lancaster17-Jan-06 11:37 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Cookies | Terms of Use | Mobile
Web06 | 2.8.190419.4 | Last Updated 16 Jan 2006
Article Copyright 2006 by mattbattey
Everything else Copyright © CodeProject, 1999-2019
Layout: fixed | fluid