Click here to Skip to main content
11,409,450 members (36,476 online)
Click here to Skip to main content

AxPipe - Multi-Threaded Binary Stream C++ Class Library

, 14 Jan 2004
Rate this:
Please Sign up or sign in to vote.
A small and efficient collection of classes and templates to create multi-stage multi-threaded data processing pipe lines

Introduction

AxPipe is a very small and efficient class library implementing an extended stream and pipe abstraction similar to Unix Shell pipes. Both push and pull model processing, run-time built multi-stage pipes, optional multi-threading applied per stage, splits, joins and a very small footprint are distinguishing features. It's fully documented, with sample code and some stock transformations included.

The current implementation is for Win32 API with Visual Studio 6 or 2002, but porting to Unix should pose little difficulty, only the threading and co-routine support needs modification. AxPipe is a light-weight library easily learned and applied. Currently it is being used for development of the next version of AxCrypt, http://axcrypt.sourceforge.net.

It's still in active development, and I'd appreciate any and all constructive feedback and suggestions for improvements, or reasons for moving to an existing framework or library that does it better/faster/smaller/neater/safer/whatever.

Using the code

The source code comes with a complete demo application and complete documentation for every namespace, class, member, method, enumeration, define, template etc.

All is also available on the project home page, http://axpipe.sourceforge.net, where updated versions will be found as well. There you will find code samples and further overviews.

Introduction

The basic paradigm is taken from the Unix Shell pipe notation, where you might write:

crypt <file.txt | compress | tar >out.tar
     but you can also write, for example,
tar <file.txt | crypt | compress >out.z

The programs above are semi-ficticious, it's just to demonstrate the principle whereby input sources such as a file can be redirected into a processing program, which sends it on, where it can be connected to another processing program, or to a final destination.

I've frequently wanted to use the same principle for programming purposes in C++, but with minimal overhead and supporting different programming models. So I wrote this package.

The following is a minimal complete program demonstrating the basic usage.

/*! \file
    \brief A most simple AxPipe program

    Read an input file, pass it through a do-nothing
    pipe, and write it to a file. No error checking
    at all for clarity.
*/

#include "stdafx.h"
#include "AxPipe.h"
#include "CFileMap.h"

/// \brief Just pass data along - do nothing with it.
class CPipeDoNothing : public AxPipe::CPipe {
    void Out(AxPipe::CSeg *pSeg) {
        // Insert your code here...
        Pump(pSeg);
    }
};

int
_tmain(int argc, _TCHAR* argv[]) {
    AxPipe::CGlobalInit axpipeInit;    // It just has to be there.

    AxPipe::CSourceMemFile sourceFile; // The source is a memory mapped file

    sourceFile.Init(argv[1]);          // Set the source file name
    // Append a stage, in it's own thread
    sourceFile.Append(new AxPipe::CThread<CPipeDoNothing>);
    // Continue to append a sink to accept the output
    sourceFile.Append((new AxPipe::CSinkMemFile)->Init(argv[2]));

    // Initialize, Process the data, End process and Finalize.
    sourceFile.Open()->Drain()->Close()->Plug();

    // Check for any errors in any parts...
    if (sourceFile.GetErrorCode()) {
        // Print a clear text representation of the problem
        fprintf(stderr, sourceFile.GetErrorMsg());
        return sourceFile.GetErrorCode();
    }

    return 0;
}

Points of Interest

If you've never seen the use for co-routines (CreateFiber() et. al. in Win32 API) here's a pretty nice use, turning a push model processing filter into a pull model ditto. Contrary, I believe, to most similar packages AxPipe allows you to optionally, and in run-time, build processing pipes with different stages and with or without threading on a stage-by-stage basis.

It's very restrictive in it's use of external dependencies, and is suitable for very small applications even with minimal run-time support. It does not use exceptions for that reason, and certainly not MFC or similar large frameworks. There are also some 'stock' transformations included, developed on a needed basis for the projects where I use AxPipe, but the long-term idea is to gather a number of common sources, transformations and sinks into a very easily re-usable library of ready-made components.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

Svante Seleborg
Web Developer Axantum Software AB
Sweden Sweden
I've been working with all aspects of software development since 1979 - from compiler construction to management. Currently I'm an independent consultant mostly specializing in computer security. Please see my homepage for contact details.

I speak C like a native, and have a pretty good grasp of C++. The most recent five years C# has been the main development language. Traditionally Unix has been the dominating environment, but currently the scales have tipped over to Windows, due to market demands but I'm equally at home developing in both environments.

When I'm not coding I'm usually sitting on one of my 4 bikes, indoors or outdoors, on the road or in the woods.

Comments and Discussions

 
GeneralInteresting but .. PinmemberNeville Franks12-Jan-04 10:37 
GeneralRe: Interesting but .. PinmemberSvante Seleborg12-Jan-04 11:22 
Neville Franks wrote:
This is interesting and useful code, but its GPL license makes it of limited use to most folks here at CP

Not to get too philosophical about it, but actually that's of course true only if you won't or can't share the derived results in the same way...

Yes, yes, I know there are a myriad of cases where GPL is not acceptable - but in those cases it's almost certainly the case that the code will be used to make money - i.e. sell in commercial software. Nothing wrong with that actually, but then perhaps I'd like a piece of the pie. That's not too unfair, is it?

In the case that GPL is not ok for your intended use, for example due to inclusion into commercial and/or proprietary code - then I think it not more than fair that we make a deal on those terms.

Thus, I'm perfectly willing to discuss any licensing you like, including withdrawal of GPL. Please remember that just because I've released under GPL, there's nothing stopping me from giving you a license under completely different terms, for example a fully free
-to-use royalty-free do-what-you-will license.

Just contact me and explain the situation, and I'm sure we can come to terms. I'm easy Smile | :)

As for snippets, I hereby give all members of Code Project a license to use snippets of the code published here under no other obligation than to properly report it's origin. Please use common sense to define 'snippet'. If in doubt - mail me and ask.

I want the stuff to be shared and used! But I also believe in the power of open source, and that requires sharing of derived works as well, otherwise we severly limit the complexity and usefulness of available open source code to what one person can do.

So I got a bit philosophical anyway... Can't help it I guess.
GeneralRe: Interesting but .. PinmemberJörgen Sigvardsson12-Jan-04 14:27 
GeneralGPL or not Pinmemberdog_spawn12-Jan-04 23:51 
GeneralRe: GPL or not Pinmemberrudisaurus20-Jan-04 8:40 
GeneralRe: GPL or not PinsussCiurdari Marius30-Jan-04 17:26 
GeneralThings to add Pinmemberdog_spawn12-Jan-04 4:16 
GeneralRe: Things to add PinmemberSvante Seleborg12-Jan-04 7:12 
GeneralRe: Things to add Pinmemberdog_spawn12-Jan-04 7:19 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.150414.5 | Last Updated 15 Jan 2004
Article Copyright 2004 by Svante Seleborg
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid