Click here to Skip to main content
Click here to Skip to main content

AxPipe - Multi-Threaded Binary Stream C++ Class Library

, 14 Jan 2004
Rate this:
Please Sign up or sign in to vote.
A small and efficient collection of classes and templates to create multi-stage multi-threaded data processing pipe lines

Introduction

AxPipe is a very small and efficient class library implementing an extended stream and pipe abstraction similar to Unix Shell pipes. Both push and pull model processing, run-time built multi-stage pipes, optional multi-threading applied per stage, splits, joins and a very small footprint are distinguishing features. It's fully documented, with sample code and some stock transformations included.

The current implementation is for Win32 API with Visual Studio 6 or 2002, but porting to Unix should pose little difficulty, only the threading and co-routine support needs modification. AxPipe is a light-weight library easily learned and applied. Currently it is being used for development of the next version of AxCrypt, http://axcrypt.sourceforge.net.

It's still in active development, and I'd appreciate any and all constructive feedback and suggestions for improvements, or reasons for moving to an existing framework or library that does it better/faster/smaller/neater/safer/whatever.

Using the code

The source code comes with a complete demo application and complete documentation for every namespace, class, member, method, enumeration, define, template etc.

All is also available on the project home page, http://axpipe.sourceforge.net, where updated versions will be found as well. There you will find code samples and further overviews.

Introduction

The basic paradigm is taken from the Unix Shell pipe notation, where you might write:

crypt <file.txt | compress | tar >out.tar
     but you can also write, for example,
tar <file.txt | crypt | compress >out.z

The programs above are semi-ficticious, it's just to demonstrate the principle whereby input sources such as a file can be redirected into a processing program, which sends it on, where it can be connected to another processing program, or to a final destination.

I've frequently wanted to use the same principle for programming purposes in C++, but with minimal overhead and supporting different programming models. So I wrote this package.

The following is a minimal complete program demonstrating the basic usage.

/*! \file
    \brief A most simple AxPipe program

    Read an input file, pass it through a do-nothing
    pipe, and write it to a file. No error checking
    at all for clarity.
*/

#include "stdafx.h"
#include "AxPipe.h"
#include "CFileMap.h"

/// \brief Just pass data along - do nothing with it.
class CPipeDoNothing : public AxPipe::CPipe {
    void Out(AxPipe::CSeg *pSeg) {
        // Insert your code here...
        Pump(pSeg);
    }
};

int
_tmain(int argc, _TCHAR* argv[]) {
    AxPipe::CGlobalInit axpipeInit;    // It just has to be there.

    AxPipe::CSourceMemFile sourceFile; // The source is a memory mapped file

    sourceFile.Init(argv[1]);          // Set the source file name
    // Append a stage, in it's own thread
    sourceFile.Append(new AxPipe::CThread<CPipeDoNothing>);
    // Continue to append a sink to accept the output
    sourceFile.Append((new AxPipe::CSinkMemFile)->Init(argv[2]));

    // Initialize, Process the data, End process and Finalize.
    sourceFile.Open()->Drain()->Close()->Plug();

    // Check for any errors in any parts...
    if (sourceFile.GetErrorCode()) {
        // Print a clear text representation of the problem
        fprintf(stderr, sourceFile.GetErrorMsg());
        return sourceFile.GetErrorCode();
    }

    return 0;
}

Points of Interest

If you've never seen the use for co-routines (CreateFiber() et. al. in Win32 API) here's a pretty nice use, turning a push model processing filter into a pull model ditto. Contrary, I believe, to most similar packages AxPipe allows you to optionally, and in run-time, build processing pipes with different stages and with or without threading on a stage-by-stage basis.

It's very restrictive in it's use of external dependencies, and is suitable for very small applications even with minimal run-time support. It does not use exceptions for that reason, and certainly not MFC or similar large frameworks. There are also some 'stock' transformations included, developed on a needed basis for the projects where I use AxPipe, but the long-term idea is to gather a number of common sources, transformations and sinks into a very easily re-usable library of ready-made components.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

Svante Seleborg
Web Developer Axantum Software AB
Sweden Sweden
I've been working with all aspects of software development since 1979 - from compiler construction to management. Currently I'm an independent consultant mostly specializing in computer security. Please see my homepage for contact details.
 
I speak C like a native, and have a pretty good grasp of C++. The most recent five years C# has been the main development language. Traditionally Unix has been the dominating environment, but currently the scales have tipped over to Windows, due to market demands but I'm equally at home developing in both environments.
 
When I'm not coding I'm usually sitting on one of my 4 bikes, indoors or outdoors, on the road or in the woods.

Comments and Discussions

 
GeneralInteresting but .. PinmemberNeville Franks12-Jan-04 9:37 
Hi Svante,
This is interesting and useful code, but its GPL license makes it of limited use to most folks here at CP.
 
Neville Franks, Author of ED for Windows. Free Trial at www.getsoft.com

GeneralRe: Interesting but .. PinmemberSvante Seleborg12-Jan-04 10:22 
GeneralRe: Interesting but .. PinmemberJörgen Sigvardsson12-Jan-04 13:27 
GeneralGPL or not Pinmemberdog_spawn12-Jan-04 22:51 
GeneralRe: GPL or not Pinmemberrudisaurus20-Jan-04 7:40 
GeneralRe: GPL or not PinsussCiurdari Marius30-Jan-04 16:26 
GeneralThings to add Pinmemberdog_spawn12-Jan-04 3:16 
GeneralRe: Things to add PinmemberSvante Seleborg12-Jan-04 6:12 
GeneralRe: Things to add Pinmemberdog_spawn12-Jan-04 6:19 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.141015.1 | Last Updated 15 Jan 2004
Article Copyright 2004 by Svante Seleborg
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid