Click here to Skip to main content
15,867,453 members
Articles / Programming Languages / C++
Article

IoBind, a serializer code factory.

Rate me:
Please Sign up or sign in to vote.
4.89/5 (10 votes)
29 Jun 20037 min read 78.7K   765   29   18
IoBind proposes a new approach to object serialization.

Introduction

IoBind is a highly flexible library for serializing objects to/from string. It uses meta-programming and policies to create reader and writer for complex objects at compile time.

In other words, IoBind is a code factory that produces customized readers and writers: you give the action to take and the objects to process and it will generate the corresponding code.

The latest versions of IoBind are available at SourceForge.

Outline

  • Quick examples
    • Converting a string to a string
    • Escaping string to XML
    • Combining policies
    • Reading back data
  • Compiling and instalation
  • The encode method
  • Policies
    • operator +
    • String manipulation
    • Base64
    • Zip
    • XML escaping, Latin1 escaping 
    • sequence container
    • pair structure
    • associative containers
  • History
  • References

Quick examples

Let's start with some quick examples to see what IoBind is about.

Converting a string to a ... string

Let see a first snippet:

C++
string s("<xml/>");
// we convert s to a string
s = encode( s, to_string_p );
cerr<<s<<endl;
-- output
<xml/>

This seems to be useless since I'm converting a string to string. Anyway, let's see what's in the second line:

  • encode is a template function that applies a conversion policy (see below) to s,
  • to_string_p is a conversion policy that takes a value and converts it to a string ( using boost::format library ).

Escaping string to XML

Now, suppose that you need to store this string into an xml document. You need replace escape characters (<, >, etc..) by &lt;, &gt;, etc:

C++
s="escape me:&\"'<>"
s=encode(s, escape_to_xml_p );
cerr<<s<<endl;
-- output
escape me:&amp;&quot;&apos;&lt;&gt;

What has happened here: the string has been converted to an escaped string using the escape_to_xml_p conversion policy.

Hence using predefined policies you can easily transform a string:

  • base64,
  • encryption,
  • zipping,
  • etc...

Combining policies

Having a set of basic policies is good but creating new policies by combining them really makes things interresting  (this is where meta-programming takes place).
Suppose that you have a vector of string, you want to convert it to string and to escape this string to xml:

C++
vector<string> v_string;
// filling v_int
v_string.push_back("a string");
v_string.push_back("<string/>");

// transforming to escaped string:
s = encode( 
       v_string.begin(), 
       v_string.end(), 
       escape_to_xml_p * sequence_to_string_p 
       );
cerr<<s<<endl;
-- output
a string,<string/>

In this call, sequence_to_string_p is a policy that converts an iterator range to a string. The operator + combines the two policies (similar to function composition) and creates a new conversion policy:

a * b applied to a value v is equivalent to a( b( v ) )

Note that you can also specify a policy to be applied to each value of the container. This, combined with a pair policy, can be used to serializes associatives containers.

Reading back data

All the "to" policies have their "from" counter part to read data. For example, we can easily read a vector<int> from a string: 

C++
vector<int> v_int;
string str("0,1,2,3,4");

// reading vector of ints
v_int = encode(str,
               sequence_from_string_p
                   % v_int
                   << from_string<INT>() 
        );
where
  • sequence_from_string_p is a generic conversion policy that read a sequence of strings. This conversion policy is a template class that depends on two template paramters:
    • Container, the type of container to fill,
    • Policy, the conversion policy to apply to each element, 
  • the operator % builds a new sequence_from_string conversion policy that adds values to a container of same type as v_int. Concretely, this operator replaces the Container template paramter with the type of v_int,
  • the operator << is similar to % but it works on the policy.

The above is rather confusing, let's go deeper to see how policies are built step by steps:

C++
sequence_from_string_p
                   % v_int
                   << from_string<INT>()
  1. sequence_from_string_p = A: this is a pre-defined sequence conversion policy: it reads a sequence of strings separated by commas,
    • Container = vector<string>
    • Policy = from_string<string>
  2. A % v_int = B: the container has been replaced by v_int type:
    • Container = vector<int>
    • Policy = from_string<string>
  3. B << from_string<INT>() </INT>: the conversion poliy applied to the elements reads a int from a string:
    • Container = vector<int>
    • Policy = from_string<int>

So with this little statement, we have build a new conversion policy that is a complex mixture of A and B.

Compiling and installation

You need Boost (see [1]) and a good compiler (don't think VC6 will manage to compile). IoBind heavily uses Boost: it uses type_traits, mpl, call_traits, spirit, (regex optional) and

format 
libraries.

The headers can be included by

C++
#include <iobind/iobind.hpp>

Note also that all the IoBind classes are in the

iobind 
namespace.

The encode method

encode is a template method that applies a conversion policy to an object. It's return type depends on the policy return type. It comes with two overloads:

C++
template<
    typename Value,
    typename Policy
>
typename Policy::return_type 
    encode(
         Value const& value_, 
         Policy const& policy_
    );
C++
// range version
template<
    typename Iterator,
    typename Policy
>
typename Policy::return_type 
    encode(
        Iterator begin_, 
        Iterator end_, 
        Policy const& policy_
    );

This is the front end of IoBind to the user. Example of use of this method were given in the secion above.

Conversion policies

The individual steering behaviors [...] are components of a larger structure, like notes of a melody or words of a story. Craig Reynolds, Steering Behaviors for Autonomous Characters, GDC99.

Conversion policies are the constitutive pieces of IoBind. You can combine them to create complex serializers.

operator *

Takes two policies and combines them. If a and b are two policies, then 

a*b( value) = a( b( value ) ).

String manipulation

This is a basic string conversion policy not very useful used alone, but it become quite handy when combined with others.

  • C++
    struct to_string;
    converts the value using boost::format. If the value does not support this, the compiler will fail.
  • C++
    template<typename Value>
    struct from_string;
    transform a string to Value using ostringstream,

Example:

C++
int i;
string str=encode(i, to_string() );
i=encode(str, from_string<std::string><INT>() );

Note that almost all policies have predifined instances: the

to_string 
policy is instanciated as to_string_p .

Base64

Converts streams to the base64 scheme.

  • C++
    struct to_base64;
    converts a stream to base64 notation,
  • C++
    struct from_base64;
    converts back a stream from base64 notation,

Example:

C++
string str="test";
str=encode( str, to_base64_p);
str=encode( str, from_base64_p);

These policies are base on the base64 iostream converter from Konstant Pilipchuk:

//  base64.hpp 
//  Autor Konstantin Pilipchuk
//  mailto:lostd@ukr.net

XML and Latin1 escaping

This policy takes care of transforming a string to XML or Latin1 conformant string. It replaces reserved characters <,>,... by &lt;, &gt;, etc...

  • C++
    struct escape_to_xml;
    
    struct escape_to_latin1;
    escapes an string to XML (< to &lt;) or Latin1,
  • C++
    struct unescape_from_xml;
    
    struct unescape_from_latin1;
    unescapes an string from XML (&lt; to <) or Latin1,

The usage is straight forward and similar to to_base64, from_base64.

Sequence container

This policy handles sequence containers such as vector, list, etc (as you will see later, it can also be used for associative containers).

  • C++
    template<
        typename Policy
    >
    struct sequence_to_string;
    converts a sequence to a string. This policy has the following constructor:
    C++
    sequence_to_string(
        policy_const_reference item_policy_,
        string_param_type begin_ = "",
        string_param_type delimiter_ = ",",
        string_param_type end_ = ""
    )
    where item_<CODE>policy_ is the policy applied to the sequence elements, and the other parameters are used to separated the data. In fact, the core of the writer is:
    C++
    output
      <<m_begin
      ..
      <<m_policy.encode(value)<<m_delimiter, // this is done multiple times
      ..
      <<m_end;
  • C++
    template<
        typename Container,
        typename Policy
    >
    struct sequence_from_string;
    reads a sequence from a string and fills a container. This policy has the following constructor:
    C++
    sequence_from_string(
        policy_const_reference item_policy_,
        string_param_type begin_ = "",
        string_param_type delimiter_ = ",",
        string_param_type end_ = ""
    )
    where the parameters have similar behavior as above. The item_policy_ is used to transform the string before adding it to the container.

These policies support other operators that take care of policy or container change:

  • <<, changes the policy,
  • %, changes the container type (only for pair_from_string).

Example converting elements of a vector<float> to base64 and back:

C++
vector<float> v_f;
for (i=1;i<5;++i)
    v_f.push_back( 1/static_cast<FLOAT>(i) );

str=encode(
        v_f.begin(), 
        v_f.end(), 
        sequence_to_string_p << to_base64()    
    );
cerr<<"\tv (to_string, base64): "<<str<<endl;
cerr<<"\tv is cleared..."<<endl;
v_f.clear();
v_f=encode(
    str,
    sequence_from_string_b( v_f,  from_string<float>() * from_base64() )
    );
cerr<<"\tv (from_string from base64): "<<encode(
        v_f.begin(), 
        v_f.end(), 
        sequence_to_string_p
        )
    <<endl;
-- output
        v (to_string, base64): MQA=,MC41AA==,MC4zMzMzMzMA,MC4yNQA=
        v is cleared...
        v (from_string from base64): 1,0.5,0.333333,0.25

Pair

This policy handles the famous std::pair structure.

  • template<
        typename FirstPolicy,
        typename SecondPolicy
    >
    class pair_to_string;
    converts a pair to a string. This class has the following constructor:
    C++
    pair_to_string(
        first_policy_const_reference first_policy_,
        second_policy_const_reference second_policy_,
        string_param_type begin_ = "(",
        string_param_type delimiter_ = ":",
        string_param_type end_ = ")"
    )
    where first/second_policy_ are the policies applied respectively to the first and second members of pair, and the other parameters are used to separated the data. In fact, the core of the writer is:
    C++
    output
      <<m_begin
      <<m_first_policy.encode(value.first)
      <<m_delimiter,
      <<m_second_policy.encode(value.second),
      <<m_end;
  • template<
        typename Pair,
        typename FirstPolicy,
        typename SecondPolicy
    >
    class pair_from_string;
    reads a pair from a string. This class has the following constructor:
    C++
    pair_from_string(
        first_policy_const_reference first_policy_,
        second_policy_const_reference second_policy_,
        string_param_type begin_ = "(",
        string_param_type delimiter_ = ":",
        string_param_type end_ = ")"
    )
    where first/second_policy_ are the policies applied respectively to the first and second members of pair, and the other parameters are used to separated the data. In fact, the core of the writer is:
    C++
    pair.first=m_frist_policy.encode(first_string);
    pair.second=m_second_policy.encode(second_string);    

These policies support new operators that take care of policy, pair type change:

  • <<, changes the first policy,
  • >>, changes the second policy,
  • %, changes the pair type (only for pair_from_string).

Example:

pair<int,string> p_fs(1,"second");
str=encode( p_fs, pair_to_string_p);
cerr<<"\tpair (1,second): "<<str<<endl;
cerr<<"\treseting pair"<<endl;
p_fs.first=0;
p_fs.second="";
p_fs=encode(
    str, 
    pair_from_string_p
        % p_fs 
        << from_string<int>()
        >> from_string<std::string>()
    );
cerr<<"\tpair (from string):"<<encode( p_fs, pair_to_string_p)<<endl;
-- output
        pair (1,second): (1:second)
        reseting pair
        pair (from string):(1:second)

Associative containers

Associative containers such as map,set, etc... are just a combination of a sequence container and a pair (speaking about serialization). Hence, using sequence_to/from_string and pair_to/from_string, you can easily build serializers for them, witout redifining new classes (the compiler will build them for you):

C++
map<float,string> m_fs;
const char* numbers[] = {"one", "two", "three", "four", "five"};

for (i=1;i<5;++i)
    m_fs[static_cast<float>(i)]=numbers[i];

// dumping to string
str=encode(
    m_fs.begin(), 
    m_fs.end(), 
    sequence_to_string_p << pair_to_string_p
    );
cerr<<"\tm (to_string): "<<str<<endl;
cerr<<"\tm is cleared..."<<endl;

// reading back the data    
m_fs.clear();
m_fs=encode(
    str,
    sequence_from_string_p % m_fs
    << (
        pair_from_string_p
        % pair<float,std::string>() 
        << from_string<float>()
        >> from_string<std::string>() 
      )                
    );
cerr<<"\tm (from_string): "<<encode(
    m_fs.begin(), 
    m_fs.end(), 
    sequence_to_string_p << pair_to_string_p
    )
    <<endl;
-- output
-- associative container:
    combination of sequence_to/from_string and pair
        m (to_string): (1:two),(2:three),(3:four),(4:five)
        m is cleared...
        m (from_string): (1:two),(2:three),(3:four),(4:five)

Zip

The zip policy uses the famous zlib C library (see [2]). It compresses a buffer to another buffer. This policy, combined with base64 can be used to store byte buffers in XML files:

C++
vector<unsigned char> buffer;
string s;
//zipping and converting to base64

s = encode( buffer, to_base64 * to_zip_p );

Others

There is room for a lot of other policies:

  • encryption, decryption,
  • url encoding, decoding,
  • ...

History

  • 06-30-2003 - Added zip, latin1, case, crc, hex policies.
  • 05-04-2003 - Initial release.

Reference

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here


Written By
Engineer
United States United States
Jonathan de Halleux is Civil Engineer in Applied Mathematics. He finished his PhD in 2004 in the rainy country of Belgium. After 2 years in the Common Language Runtime (i.e. .net), he is now working at Microsoft Research on Pex (http://research.microsoft.com/pex).

Comments and Discussions

 
GeneralSerializing a Spirit Grammar Pin
Karim Attaleb20-Sep-05 7:12
Karim Attaleb20-Sep-05 7:12 
QuestionVersion Control? Pin
Dave Handley13-Oct-04 11:47
Dave Handley13-Oct-04 11:47 
Questiongreat article - similar to java chained streams? Pin
Jim Crafton4-Jun-03 6:48
Jim Crafton4-Jun-03 6:48 
AnswerExample ? Pin
Jonathan de Halleux4-Jun-03 6:56
Jonathan de Halleux4-Jun-03 6:56 
GeneralRe: Example ? Pin
Jim Crafton4-Jun-03 11:23
Jim Crafton4-Jun-03 11:23 
GeneralRe: Example ? Pin
Jonathan de Halleux4-Jun-03 11:38
Jonathan de Halleux4-Jun-03 11:38 
GeneralRe: Example ? Pin
Jim Crafton4-Jun-03 17:30
Jim Crafton4-Jun-03 17:30 
GeneralRe: Example ? Pin
Jonathan de Halleux4-Jun-03 20:35
Jonathan de Halleux4-Jun-03 20:35 
GeneralRe: Example ? Pin
Jim Crafton5-Jun-03 2:58
Jim Crafton5-Jun-03 2:58 
Jonathan de Halleux wrote:
Looks like you got a lot of docs to write on your hand

Ackkkk, don't get me started! And I HATE writing - absolutely despise it! But at least I have a bit of a process for churning out the actual html output now that works better than what i was using before. Long live docbook!


¡El diablo está en mis pantalones! ¡Mire, mire!

Real Mentats use only 100% pure, unfooled around with Sapho Juice(tm)!
General-&gt; WikiWeb Pin
Jonathan de Halleux5-Jun-03 3:35
Jonathan de Halleux5-Jun-03 3:35 
GeneralRe: -&gt; WikiWeb Pin
Jim Crafton5-Jun-03 7:08
Jim Crafton5-Jun-03 7:08 
GeneralRe: -&gt; WikiWeb Pin
Jonathan de Halleux5-Jun-03 20:22
Jonathan de Halleux5-Jun-03 20:22 
GeneralRe: -&gt; WikiWeb Pin
Jim Crafton6-Jun-03 3:38
Jim Crafton6-Jun-03 3:38 
GeneralRe: -&gt; WikiWeb Pin
Jonathan de Halleux6-Jun-03 3:49
Jonathan de Halleux6-Jun-03 3:49 
QuestionExcellent but link? Pin
Michael A. Barnhart4-Jun-03 6:21
Michael A. Barnhart4-Jun-03 6:21 
AnswerRe: Excellent but link? Pin
Jonathan de Halleux4-Jun-03 6:25
Jonathan de Halleux4-Jun-03 6:25 
AnswerRe: Excellent but link? Pin
Jonathan de Halleux4-Jun-03 6:29
Jonathan de Halleux4-Jun-03 6:29 
GeneralRe: Excellent but link? Pin
Michael A. Barnhart4-Jun-03 9:29
Michael A. Barnhart4-Jun-03 9:29 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.