IoBind, a serializer code factory.

Jonathan de Halleux

4.89/5 (9 votes)

Jun 4, 2003

7 min read

80083

765

IoBind proposes a new approach to object serialization.

Introduction

IoBind is a highly flexible library for serializing objects to/from string. It uses meta-programming and policies to create reader and writer for complex objects at compile time.

In other words, IoBind is a code factory that produces customized readers and writers: you give the action to take and the objects to process and it will generate the corresponding code.

The latest versions of IoBind are available at SourceForge.

Outline

Quick examples
- Converting a string to a string
- Escaping string to XML
- Combining policies
- Reading back data
Compiling and instalation
The encode method
Policies
- operator +
- String manipulation
- Base64
- Zip
- XML escaping, Latin1 escaping
- sequence container
- pair structure
- associative containers
History
References

Quick examples

Let's start with some quick examples to see what IoBind is about.

Converting a string to a ... string

Let see a first snippet:

string s("<xml/>");
// we convert s to a string
s = encode( s, to_string_p );
cerr<<s<<endl;

-- output
<xml/>

This seems to be useless since I'm converting a string to string. Anyway, let's see what's in the second line:

encode is a template function that applies a conversion policy (see below) to s,
to_string_p is a conversion policy that takes a value and converts it to a string ( using boost::format library ).

Escaping string to XML

Now, suppose that you need to store this string into an xml document. You need replace escape characters (<, >, etc..) by <, >, etc:

s="escape me:&\"'<>"
s=encode(s, escape_to_xml_p );
cerr<<s<<endl;

-- output
escape me:&amp;&quot;&apos;&lt;&gt;

What has happened here: the string has been converted to an escaped string using the escape_to_xml_p conversion policy.

Hence using predefined policies you can easily transform a string:

base64,
encryption,
zipping,
etc...

Combining policies

Having a set of basic policies is good but creating new policies by combining them really makes things interresting (this is where meta-programming takes place).
Suppose that you have a vector of string, you want to convert it to string and to escape this string to xml:

vector<string> v_string;
// filling v_int
v_string.push_back("a string");
v_string.push_back("<string/>");

// transforming to escaped string:
s = encode( 
       v_string.begin(), 
       v_string.end(), 
       escape_to_xml_p * sequence_to_string_p 
       );
cerr<<s<<endl;

-- output
a string,<string/>

In this call, sequence_to_string_p is a policy that converts an iterator range to a string. The operator + combines the two policies (similar to function composition) and creates a new conversion policy:

a * b applied to a value v is equivalent to a( b( v ) )

Note that you can also specify a policy to be applied to each value of the container. This, combined with a pair policy, can be used to serializes associatives containers.

Reading back data

All the "to" policies have their "from" counter part to read data. For example, we can easily read a vector<int> from a string:

vector<int> v_int;
string str("0,1,2,3,4");

// reading vector of ints
v_int = encode(str,
               sequence_from_string_p
                   % v_int
                   << from_string() 
        );

where

sequence_from_string_p is a generic conversion policy that read a sequence of strings. This conversion policy is a template class that depends on two template paramters:
- Container, the type of container to fill,
- Policy, the conversion policy to apply to each element,
the operator % builds a new sequence_from_string conversion policy that adds values to a container of same type as v_int. Concretely, this operator replaces the Container template paramter with the type of v_int,
the operator << is similar to % but it works on the policy.

The above is rather confusing, let's go deeper to see how policies are built step by steps:

sequence_from_string_p
                   % v_int
                   << from_string()

sequence_from_string_p = A: this is a pre-defined sequence conversion policy: it reads a sequence of strings separated by commas,
- Container = vector<string>
- Policy = from_string<string>
A % v_int = B: the container has been replaced by v_int type:
- Container = vector<int>
- Policy = from_string<string>
B << from_string() : the conversion poliy applied to the elements reads a int from a string:
- Container = vector<int>
- Policy = from_string<int>

So with this little statement, we have build a new conversion policy that is a complex mixture of A and B.

Compiling and installation

You need Boost (see [1]) and a good compiler (don't think VC6 will manage to compile). IoBind heavily uses Boost: it uses type_traits, mpl, call_traits, spirit, (regex optional) and formatlibraries.

The headers can be included by

#include <iobind/iobind.hpp>

Note also that all the IoBind classes are in the iobindnamespace.

The `encode` method

encode is a template method that applies a conversion policy to an object. It's return type depends on the policy return type. It comes with two overloads:

template<
    typename Value,
    typename Policy
>
typename Policy::return_type 
    encode(
         Value const& value_, 
         Policy const& policy_
    );

// range version
template<
    typename Iterator,
    typename Policy
>
typename Policy::return_type 
    encode(
        Iterator begin_, 
        Iterator end_, 
        Policy const& policy_
    );

This is the front end of IoBind to the user. Example of use of this method were given in the secion above.

Conversion policies

The individual steering behaviors [...] are components of a larger structure, like notes of a melody or words of a story. Craig Reynolds, Steering Behaviors for Autonomous Characters, GDC99.

Conversion policies are the constitutive pieces of IoBind. You can combine them to create complex serializers.

operator *

Takes two policies and combines them. If a and b are two policies, then

a*b( value) = a( b( value ) ).

String manipulation

This is a basic string conversion policy not very useful used alone, but it become quite handy when combined with others.

```
struct to_string;
```
converts the value using boost::format. If the value does not support this, the compiler will fail.

template<typename Value>
struct from_string;

transform a string to Value using ostringstream,

Example:

int i;
string str=encode(i, to_string() );
i=encode(str, from_string<std::string>() );

Note that almost all policies have predifined instances: the to_stringpolicy is instanciated as to_string_p .

Base64

Converts streams to the base64 scheme.

```
struct to_base64;
```
converts a stream to base64 notation,
```
struct from_base64;
```
converts back a stream from base64 notation,

Example:

string str="test";
str=encode( str, to_base64_p);
str=encode( str, from_base64_p);

These policies are base on the base64 iostream converter from Konstant Pilipchuk:

//  base64.hpp 
//  Autor Konstantin Pilipchuk
//  mailto:lostd@ukr.net

XML and Latin1 escaping

This policy takes care of transforming a string to XML or Latin1 conformant string. It replaces reserved characters <,>,... by <, >, etc...

struct escape_to_xml;

struct escape_to_latin1;

escapes an string to XML (< to <) or Latin1,

struct unescape_from_xml;

struct unescape_from_latin1;

unescapes an string from XML (< to <) or Latin1,

The usage is straight forward and similar to to_base64, from_base64.

Sequence container

This policy handles sequence containers such as vector, list, etc (as you will see later, it can also be used for associative containers).

template<
    typename Policy
>
struct sequence_to_string;

converts a sequence to a string. This policy has the following constructor:

sequence_to_string(
    policy_const_reference item_policy_,
    string_param_type begin_ = "",
    string_param_type delimiter_ = ",",
    string_param_type end_ = ""
)

where item_<CODE>policy_ is the policy applied to the sequence elements, and the other parameters are used to separated the data. In fact, the core of the writer is:

output
  <<m_begin
  ..
  <<m_policy.encode(value)<<m_delimiter, // this is done multiple times
  ..
  <<m_end;

template<
    typename Container,
    typename Policy
>
struct sequence_from_string;

reads a sequence from a string and fills a container. This policy has the following constructor:

sequence_from_string(
    policy_const_reference item_policy_,
    string_param_type begin_ = "",
    string_param_type delimiter_ = ",",
    string_param_type end_ = ""
)

where the parameters have similar behavior as above. The item_policy_ is used to transform the string before adding it to the container.

These policies support other operators that take care of policy or container change:

<<, changes the policy,
%, changes the container type (only for pair_from_string).

Example converting elements of a vector<float> to base64 and back:

vector<float> v_f;
for (i=1;i<5;++i)
    v_f.push_back( 1/static_cast<FLOAT>(i) );

str=encode(
        v_f.begin(), 
        v_f.end(), 
        sequence_to_string_p << to_base64()    
    );
cerr<<"\tv (to_string, base64): "<<str<<endl;
cerr<<"\tv is cleared..."<<endl;
v_f.clear();
v_f=encode(
    str,
    sequence_from_string_b( v_f,  from_string<float>() * from_base64() )
    );
cerr<<"\tv (from_string from base64): "<<encode(
        v_f.begin(), 
        v_f.end(), 
        sequence_to_string_p
        )
    <<endl;

-- output
        v (to_string, base64): MQA=,MC41AA==,MC4zMzMzMzMA,MC4yNQA=
        v is cleared...
        v (from_string from base64): 1,0.5,0.333333,0.25

Pair

This policy handles the famous std::pair structure.

template<
    typename FirstPolicy,
    typename SecondPolicy
>
class pair_to_string;

converts a pair to a string. This class has the following constructor:

pair_to_string(
    first_policy_const_reference first_policy_,
    second_policy_const_reference second_policy_,
    string_param_type begin_ = "(",
    string_param_type delimiter_ = ":",
    string_param_type end_ = ")"
)

where first/second_policy_ are the policies applied respectively to the first and second members of pair, and the other parameters are used to separated the data. In fact, the core of the writer is:

output
  <<m_begin
  <<m_first_policy.encode(value.first)
  <<m_delimiter,
  <<m_second_policy.encode(value.second),
  <<m_end;

template<
    typename Pair,
    typename FirstPolicy,
    typename SecondPolicy
>
class pair_from_string;

reads a pair from a string. This class has the following constructor:

pair_from_string(
    first_policy_const_reference first_policy_,
    second_policy_const_reference second_policy_,
    string_param_type begin_ = "(",
    string_param_type delimiter_ = ":",
    string_param_type end_ = ")"
)

pair.first=m_frist_policy.encode(first_string);
pair.second=m_second_policy.encode(second_string);

These policies support new operators that take care of policy, pair type change:

<<, changes the first policy,
>>, changes the second policy,
%, changes the pair type (only for pair_from_string).

Example:

pair<int,string> p_fs(1,"second");
str=encode( p_fs, pair_to_string_p);
cerr<<"\tpair (1,second): "<<str<<endl;
cerr<<"\treseting pair"<<endl;
p_fs.first=0;
p_fs.second="";
p_fs=encode(
    str, 
    pair_from_string_p
        % p_fs 
        << from_string<int>()
        >> from_string<std::string>()
    );
cerr<<"\tpair (from string):"<<encode( p_fs, pair_to_string_p)<<endl;

-- output
        pair (1,second): (1:second)
        reseting pair
        pair (from string):(1:second)

Associative containers

Associative containers such as map,set, etc... are just a combination of a sequence container and a pair (speaking about serialization). Hence, using sequence_to/from_string and pair_to/from_string, you can easily build serializers for them, witout redifining new classes (the compiler will build them for you):

map<float,string> m_fs;
const char* numbers[] = {"one", "two", "three", "four", "five"};

for (i=1;i<5;++i)
    m_fs[static_cast<float>(i)]=numbers[i];

// dumping to string
str=encode(
    m_fs.begin(), 
    m_fs.end(), 
    sequence_to_string_p << pair_to_string_p
    );
cerr<<"\tm (to_string): "<<str<<endl;
cerr<<"\tm is cleared..."<<endl;

// reading back the data    
m_fs.clear();
m_fs=encode(
    str,
    sequence_from_string_p % m_fs
    << (
        pair_from_string_p
        % pair<float,std::string>() 
        << from_string<float>()
        >> from_string<std::string>() 
      )                
    );
cerr<<"\tm (from_string): "<<encode(
    m_fs.begin(), 
    m_fs.end(), 
    sequence_to_string_p << pair_to_string_p
    )
    <<endl;

-- output
-- associative container:
    combination of sequence_to/from_string and pair
        m (to_string): (1:two),(2:three),(3:four),(4:five)
        m is cleared...
        m (from_string): (1:two),(2:three),(3:four),(4:five)

Zip

The zip policy uses the famous zlib C library (see [2]). It compresses a buffer to another buffer. This policy, combined with base64 can be used to store byte buffers in XML files:

vector<unsigned char> buffer;
string s;
//zipping and converting to base64

s = encode( buffer, to_base64 * to_zip_p );

Others

There is room for a lot of other policies:

encryption, decryption,
url encoding, decoding,
...

History

06-30-2003 - Added zip, latin1, case, crc, hex policies.
05-04-2003 - Initial release.

IoBind, a serializer code factory.

Introduction

Outline

Quick examples

Converting a string to a ... string

Escaping string to XML

Combining policies

Reading back data

Compiling and installation

The encode method

Conversion policies

operator *

String manipulation

Base64

XML and Latin1 escaping

Sequence container

Pair

Associative containers

Zip

Others

History

Reference

The `encode` method