CompactFormatterPlus: Generic Serializer for Full and Compact Framework

• Download source code - 636.63 KB

Introduction

The advent of .NET Framework is the most significant event in the programming world over the last 10 years and I think it is still underestimated. It is the most solid and comprehensive framework ever. However, nothing and nobody is perfect. This article is about .NET Framework communication and compact framework communication (serialization) in particular.

Communication System Model

What is a communication framework? Generally a communication framework is an object delivery system. No more and no less. If this system implements RPC/RMI, another layer is built on top of that. The object delivery system in turn consists of just two sub layers:

1. Transport layer (physically delivers the stream of bytes to the destination) 
2. Converter from the byte stream to the object (a serializer) And nothing else.

The rest can be hidden inside this ideal communication framework.
The last statement is true only for the systems that describe the objects in the same terms (same OS, same .NET Framework).

Minimal (Optimal) Communication System

Minimal communication system has just one method Send(Object). Does the programmer have to care about how the object is converted to the byte stream, how it is managed by the threads, how it is queued, fragmented, etc.? No, it is the communication framework's job.

WCF

When I first looked at WCF, I had a feeling that I was missing something. It could not be true: automatic generic serialization had gone! Instead it has a semi manual very restrictive serialization. Judge for yourself:

public class Person 
{ 
    public string FirstName; 
    public string LastName; 
    public int ID; 
    ArrayList alist = new ArrayList(); 
    public ClassB _b; 
} 

In WCF, the decoration of such a primitive object for serialization looks like:

[DataContract(Name = "Customer", Namespace = "http://www.contoso.com")] 
class Person : IExtensibleDataObject 
{ 
    [DataMember()] 
    public string FirstName; 
    [DataMember] 
    public string LastName;     
    [DataMember()] public int ID; 
    [DataMember()] 
    ArrayList alist = new ArrayList(); 
    [DataMember()] public ClassB _b; 
    private ExtensionDataObject extensionData_Value; 
    public ExtensionDataObject ExtensionData 
    { 
        get 
        {     
            return extensionData_Value; 
        } 
        set 
        { 
            extensionData_Value = value; 
        }
     } 
} 

Does everybody like that? Apparently not. The Web is full of blogs suggesting numerous "smart" solutions overcoming the restrictive nature of WCF. The limited WCF serialization is "successfully" worked around by using BinaryFormatter and then putting the byte array as a parameter. Is it the price paid for the interoperability? Microsoft apparently sacrificed versatility for interoperability simply by limiting the functionality. The logical extreme of such an approach would be a return to the good old raw socket communication. Hence most of the WCF programs run on Windows, would it be more sensible having the generic serializer as a default and the rest as an option?

What Makes the Communication Framework OS Specific

The only thing that makes the system OS specific is the formatter (the serializer). It is the only component in the system that converts the stream to the object. The stream is always the same. As the sculpture is made of stone (any stone) so the object is made of stream. In fact, WCF has all these bits and pieces but the way the system is composed is far from perfect.

Interoperability

As I already mentioned, the Serializer makes the framework OS specific. In turn the serializer exists in the framework [say .NET]. What if the OS does not support the frameworks and does not have even the notion of the serializer? In this particular case, manual (or semi manual) processing of the byte stream is appropriate but that should be an option, not a default.

If You Like Everything Automatic, Why Not to Use Remoting?

Standard remoting is based on just synchronous communication model. It is unidirectional. The attempts to use call backs for the duplex communication do not work with the client behind proxy or NAT because Remoting opens a second connection for the events. Since the firewall is a standard network component, we may forget about bidirectional communication using remoting.

Total Recall

.NET standard communication suffers from the same disease that Windows OS suffered 10 years ago. Core Windows API is the same as it was 10 years ago, but the programming of Windows Form application 10 years back was a serious ordeal. Why? Because the programmer had to provide all the parameters manually. In order to provide all the parameters, he needed to know all Windows internals and how to use them. In .NET environment, all the defaults are set by the framework. We drag the controls from the toolbox and the rest is done automatically. If we need extra features, we can select them from the property grid or type them in the code.

In real life, we send the object (the letter or the parcel) to the recipient. Do we really care how the letter is delivered, what is the postman's name or the name of his pet dog? No, we don't. Usually we drop the letter in the postbox and the rest is done by default. If we want some extra features, like better security or delivery confirmation, we can get that as an extra. If the letter delivery looked like .NET communication (WCF in particular), the procedure of the letter delivery would have certainly been like that: we select from the catalog the number plate of the car that carries the letter, the name of the driver, the flight number, the type of the plane, envelope color, brand …. Most probably that would stop us from sending the letter at all.

Alternatives

While high priests of IT industry keep saying that everything is better than ever, the communication frameworks (mostly .NET socket based) keep multiplying.
The Code Project website hosts at least 10 of them. The design of a primitive communication class with a limited functionality (often it is enough) is not too hard, but the major showstopper was a serializer, specifically for the Compact Framework.

Compact Framework Remoting

Unfortunately Remoting and Binary formatter is not implemented (and apparently will not be) for the compact framework. The only choice is using limited XMLSerializer (which is not generic) and the implementation of the system where you can simply drop the object on one side and get it on the other is not technically possible. Apart from the serializer, other components like channels or threads are readily available in CF.

Compact Framework Serialization

CompactFormatterPlus : Generic Binary Serializer for Compact Framework

This work is based on a brilliant Angelo Scotto's CompactFormatter for the full and Compact Framework. The original CompactFormatter was written for .NET 1.1.
However the years went by and the opportunities for improvement emerged. Basically the reasons for the redesign are:

1. Compact Framework 2.0 has serializable attribute. This attribute was introduced only for compatibility with a full framework. Other than that, the practical value of this attribute is questionable because the classes built before this attribute was introduced obviously do not have it. And solely relying only on this attribute will make quite serializable classes non serializable. Original CompactFormatter has a similar attribute but it is not compatible with the full framework. Checking for this attribute (Angelo's serializable attribute) was removed. It made the classes created for the full framework and compact framework compatible. Though it has some drawback – totally non serializable objects will be attempted to serialize. 
2. Serialization of the DataSet and DataTable in CompactFormatter is not full. DataSet serialization designed by SimmoTech is far more advanced. Simotech serialization and the code can be found in The Code Project article. This code was incorporated into CompactFormatter as a surrogate.
3. For performance reasons, the original CompactFormatter puts the indexes of the classes in the stream instead of the class name. That is good if the serializer exists consistently during the communication session. Practically that is not always the case. The serializer may be instantiated dynamically and CompactFormatter certainly fails.
4. The way the classes are instantiated when deserialized: CompactFormatter uses incomplete assembly name or fully qualified assembly name. Either method fails if the communication platforms are different. For instance, fully qualified assembly name is meaningless when the object is serialized on full framework and deserialized on the compact. Only the primitive types and the types defined within the serializer will work.
5. Automatic surrogate generation tool can be used now for the surrogates.

Extreme Performance and Serialization Studio

Original CompactFormatter uses surrogates, overrides and custom serialization for the better performance. This certainly works; however writing the surrogate is always tricky and tedious business even for the experienced programmer. To make this task easier, custom serialization and surrogates were removed.

Instead now there is just one custom serialization – an override.

Override (in Angelo's Scotto terms) is the original mechanism that allows for injecting custom serialization modules for the specific class. Also CompactFormatter has been redesigned for including automatically generated binary serializers. Serialization Studio is the tool that generates binary super fast serializers (10-60 times faster than BinaryFormatter).
The tool is free and can be downloaded from here. This tool is specifically tuned to generate the surrogate for CompactFormatterPlus.

Adding the fast serializer to CompactFormatterPlus:

CompactFormatterPlus cfp = new CompactFormatterPlus(); 
MyFastSerializer mfs = new MyFastSerializer(); 
cfp.addFastSerializer(mfs); 

What if the Fast Serializer cannot serialize the object? The example of this situation can be illustrated by the code below:

public class MyClass1 
{ 
    public int SomeInteger; 
    public object UnknownObject; 
} 

If the fast serializer is generated for this class, UnknownObject has to be serialized at runtime because its type is not known at compilation time. If this happens, the generic serializer takes over the control and continues the serialization. In turn, if the CompactFormatterPlus detects the field (class) that can be serialized by the fast serializer, the control returns recursively back to the fast serializer. Actually the programmer does not have to know such intricacies of the serialization. The fast serializer (if any) only has to be added to CompactFormatterPlus. The runtime of the fast serializer consists of FSWriter and FSReader. It has some extra methods against the original Binary reader and writer. It looks a bit ugly because instead of inheriting everything from Binary reader (writer), it has its own implementation of all overloads. The reason for that is: CF compiler apparently has a bug. Overloaded Write (byte) evaluates to [presumably] Write(double). [8 bytes instead of 1]

Using the Code

On the PC side, the classes that have to be serialized should be included in the solution as the source code or the library. In the case of library, it has to be referenced in the PC project. Your class must have the same namespace name and the class name as the namespace and the class name on PDA. All the above are true for PDA.
The solutions are separate. PC files should not be used or referenced on the PDA side and vice versa. Common are just the names.

Sometimes it is possible to use CF files (EXE and DLLs) on PC and PC files on PDA. But that is a bad practice. It may work or fail with a weird and irrelevant message.
Typical scenario: During the debugging session, if you have PC generated assembly in your mobile solution, this file will drag all referenced assemblies to your PDA from PC and you will run out of memory.

CompactFormatterPlus Serializable Types

• All primitive types
• Arrays of primitive types
• DateTime
• ArrayList
• Hashtable
• List<T>
• Dictionary<T Key, T1 Value>
• DataSet
• DataTable
• Complex objects composed of the above types

Credits

The user by the name Fabien Castell has improved the typed dataset serialization and the instantiation of the objects if they are defined in different assemblies. The code can be downloaded from the original website.

New Version

If a newer version is released, it can be downloaded here.