Click here to Skip to main content
Click here to Skip to main content
Alternative Article

UniversalSerializer

, 10 Sep 2014 Ms-RL
Rate this:
Please Sign up or sign in to vote.
A universal and easy serializer for .NET.

What is it ?

UniversalSerializer is a Serializer/Deserializer for .NET/Silverlight/PCL/Windows Phone/Windows Store.
In other words, it saves and loads complex object values/instances. The ability to serialize any type is its vocation. The stream format can be binary, JSON, or XML.

It contains DLLs for:

  • .NET 3.5, 4.0, 4.5 or later.
    [ Mono not tested, help wanted ]
  • Silverlight 4 or later.
  • Portable Class Library 4. Making it usable on Windows Store/Windows RT.
    [ Windows Store not tested, help wanted ]
  • Windows Phone 7 & 8 (Silverlight).
  • An additional DLL for specific WPF's types.
  • An additional DLL for specific Windows Form's types.

 

Summary

The objective of UniversalSerializer is to be able to serialize any type, including very complex types, with no effort.

  • No need to add attributes nor interfaces to the types (classes & structures).
  • When a class instance is referenced several times, it is serialized once only.
  • Circular references are allowed.
  • Existing serialization or transcoding mechanisms are reused. Currently: [Serializable], ISerializable, [ValueSerializer], and [TypeConverter].
  • Types with no default constructor are allowed, if a parametric constructor can be found (it is automatic).
  • Not-generic ICollection classes can be serialized if an Add or an Insert method can be found (it is automatic).

Of course, all ordinary constructions as classes, structures, inheritance, public properties, public fields, enumerations, collections, dictionaries, generics, polymorphism, etc.. are serialized by UniversalSerializer.

When a type is not serializable out-of-the-box, UniversalSerializer offers two categories of modifiers:

  • Containers (ITypeContainers): We can include the problematic type (or set of types) in a custom class that will manage its serialization and deserialization.
  • A set of filters: We can block some types, and force the serializer to store selected private fields.

My best wish is that people add more Modifiers (Containers and Filters) in the future, and maybe one day all types can be serializable.

UniversalSerializer can serialize to three stream formats: custom binary, JSON, and XML.

Security

The DLLs are safe (that is: not unsafe), they do not use pointers, not even IL code emission (except one function on the Phone 7.1 project).

Examples of usage

Example with a file name

<span id="ArticleContent">
var data = new Hashtable(); data.Add(0, 1);
using (var s = new UniversalSerializer(@"d:\temp\serialized.bin"))
{
  s.Serialize(data);
  var data2 = s.Deserialize<Hashtable>();
}</span>

That is that simple!

Example with a stream

<span id="ArticleContent">
var data = new Hashtable(); data.Add(0, 1);
using (var ms = new MemoryStream())
{ 
   var s = new UniversalSerializer(ms); 
   s.Serialize(data);  
   var data2 = s.Deserialize<Hashtable>();
}</span>

Example to XML

<span id="ArticleContent">
var data = new Hashtable(); data.Add(0, 1);
using (FileStream fs = new FileStream("TestXmlFormatter.xml", FileMode.Create)){
  Parameters parameters = new Parameters()
  { 
    Stream = fs, SerializerFormatter = SerializerFormatters.XmlSerializationFormatter 
  };
  UniversalSerializer ser = new UniversalSerializer(parameters);

  var data = new Hashtable(); data.Add(0, 1);
  ser.Serialize(data);
  var deserialized = ser.Deserialize<Hashtable>();
} </span>

Example to JSON

<span id="ArticleContent">
var data = new Hashtable(); data.Add(0, 1);
using (FileStream fs = new FileStream("TestXmlFormatter.xml", FileMode.Create))
{
  Parameters parameters = new Parameters()
  { 
    Stream = fs, SerializerFormatter = SerializerFormatters.JSONSerializationFormatter
  };
  UniversalSerializer ser = new UniversalSerializer(parameters);

  ser.Serialize(data);
  var deserialized = ser.Deserialize<Hashtable>();
} </span>

Example for WPF

There is a specialized DLL for WPF, that manages more WPF types:

<span id="ArticleContent">
var data = new System.Windows.Window() { Title = "Hello!" };
using (var s = new UniversalSerializerWPF(@"d:\temp\serialized.bin"))
{
  s.Serialize(data);
  var data2 = s.Deserialize<System.Windows.Window>();
}</span>

Example for Windows Forms

There is a specialized DLL for Windows Forms, that manages more Windows Forms types:

<span id="ArticleContent">
var data = new System.Windows.Forms.Form() { Text = "Hello!" };
using (var s = new UniversalSerializerWinForm(@"d:\temp\serialized.bin"))
{
  s.Serialize(data);
  var data2 = s.Deserialize<System.Windows.Forms.Form>();
}</span>

Documentation

In the archive, in the directory 'Documentation', there are these texts:

  • Summary [ index.html ]
  • Presentation
  • Examples
  • Solutions and projects (DLLs)
  • Good practices
  • Attributes
  • Containers
  • Filters
  • Errors on execution
    Very important for resolving problems of serialization, and for differentiating the 3 categories of errors.

 

Compared possibilities of .NET serializers

Tests are done by the Benchmark.
I do not intend to compare with all serializers, they are too many. I am more focused on technics and limitations.

class condition ↓ UniversalSerializer BinaryFormatter DataContractSerializer JavaScriptSerializer Protobuf.net SoapFormatter XmlSerializer
Not authored (not modifiable) yes no yes yes no no yes
Field yes yes yes yes yes yes yes
Readonly field yes yes yes yes yes yes no
Property yes yes yes yes yes yes yes
All primitive types yes yes yes no yes yes yes
In object yes yes no no no yes no
No default constructor yes yes yes no no yes no
Default construction needed yes no no yes yes no yes
Parametric construction needed yes no no no no no no
Reference yes yes yes no no yes no
Inheritance yes yes no no no yes no
Circular reference yes yes yes no no yes no
Circular reference in a generic list yes yes yes no no no no
Generics yes yes yes yes yes no yes
Generic dictionary yes no no no no no no
Complex type (WPF Window) yes no no no no no no

Details:

  • Not authored (not modifiable)
    The class is by another author, we can not add attributes, interfaces or even modify it.
  • Field
    A field is serialized.
  • Readonly field
    Example in C#:
    <span id="ArticleContent">
    public readonly int i;</span>
  • Property
    An automatic property and its hidden field.
  • All primitive types
    bool, DateTime, sbyte, byte, short, ushort, int, uint, long, ulong, Single, double, Decimal, char, string.
  • In object
    Main data is boxed in an Object.
  • No default constructor
    A class with a parametric constructor only.
  • Default construction needed
    A class that needs to be constructed by its default constructor.
  • Parametric construction needed
    This class has to be constructed by parameters. A default (no-param) construction leads to data corruption.
  • Reference
    Instance duplication is tested. The serializer must reference one instance several times.
  • Circular reference
    A type contains the same type as a field.
  • Circular reference in a generic list
    A type T contains a List<T> as a field.
  • Generics
    A generic class is tested.
  • Generic dictionary
    This class inherits Dictionary<int, string> and contains a value.
  • Complex type (WPF Window)
    A Window that contains some controls.

Notes:

  • BinaryFormatter, DataContractSerializer and SoapFormatter do not take constructors into account.
    That can lead to data corruption, for example when a constructor registers the instance in a static list.

For a performance test, please read this chapter.

Why and how is it done ?

I needed a universal serializer, able to serialize anything without modification.

But the existing serializers have drawbacks:

  • Some need special attributes (BinaryFormatter, Protobuf.net) or interfaces.
  • Others do not take fields into account (sharpSerializer).
  • References are a problem. Usually instances are duplicated, and deserialized objects point to different instances. Circular references are rarely managed.
  • None can manage parameter constructors when there is no default constructor, as far as I know. They use a system construction with potential side-effects.
  • Particular .NET classes need more attention but are sealed therefore can not be inherited with serialization attributes.

[ More details in the serializers comparison ]

So the solution needed a set of mechanisms and techniques. We will see these mechanisms in the next chapters.

Serialize no-default-constructor classes

In .NET, default (no-parameters) class constructors are not compulsory. But they are needed by almost all serializers. And they are frequent in the framework. Example: System.Windows.Controls.UIElementCollection.

Some serializers use FormatterServices.GetUninitializedObject, but usually when a class has a parametric constructor and no default constructor, the parametric construction is needed to correctly initialize the instance.

The solution in UniversalSerializer is to search for parametric constructors, and to find a correspondence in the class between the constructor parameters and the fields, even if these fields are private.

In UIElementCollection, we have this constructor:

<span id="ArticleContent">
public UIElementCollection( UIElement visualParent, FrameworkElement logicalParent )</span>

And these fields are available in the same class:

  • <span id="ArticleContent">
    private readonly UIElement _visualParent;</span>
  • <span id="ArticleContent">
    private readonly FrameworkElement _logicalParent;</span>

Types are equal and their name is very close. Enough to let UniversalSerializer try to create an instance with these values. And it works !

Serialize not-generic ICollection classes

For some obscure reason, the ICollection interface does not provide Add or Insert methods. In other words, it defines a read-only collection, contrary to the generic ICollection<T>. Normally, we could not deserialize a class that implements ICollection and not ICollection<T>.

Fortunately, in the real world nearly all of these classes have a Add or a Insert method, at least for internal use. UniversalSerializer finds these methods and uses them to deserialize the collection class instance.

Reuse existing mechanisms and manage them in ITypeContainers

In some cases, it is more efficient or only possible to use existing transcoding mechanisms.

When I tried to serialize WPF controls, I discovered that:

  • [TypeConverter] lets transcode the object to some types which can be serialized. (string, byte array, etc.). Example: System.Windows.Input.Cursor.
  • [ValueSerializer] lets transcode from and to a string. Example: System.Windows.Media.FontFamily.
  • [Serializable] (and ISerializable) allows the use of BinaryFormatter. Example: System.Uri.

If you consider FontFamily, you will understand that transcoding it to a string is much easier than trying to save its properties. And safer, because setting a property can lead to unpredictable consequences on unknown or complex classes.

For these attributes, I have created the mechanism of ITypeContainer. A container replaces the source instance by its transcoded value, usually a string or a byte array. A container can be applied to a set of types, for example all the attribute-compatible types.

Examples and details can be found below.

Filters

Some types need special process, which is done by a set of filters.

Type validator filter

This filter allows you to prevent UniversalSerializer from serializing some problematic types.

For example, I met some classes using System.IntPtr. Serializing this type only leads to problems since they are only used internally in the classes, even when they are stored in public properties/fields.

Private fields adder filter

This filter tells the serializer to add a particular private field to the serialization data.

For example, System.Windows.Controls.Panel needs _uiElementCollection to fill its Children property, since Children is read-only. With the filter, the solution is easy. And any type that inherits Panel, such as StackPanel, will benefit this filter.

ForcedParametricConstructorTypes

It is not a filter but a list of types. When a type is in this list, UniversalSerializer ignores its default (not parametric) constructor and searches for a parametric constructor. Example: System.Windows.Forms.PropertyManager. It is much easier to use its parametric constructor than to write a ITypeContainer for this type.

CanTestDefaultConstructor (new in version 2.14.3)

UniversalSerializer usually tries to build one instance per type using its default constructor (when available). The problem is some types should not be constructed outside deserialization, for example when their constructor increments a static counter. This filter lets UniversalSerializer avoid this construction test.

DefaultConstructorTestCleaner (new in version 2.14.3)

UniversalSerializer usually tries to build one instance per type using its default constructor (when available). Some types, as System.Windows.Window, needs a cleaner to be called before instance destruction. In the example of WPF Window, we have to call Close(), otherwise the application will not be closed correctly (it waits for all WPF windows to be closed).

References

Consider this code:

<span id="ArticleContent">
var data = new System.Windows.Controls.TextBox[2];
data[0] = new System.Windows.Controls.TextBox() { Text = "TextBox1" };
data[1] = data[0]; // Same reference
using (var s = new UniversalSerializerWPF(@"d:\temp\serialized.bin"))
{
  s.Serialize(data);
  var data2 = s.Deserialize<System.Windows.Controls.TextBox[]>();
  data2[0].Text = "New text"; // Affects the two references.
  bool sameReference = object.ReferenceEquals(data2[0], data2[1]);
}</span>
  1. UniversalSerializer serializes only one instance of the TextBox.
  2. It deserializes only one instance of TextBox, and creates two references pointing to it. Proofs: sameReference is true, and Text is identical in both references.

Custom modifiers: ITypeContainer and filters

Given that the main objective of UniversalSerializer is to be able to serialize any type, it is essential we share our experience.

So I tried to make the containers and filters creation as easy as possible, to help you play with them and share your solutions.

Making a ITypeContainer [ Modified on version 3.14.5 ]

The objective is to replace the problematic instance by an easy instance. In general, the container will contain very simple types (string, int, byte array, etc..).

Let's take an example:

<span id="ArticleContent">
/// <summary>
/// . No default (no-param) constructor.
/// . The only constructor has a parameter with no corresponding field.
/// . ATextBox has a private 'set' and is different type from constructor's parameter.
/// </summary>
public class MyStrangeClassNeedsACustomContainer
{
    /// <summary>
    /// It is built from the constructor's parameter.
    /// Since its 'set' method is not public, it will not be serialized directly.
    /// </summary>
    public TextBox ATextBox { get; private set; }
    public MyStrangeClassNeedsACustomContainer(int NumberAsTitle)
    {
        this.ATextBox = new TextBox() { Text = NumberAsTitle.ToString() };
    }
}</span>

As written in the summary, this class causes some difficulties to the serializer(s).

To overcome the problem, we create a container:

<span id="ArticleContent">
class ContainerForMyStrangeClass : ITypeContainer
{
    #region Here you add data to be serialized in place of the class instance

    public int AnInteger; // We store the smallest, sufficient and necessary data.

    #endregion Here you add data to be serialized in place of the class instance


    public ITypeContainer CreateNewContainer(object ContainedObject)
    {
        MyStrangeClassNeedsACustomContainer sourceInstance = ContainedObject as MyStrangeClassNeedsACustomContainer;
        return new ContainerForMyStrangeClass() { AnInteger = int.Parse(sourceInstance.ATextBox.Text) };
    }

    public object Deserialize()
    {
        return new MyStrangeClassNeedsACustomContainer(this.AnInteger);
    }

    public bool IsValidType(Type type)
    {
        return Tools.TypeIs(type, typeof(MyStrangeClassNeedsACustomContainer));
    }

    public bool ApplyEvenIfThereIsAValidConstructor
    {
        get { return false; }
    }

    public bool ApplyToStructures
    {
        get { return false; }
    }
}</span>

A detail: all methods behave as static methods (but are not), except Deserialize().

Let's see them more in details:

  • <span id="ArticleContent">
    public int AnInteger</span>
    It is not part of the ITypeContainer interface. Here we will store the information we will need later at deserialization.
  • <span id="ArticleContent">
    ITypeContainer CreateNewContainer(object ContainedObject)</span>
    Used during serialization. This is a kind of constructor for this container instance. The parameter will be the source class instance to serialize.
  • <span id="ArticleContent">
    object Deserialize()</span>

    Used during deserialization. The container instance will produce a new instance, a copy of the source instance, using our field AnInteger.

  • <span id="ArticleContent">
    bool IsValidType(Type type)</span>
    Used during serialization. Returns true is the type inherits from, or is, the source type. This is a filter. We can choose to accept inherited types or not, to accept several compatible types, etc..
  • <span id="ArticleContent">
    bool ApplyEvenIfThereIsAValidConstructor</span>
    Used during serialization. Returns true if this container applies to class types with a default (no-param) constructor. Can be useful to very general containers.
  • <span id="ArticleContent">
    bool ApplyToStructures</span>
    Used during serialization. Returns true if this container applies to structure types, and not only class types. Can be useful to very general containers.

Steps are:

  1. The serializer checks if the source type (MyStrangeClassNeedsACustomerContainer) is managed by a container. Our container class (ContainerForMyStrangeClass) answers yes, via IsValidType(), ApplyEvenIfThereIsANoParamConstructor and ApplyToStructures.
  2. The serializer builds an instance of our container, via CreateNewContainer(). CreateNewContainer builds an instance and sets its field AnInteger.
  3. The serializer stores (serializes) this container instance in place of the source instance.
  4. The deserializer retrieves (deserializes) the container instance.
  5. The deserializer calls Deserialize() and obtains a copy of the source class instance. Deserialize() creates this copy using its field AnInteger.

[ New on Version 3.14.5 ] We declare the CustomModifiers :

<span id="ArticleContent">
public class CustomContainerTestModifiers : CustomModifiers
{
  public CustomContainerTestModifiers()
    : base(Containers: new ITypeContainer[] {
      new ContainerForMyStrangeClass() // ?-----
      })
  {
  }
}</span>

This declaration will automatically be found by the deserializer. [ New in version 3.14.5 ]

Now we serialize it: [ Simplified on version 3.14.5 ]

<span id="ArticleContent">
/* This example needs a custom ITypeContainer.
Normally, this class can not be serialized (see details in its source).
But thanks to this container, we can serialize the class as a small data (an integer).
 */

var data = new MyStrangeClassNeedsACustomContainer(123);

using (MemoryStream ms = new MemoryStream())
{
  var p = new Parameters() { Stream = ms };
  UniversalSerializer ser = new UniversalSerializer(p);

  ser.Serialize(data);
  var data2 = ser.Deserialize<MyStrangeClassNeedsACustomContainer>();

  bool ok = data2.ATextBox.Text == "123";
}</span>

As you can see, the implementation is very easy.

Tool help functions

The Tools static class offers some help:

  • <span id="ArticleContent">
    Type Tools.TypeIs(Type ObjectType, Type SearchedType)</span>
    It is equivalent to the C#'s 'is', but for Types. For example, TypeIs((typeof(List<int>), typeof(List<>)) returns true.
  • <span id="ArticleContent">
    Type DerivedType(Type ObjectType, Type SearchedType)</span>

    Returns the type corresponding to SearchedType that is inherited by ObjectType. For example, DerivedType(typeof(MyList), typeof(List<>)) returns typeof(List<int>) when MyList is

    <span id="ArticleContent">
    MyList: List<int> { }.</span>
  • <span id="ArticleContent">
    FieldInfo FieldInfoFromName(Type t, string name)</span>
    Returns the FieldInfo of the named field of the type. We will use it in the next chapter.

Making a set of filters [ Modified on version 3.14.5 ]

Please note the filter mechanism is totally independent from the ITypeContainers. They can be used together, or separately.

Let's take an example:

<span id="ArticleContent">
public class ThisClassNeedsFilters
{
  public ShouldNotBeSerialized Useless;
  private int Integer;
  public string Value { get { return this.Integer.ToString(); } }
  public ThisClassNeedsFilters()
  {
  }
  public ThisClassNeedsFilters(int a)
  {
    this.Integer = a;
    this.Useless = new ShouldNotBeSerialized();
  }
}
public class ShouldNotBeSerialized
{
}</span>

This class (ThisClassNeedsFilters) have some problems:

  • It contains a ShouldNotBeSerialized. Let's imagine the class ShouldNotBeSerialized has to be avoided for some reasons, I don't know why, maybe it is poisoned!
  • The field Integer is not public and therefore is ignored by the serializer(s).
  • Even the constructor parameter name is different from any field or property. Anyway the serializer does not need this constructor, as it already has a default constructor.

To overcome these problems, we write a custom set of filters:

<span id="ArticleContent">
/// <summary>
/// Tells the serializer to add some certain private fields to store the type.
/// </summary>
static FieldInfo[] MyAdditionalPrivateFieldsAdder(Type t)
{
    if (Tools.TypeIs(t, typeof(ThisClassNeedsFilters)))
        return new FieldInfo[] { Tools.FieldInfoFromName(t, "Integer") };
    return null;
}
/// <summary>
/// Returns 'false' if this type should not be serialized at all.
/// That will let the default value created by the constructor of its container class/structure.
/// </summary>
static bool MyTypeSerializationValidator(Type t)
{
    return ! Tools.TypeIs(t, typeof(ShouldNotBeSerialized));
}</span>

They are self-explanatory:

  • <span id="ArticleContent">
    FieldInfo[] MyAdditionalPrivateFieldsAdder(Type t)</span>
    makes the serializer add a private field (Integer) to every source instance of this type (ThisClassNeedsFilters).
  • <span id="ArticleContent">
    bool MyTypeSerializationValidator(Type t)</span>
    prevents the serializer from storing any instance of this type (ShouldNotBeSerialized). Consequently, any instance of ThisClassNeedsFilters will not set the Useless field.

[ New on Version 3.14.5 ] We declare the CustomModifiers :

<span id="ArticleContent">
public class CustomFiltersTestModifier : CustomModifiers
{
  public CustomFiltersTestModifier()
    : base(FilterSets : new FilterSet[] {
      new FilterSet() { 
        AdditionalPrivateFieldsAdder=MyAdditionalPrivateFieldsAdder, 
        TypeSerializationValidator=MyTypeSerializationValidator } })
  {        
  }
}</span>

This declaration will automatically be found by the deserializer. [ New in version 3.14.5 ]

Now we serialize it: [ Simplified on Version 3.14.5 ]

<span id="ArticleContent">
/* This example needs custom filters.
Normally, this class can be serialized but with wrong fields.
Thanks to these filters, we can serialize the class appropriately.
 */

using (MemoryStream ms = new MemoryStream())
{
  var p = new Parameters() { Stream = ms };
  var ser = new UniversalSerializer(p);

  var data = new ThisClassNeedsFilters(123);
  ser.Serialize(data);
  var data2 = ser.Deserialize<ThisClassNeedsFilters>();

  bool ok = data2.Value == "123" && data2.Useless == null;
}</span>

The implementation is even easier than with ITypeContainer.

The source code of UniversalSerializer

All sources are C#.

Solutions can be compiled with Visual Studio 2010, 2012 and 2013. Compilation only needs .NET 4, Silverlight 4, or PCL 4, not any third-party DLLs, the source code is complete.

DLLs

There are 9 available DLLs:

  • UniversalSerializer3.dll: The main general DLL for .NET 4.0, with very few framework dependencies.

  • UniversalSerializer3.dll: The same general DLL, for .NET 4.5.
  • UniversalSerializer3.dll: The same general DLL, for .NET 3.5.

  • UniversalSerializer3ForSilverlight.dll: For Silverlight 5 or later. It should be compilable for Silverlight 4 (not tested).

  • UniversalSerializerPortableLib3.dll: A Portable Class Library (PCL) you can use on many frameworks. Tested under .NET. It should be usable under Windows Store 8, Windows RT in general, MonoDroid, etc, but since I do use these frameworks I let you try and give feedback. Smile | :)

  • UniversalSerializer3ForWindowsPhone7.dll: For Windows Phone 7.5+ (Silverlight). Please read important notes in its directory.
  • UniversalSerializer3ForWindowsPhone8.dll: For Windows Phone 8 (Silverlight).

  • UniversalSerializerWPF3.dll: A specialized DLL for WPF, that manages specific WPF types.

  • UniversalSerializerWinForm3.dll: A specialized DLL for WinForms, that manages specific Windows Forms types.

Solutions

.NET 4.0

  • "UniversalSerializer Lib\UniversalSerializer.csproj" This project compiles the main library. You can add this project to your own solution.
  • "UniversalSerializer Desktop Debug Libs\UniversalSerializer Desktop Debug Libs.sln"
    This solution compiles the 4 DLLs for the desktop, in debug mode.
  • "UniversalSerializer Desktop Release Libs\UniversalSerializerReleaseLibs.sln"
    This solution compiles the 4 DLLs for the desktop, in release mode.

.NET 4.5

  • "UniversalSerializer Lib .NET 4.5\UniversalSerializer.csproj"
    This project compiles the main library, optimized for .NET 4.5. You can add this project to your own solution.

.NET 3.5

  • ".NET 3.5\UniversalSerializer Lib .NET 3.5\UniversalSerializer.sln"
    This solution compiles the main library, adapted to .NET 3.5.

Silverlight

  • "UniversalSerializer Silverlight\UniversalSerializer Silverlight.sln"
    This solution contains a test application for Silverlight and the corresponding DLL project.

PCL (Portable Class Library)

  • "UniversalSerializerPortableLib\UniversalSerializerPortableLib.sln"
    This solution contains a test application and the PCL (Portable Class Library) project.
    Note: if the project refuses to load in VS, edit it, make sure this text is all on one line: "<ProjectTypeGuids></ProjectTypeGuids>", then reload the project.

Windows Forms

  • "UniversalSerializerWinForm\UniversalSerializerWinForm.sln"
    This solution contains a test application for Windows forms and the corresponding DLL projects (the general DLL and a specialized DLL for WinForm types).

WPF

  • "UniversalSerializerWPF\UniversalSerializerWPF Tester\UniversalSerializer Tester.sln"
    This solution contains a test application for WPF and the corresponding DLL projects (the general DLL and a specialized DLL for WPF types).

Windows Phone (Silverlight)

  • "UniversalSerializer Windows Phone 8\UniversalSerializer Windows Phone 8.sln"
    This solution contains a test application for Silverlight for Windows Phone 8 and the corresponding DLL project.
  • "UniversalSerializer Windows Phone 7.1 experimental\UniversalSerializer Windows Phone 7.1.sln"
    This solution compiles for WP 7.1 and later. Tested in VS 2010 & 2012 Phone emulators. Please read notes "EXPERIMENTAL.txt", that is important on WP 7.1.

Windows Store

  • "UniversalSerializer Windows Store 8 PCL\UniversalSerializer Windows Store 8 PCL.sln"
    This solution contains a test application for Windows Store 8 using the portable library.
    Not tested.

Applications

  • "UniversalSerializer Benchmark\UniversalSerializer Benchmark.sln"
    This application compares resource consumption of 9 serializers, including 3 versions of UniversalSerializer (binary, XML and JSON).

Remarks

  • Some source files are used in various projects. You should not delete directories nor files, you could lose a needed source file.
  • There is no compiled files in this archive, except external serializers for the Benchmark.
  • All projects were made under Visual Studio 2012, and are compatible with Visual Studio 2010 and 2013.
  • Some solution directories contain additional informations in text files, please read them.

Important points

  • Versioning: The serialization/deserialization process identifies types by their complete name (from type.AssemblyQualifiedName). This kind of name depends on the type's assembly version, therefore please take care of framework and DLL versioning!
  • File format: The current stream format is subject to changes in the future. For that reason, DLL names have a version number. If you serialize to a file, I suggest you to add a version number to the file name.

Version 1

The first version was based on fastBinaryJSON and extended it to allow serialization of many types, using modifiers (containers and filters). The resulting source code was not optimized. It was more a prototype, or a proof-of-concept, than a well-designed software. It was slow, ate a lot of memory and produced big files, but it was able to serialize very complex data at least.

Version 2

New source code

I have written a new serializer, in fact a new software, from nothing. The source code does not contain anything from FastBinaryJSON any more.

Differences with the old FastBinaryJSON-based version 1 are:

  • 70 X faster, files are 100 X smaller, it needs 110 X less RAM. As measured in the benchmark (200 k items, 5 loops, 3 bytes structure). Now this serializer is fast and efficient.
  • The serializer works with streams, not with byte arrays. The objective are to consume very few memory and to write to disk (or to any stream destination) as quick as possible. An important consequence is we can serialize to a compressed stream directly, there is no intermediary process or structures. I see that as essential in order to serialize very big data or collections.
  • Three file formats are supported: custom binary, JSON and XML. And you can create your own formats using the Formatter interface.
  • A new multiplex binary format. This format has been designed with .NET structures in mind, and to facilitate, accelerate and reduce size of the serialized data. This format integrates the concepts of collections, dictionaries, properties, fields, instances, and references. Type descriptors and object values are multiplexed in the same stream. Please note this format has nothing in common with the one of FastBinaryJSON, therefore is not compatible with UniversalSerializer version 1.
  • Some improvements over version 1:
    • Parametric constructors have now priority over containers for structures (example: KeyValuePair<,>).
    • Nullable<T> is managed correctly.
    • Integers can optionally be compressed as 7-bits variable length. Please note that uncompressed integers can occasionally be compressed better by an external compressor (as WinRar).

New license

As the code is completely genuine, contrary to the version 1, I have had the possibility to change the license. I chose Ms-RL, a license that let you use the source code in any project. The only notable condition is if you modify a source file, you have to set this modification public (only the file you modified, not any other files). I hope that will help to share containers and improvements that can be useful to many.

Compatibility with files from version 1

Since the formats are completely different, there is no compatibility at all. If you want to read data you wrote in the old format, I suggest you to reference the old "UniversalSerializer1.dll". DLLs version 1 and 2 can be linked to the same application, since they use different name spaces.

Adapt your sources to the new API

There are a few differences from API of version 1:

  • The name space is now UniversalSerializer2.
  • The serializer now works with streams.
  • Now you have to make a new instance of UniversalSerializer first, then call Serialize and/or Deserialize().
  • ITypeContainer.ApplyEvenIfThereIsANoParamConstructor has been changed for ITypeContainer.ApplyEvenIfThereIsAValidConstructor.

Version 3

The file stream format now includes a modifiers' assembly list. This way, the deserializer always knows what modifiers have been used during serialization.

This important information in the stream is the reason of structural modifications in the way modifiers are declared and used. Please read examples in the corresponding upper chapters.

Performances

During development, I needed to compare this serializer with the existing serializers, in order to eliminate possible weaknesses in my code. I created a simple benchmark that counts the resource needs: processor, stream and RAM. Although benchmarks can be discussed indefinitely, because they are far from perfection and they depend on many parameters, they are useful. We only have to not forget they are not very precise.

The data we will serialize and deserialize as a test is an array of a small structure constituted by 3 bytes. The array contains 200,000 items of this structure, and we serialize and deserialize it 5 times. Read below a chapter giving more details about the test conditions.

Note: this benchmark can serialize much more complex types and situations. You can select them in the UI.

Processor



UniversalSerializer is listed 4 times: binary format, JSON format, XML format and the old version 1. The slowest of all serializers is the old version 1.0 of UniversalSerializer, which was based on FastBinaryJSON but with a big additional resource waste (shame on me !).

File length



The produced file length can be important to your project. That is why I did an effort in order to reduce the data waste. In fact UniversalSerializer tries to eliminate all superfluous data in its structure. In this example, a perfect uncompressed file length would be 600,000 bytes long. UniversalSerializer is only 338 bytes upper, due to the type descriptors.

Protobuf-net is a bit particular because it seems to do a kind of compression. That is more evident with another data structure containing 3 Int32 (in place of 3 bytes): Protobuf-net produces the same file length: about 8,8 bytes per structure. For information, by default UniversalSerializer compresses integers (except bytes) with a 7-bits compression scheme and produces a 897,750 bytes file when serializing the 3 Int32 structure array (that is 4,5 bytes/structure on average). This compression can be switched off.

The old UniversalSerializer version 1 goes on being ridiculous. I let it in the diagram, it is too funny !

RAM



Counting the RAM consumption is a tough and uncertain job. Here I present the GC Memory consumption, using System.GC.GetTotalMemory(..). It should be noted that we obtain different numbers using System.Diagnostics.Process.GetCurrentProcess().WorkingSet64. But after many tries, it seems the GC method is more stable and significant.

I did a particular effort in UniversalSerializer to reduce the RAM consumption. The result seems to reflect these efforts. Once again, the old UniversalSerializer version 1 produces astronomical numbers, consuming about 700 Mio of RAM. What a joke ! That is 110 times the consumption of the new version 2. That makes me laugh. Wink | ;)

An important lesson I leaned here is: the bigger is the array we serialize, the bigger will be the resources consumed by a serializer, in fact a lot more than the source data's size. This lesson seems to be valid for all serializers (yes, including UniversalSerializer).

Whole resources



Here we try to evaluate the whole resource consumption of each serializer. Numbers are percentages based on resource consumption of UniversalSerializer (binary). The Total number is the mean of the three resource percentages. For example, Protobuf-net: time=137%, file length=277%, ram=208%, mean=(137+277+208)/3=207 %.

The usual joke is the total resource consumption of the old UniversalSerializer version 1: 9,187 %. Bigger than an elephant ! It was too enormous for the diagram, I had to cut-it off. Wink | ;)

Conclusion

My personal conclusion is I reached my goals: create a resource-saving serializer that let us serialize many kinds of types. The XML and JSON formatters consume too many resources, but I know I can improve them greatly by replacing the .NET classes currently in use.

Test conditions

Some details about conditions of this test.

The benchmark project sources can be found in the main source archive of UniversalSerializer. That lets you examine it and tell me if you think something is wrong in my methodology.

A capture of this application:



The structure we serialize in this example:

<span id="ArticleContent">
public struct MyByteColor{
    public byte R;
    public byte G;
    public byte B;
}</span>

Please note other types can be tested in the benchmark. Notably WPF's Window: a very complex set of types

Some points to be considered:

  • Tests are run once only (to avoid pre-caching). Note: A second run of UniverslSerializer is much quicker since all type analysis is done already. But I did not cheat, these diagrams show first runs only. Smile | :)
  • Windows is set to not use any virtual memory files, to avoid disk accesses when a lot of memory is needed by a serializer.
  • The file stream is written to a RAM-disk (to avoid disk accesses). This RAM-disk is formatted as FAT-32, to avoid indexing or other interfering accesses.
  • GC memory is continuously (every 60 ms) measured by a timer in another thread, in order to obtain its peak value and not only its final value.

History

Version 3.14.10.11, 2014-10-14

  • Added: Error descriptions to the documentation.
  • Added: "Good practices" to the documentation.
  • Improved: Error numbering connected to the documentation.
  • Improved: incorrect external converters in .NET do not block containers anymore.
  • Added: more WPF types are managed now.
  • Added: separated WinForm test solution for .NET 3.5 .
  • Improved: More types in System.Windows.Media are now serialized.
  • Corrected: Main DLL for .NET 3.5 had a problem in a pure .NET 3.5 solution.

Version 3.14.9.10, 2014-09-10

  • Improved: speed of type analysis.
  • Corrected: "ForceSerializeAttribute" did not work correctly on fields.
  • Corrected: in Tester solution, "ForceSerializeOnPrivateFieldAndProperty.ForcedPrivateField" was not a field. (oups!)

Version 3.14.7.9, 2014-07-25

  • New: TypeMismatchException, when type casting is wrong after deserialization.
  • New: When deserializing in a loop, sends EndOfStreamException in the end.
  • Added: Many new test structures in the Benchmark.
  • Corrected: Private properties marked by ForceSerialize are now serialized.
  • Corrected: exception in loop serialization test with JavascriptSerializer in Benchmark.
  • Corrected: PCL for Windows 8 now has the new attributes.
  • Changed: version numbering is: Main.Year-2000.Month.Release#.

Version 3.14.6.2, 2014-06-10

  • New directory "Documentation" with some html texts.
  • New attributes "ForceSerializeAttribute" and "ForceNotSerializeAttribute".
  • New: takes "EditorBrowsableAttribute" into account to not serialize the field or property. That helps with Windows Forms.
  • Corrected problems in type analysis when using filters on some conditions.

Version 3.14.6, 2014-06-04

  • Added: new columns in the tables that benchmark produces: "Bytes/ms", "Data/file lengths", "data length/GC memory", "data length/Working set".
  • Modified: Minor changes in the benchmark's UI.
  • Improved: exception message in CLRBinaryFormatterContainer.
  • New: test serie 'DifficultiesTests()' in Tester solution.
  • New solution "UniversalSerializer Lib .NET 4.5", optimized for .NET 4.5 using "AggressiveInlining".
  • New solution "UniversalSerializer Lib .NET 3.5", less optimized than solutions for .NET 4.x, but can be necessary.
  • Corrected: the .NET 4.0 solution "UniversalSerializer Lib" has no attribute "AggressiveInlining" any more. No more incompatibility with computers where .NET 4.5 is not installed.
  • Corrected: In the benchmark, DataContractSerializer now can serialize instances and circular references.

Version 3.14.5.2 revision 2, 2014-05-16

  • Some corrections.

Version 3.14.5.2, 2014-05-13

  • Corrected: problem with circular types.
  • Added: new types in Test solution.
  • Added: new types in Benchmark.

Version 3.14.5, 2014-05-05

  • Modified: API and stream formats are version 3.0.
  • Modified: Namespaces are renamed. UniversalSerializerLib2 -> UniversalSerializerLib3.
  • Modified: Modifiers (filters & containers) are declared as classes now, and automatically found by serializer.
  • Added: option Parameters.ModifiersAssemblies to declare assemblies that define modifiers. Useful when modifiers can not be automatically found by serializer.
  • Modified: Stream format version is now 3.0. It can not be read by previous UniversalSerializer (version 2.x)'s DLL.
  • Improved: Saves DateTime.Kind in DateTimes.
  • Improved: Benchmark's interface has been clarified a bit.
  • Improved: All projects pass VS's code analysis with no error.
  • New: test structures (as PrimitiveValueTypesStructure).
  • New: Logs warnings in the IDE.
  • New: solution "UniversalSerializer Windows Store 8 PCL".
  • New: solution "UniversalSerializer Windows Phone 7.1 experimental". Not perfect, but can be useful.
  • Removed: option CustomModifiers.DoNotDuplicateStrings. Strings are always saved as references, in stream format 3.0.
  • Removed: FastJSON and FastBinaryJSON from the benchmark solution, because they do not pass the new deserialized type checks.
  • Removed: old UniversalSerializer version 1 from the benchmark solution.
  • Corrected: Some regressions in benchmark.
  • Note: DLL version 3 can read and modify version 2 streams, but not version 1 streams.

Version 2.14.3, 2014-03-17

  • New filter: CanTestDefaultConstructor.
  • New filter: DefaultConstructorTestCleaner. Useful for WPF's System.Windows.Window.
  • New container: DependencyPropertyContainer. Uses the right static DependencyProperty.
  • Improved: benchmark tells when file paths does not exist.
  • Improved: CLRTypeConverterContainer.cs checks type conversion capability first.
  • Modified: benchmark is now 64 bits, as the majority of installed Windows now.
  • Fixed: test solution for Windows Phone 8 now works well.
  • Fixed: problem with public readonly fields as constructor parameters.
  • Fixed: some translations in the source code.
  • Note: File formats are compatible with version 2.0 .

Version 2.0, 2013-10-09

  • Brand new serializer. All code is new, no code from version 1.0.
  • 70 X faster, files are 100 X shorter, and it needs 110 X less RAM that the old version 1.
  • Note: file formats have nothing in common with version 1.0. DLL version 2.0 can not read files written with version 1.0.

Version 1.0, 2013-07-13

This version was based on FastBinaryJSON, with many extensions and modifications.

Future

I would like to improve some aspects:

  • Add more Containers and Filters. You can help me. Tell me when types are not serialized correctly. And please share your modifiers (containers and filters).
  • Make a DLL for Windows Runtime : better use Portable library (PCL).

Make the serializers' job easier

This experience taught me why serializers have difficulties with some types and how we can facilitate its job when we author a class.

  1. Write a default (no-param) constructor when possible. The serializer will reconstruct the instance from its fields and properties.
  2. If you can not write a default (no-param) constructor, write a public parametric constructor with corresponding private fields. The field should have same type and same name. With UniversalSerializer, the name can be slightly different: Param -> param or _param or _Param.
  3. Implement a ValueSerializerAttribute or a TypeConverterAttribute when an instance can be constructed from something as simple as a string. Especially when your class contains many public optimisation fields or properties. For example, FontFamily can be constructed from a simple string, no matter if it contains many other informations, they all come from this simple string. I don't know if many serializers take these attributes into accounts, but at least UniversalSerializer does.
  4. All optimization (as caching) fields or properties should be private, when possible. See above.
  5. When you create an object collection class, please implement IList. Because ICollection does not allow to add items to the collection.
  6. When you create a generic collection class, please implement ICollection<>. Because IEnumerable<> does not allow to add items to the collection.

Thanks

  • I thank Mehdi Gholam for his fastBinaryJSON. I learned interesting things reading his code. Thank you for sharing it.
  • Thank the Mono team also, I used two functions of this nice framework.
  • And I thank the Protobuf-net team, not only because I used one function of their work, but because their serializer forced me to improve mine a lot also. A useful competition. Smile | :)
    Note: UniversalSerializer does not derive in anything from Protobuf-net, except this only one small function.

Previous versions

FAQ

  • Q: All errors (exceptions and debugger messages)
    A: Please read the text "Documentation\Errors.html", that should help you a lot.
  • Q: Error "Can not find type or namepace UniversalSerializer3" during compilation.
    A : Ensure your project is for the same framework version as the one of the DLL.

How to report errors, difficulties or problems

If you receive an error number, you will find more details in "Documentation\Errors.html" in the archive.

Because of the particularity of a serializer, you have to differentiate what kind of error or problem you have.
There are 3 very different kinds of error:

  1. You have designed a type in a way that makes it incompatible with UniversalSerializer.
    Please read Good practices, that may help you in adapting your type.
    If the type comes from an external DLL, please inform its author about UniversalSerializer.
  2. A framework (.NET, Silverlight, Mono, etc) type is not managed correctly.
    It may require an adaptation in UniversalSerializer. I do not consider that as a true error, it only means a standard type needs more consideration (creation of filters and/or containers).
  3. A general problem in UniversalSerializer.
    That will let me correct a problem that potentially affect many programmers.

In general, problems of the first category can be solved by yourself.
Problems of category 2 should be solved by me since every programmer will benefit the improvement. Although you can solve it by yourself and inform me later.
Problems of category 3 require my attention.

I need to reproduce the problem you have. When the problematic class is very short, you can let a question in the section Comments and Discussions. But when your classes are very complex, it is better you send me a complete Solution. I wrote a message below that lets you send me a private reply. Send me an email address and I will send you mine. Of course, we do not want to publish email addresses here, due to spam.

Other useful information can be:

  • What version of the DLLs do you use ?
  • The DLLs are compiled as debug, or as release ?
  • Do you link to a specific types DLL (for WPF or for WinForms) ?
  • What type of project is it ? WPF, Console, WinForm, Silverlight, Windows Store, Mono, etc ? -> please note there are different DLLs in the archive, notably for .NET, PCL, Silverlight and Windows Phone.
  • What version of Visual Studio (or other IDE) do you use ?
  • What version of .NET is installed on your system ?

Suggestions:

  • In case of problem, debugging will be made easier when you add the DLLs' project to your own solution, and not only link their compiled file.
  • Simplify as much as possible your class, until you determine the exact data that is problematic.
  • Inner exceptions often give more useful information than the upper exception.

License

This article, along with any associated source code and files, is licensed under Microsoft Reciprocal License

Share

About the Author

Hi !
 
I made my first program on a Sinclair ZX-81.
Since that day, I have the virus of computers. Smile | :)
 
Some articles in my blog:
https://chrisbertrandprogramer.wordpress.com/

Comments and Discussions

 
GeneralEntity Framework [modified] PinmemberChristophe Bertrand16-May-14 23:04 
GeneralRe: Entity Framework Pinmembermph6216-May-14 23:37 
GeneralRe: Entity Framework PinmemberChristophe Bertrand17-May-14 0:38 
QuestionUnzip errors Pinmemberscosta_FST19-Mar-14 1:13 
AnswerRe: Unzip errors PinmemberChristophe Bertrand19-Mar-14 2:01 
GeneralRe: Unzip errors Pinmemberscosta_FST19-Mar-14 2:15 
GeneralRe: Unzip errors PinmemberChristophe Bertrand19-Mar-14 2:28 
AnswerRe: Unzip errors [modified] PinmemberChristophe Bertrand19-Mar-14 2:06 
GeneralRe: Unzip errors Pinmemberscosta_FST19-Mar-14 2:29 
GeneralRe: Unzip errors PinmemberChristophe Bertrand20-Mar-14 7:24 
GeneralRe: Unzip errors Pinmemberscosta_FST21-Mar-14 0:05 
QuestionPerformance PinmemberKenneth Kasajian18-Mar-14 17:30 
AnswerRe: Performance PinmemberChristophe Bertrand19-Mar-14 1:50 
QuestionCompile errors PinprofessionalHarold Chattaway18-Mar-14 5:33 
this could be a very useful utility, but I can't get it to compile..keep getting this:
Error	4	Error emitting 'System.Runtime.CompilerServices.MethodImplAttribute' attribute -- 'Incorrect argument value.'	C:\Users\nbku0x0\Documents\UniversalSerializer\UniversalSerializer Lib\BinarySerializationFormatter.cs	104	5	UniversalSerializer
It comes up 32 times...
 
iam using the UniversalSerializerReleaseLibs solution.
 
thanks
Harold
AnswerRe: Compile errors Pinmemberscosta_FST19-Mar-14 1:02 
AnswerRe: Compile errors PinmemberChristophe Bertrand19-Mar-14 1:35 
GeneralRe: Compile errors Pinmemberscosta_FST19-Mar-14 2:09 
GeneralRe: Compile errors PinmemberChristophe Bertrand19-Mar-14 2:31 
GeneralRe: Compile errors PinmemberChristophe Bertrand4-Jun-14 22:51 
QuestionNuGet package PinmemberSirTrolle18-Mar-14 5:04 
AnswerRe: NuGet package PinmemberChristophe Bertrand19-Mar-14 1:21 
GeneralNoise Pinmemberkarel_tandd17-Mar-14 23:51 
GeneralRe: Noise PinmemberChristophe Bertrand19-Mar-14 1:18 
GeneralRe: Noise Pinmemberkarel_tandd17-Apr-14 18:40 
GeneralRe: Noise PinmemberChristophe Bertrand2-May-14 6:45 
QuestionInteresting work [modified] PinmemberShuqian Ying17-Mar-14 17:08 
AnswerRe: Interesting work PinmemberChristophe Bertrand19-Mar-14 1:14 
GeneralRe: Interesting work PinmemberShuqian Ying20-Mar-14 12:27 
AnswerRe: Interesting work PinmemberChristophe Bertrand21-Mar-14 6:17 
GeneralLove it Pinprofessionalcivil_monkey717-Mar-14 11:58 
GeneralRe: Love it PinmemberChristophe Bertrand19-Mar-14 0:54 
QuestionFormatting Fixed PinprotectorMarco Bertschi17-Mar-14 10:53 
AnswerRe: Formatting Fixed PinmemberChristophe Bertrand17-Mar-14 11:11 
GeneralRe: Formatting Fixed PinprotectorMarco Bertschi17-Mar-14 11:19 
GeneralGreat Article PinprofessionalPaw Jershauge9-Oct-13 22:06 
GeneralRe: Great Article PinmemberChristophe Bertrand10-Oct-13 23:07 
GeneralMy vote of 5 PinmemberChampion Chen9-Oct-13 17:36 
GeneralRe: My vote of 5 PinmemberChristophe Bertrand10-Oct-13 23:03 
QuestionFormatterServices.GetUnInitializedobjeect PinmemberAlex Sieberer27-Aug-13 12:05 
AnswerRe: FormatterServices.GetUnInitializedobjeect PinmemberChristophe Bertrand28-Aug-13 0:02 
GeneralRe: FormatterServices.GetUnInitializedobjeect PinmemberAlex Sieberer30-Aug-13 11:23 
GeneralRe: FormatterServices.GetUnInitializedobjeect PinmemberChristophe Bertrand30-Aug-13 23:09 
GeneralMy vote of 5 PinprofessionalH. Mueller26-Jul-13 5:12 
GeneralMy vote of 5 Pinmembersam.hill14-Jul-13 20:50 
GeneralRe: My vote of 5 PinmemberChristophe Bertrand16-Jul-13 10:00 
GeneralMy vote of 5 PinmemberThorsten Bruning14-Jul-13 0:21 
GeneralRe: My vote of 5 PinmemberChristophe Bertrand16-Jul-13 9:59 
GeneralMy vote of 5 PinmemberBillWoodruff13-Jul-13 18:41 
GeneralRe: My vote of 5 PinmemberChristophe Bertrand13-Jul-13 23:07 
GeneralMy vote of 5 PinmvpMehdi Gholam13-Jul-13 6:17 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.1411023.1 | Last Updated 10 Sep 2014
Article Copyright 2013 by Christophe Bertrand
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid