Click here to Skip to main content
Click here to Skip to main content

fastBinaryJSON

, 2 May 2014
Rate this:
Please Sign up or sign in to vote.
A binary JSON serializer based on fastJSON (support for MonoDroid)

Preface

The code is now on CodePlex under Git source control (http://fastbinaryjson.codeplex.com/).

Introduction

fastBinaryJSON is based on my fastJSON article (http://www.codeproject.com/Articles/159450/fastJSON) and code which is a polymorphic object serializer. The main purpose for fastBinaryJSON is speed in serializing and deserializing data for the use in data transfer and storage to disk. It was created for my upcoming RaptorDB - Document Database engine, for performance.

Features

fastBinaryJSON has the following feature list:

  • Based on fastJSON code base (very fast, polymorphic)
    • Supports : HashTables, Dictionary, Generic Lists, Datasets, ...
  • Typically 2-10% faster on serialize, 17%+ faster on deserialize.

Why?

Why another serializer you may ask, why not just use fastJSON? The answer to this is simple : performance. JSON while a great format has the following problem:

  • JSON is a text format, so you loose type information on serializing which makes deserializing the data again time consuming.

Why not BSON?

Why not just use BSON (http://bsonspec.org/) you may ask? The answer is the following:

  • Looking at the specifications on the above site, you feel overwhelmed as it is hard to follow.
  • You feel that the specs have evolved over time and a lot of the coding parts have been deprecated.
  • BSON encodes lengths into the stream which inflate the data, this might be fine for the use case the authors envisioned, but for data transfer and storage it just makes things larger than they need to be.
  • Because of the length prefixes, the encoding of the data object must be done in two passes, once to output the data, and a second time to set the length prefixes.

I initially started off by doing a BSON conversion on fastJSON but it got too complicated, so it was scrapped.

How is data encoded in fastBinaryJSON?

JSON is an extremely simple format, so fastBinaryJSON takes that simplicity and add the needed parts to do binary serialization. fastBinaryJSON follows the same rules as the JSON specification (http://json.org) with the following table showing how data is encoded:

As you can see from the above all the encoding rules are the same as JSON and primitive data types have been given 1 byte tokens for encoding data. So the general format is :

TOKEN,  { DATA } : where DATA can be 0 or more bytes

Strings can be encoded in 2 ways, as UTF8 or Unicode, where UTF8 is more space efficient and Unicode is faster.

String keys or property names are encoded as a special UTF8 stream which is limited to 255 bytes in length to save space (you should not have a problem with this as most property names are short in length).

Performance tests

To get a sense of the performance differences in fastBinaryJSON against fastJSON the following tests were performed, times are in milliseconds, each test was done on 1000 objects and repeated 5 times, the AVG column is the average of the test excluding the first which is skewed by initialization times:


As you can see in the DIFF column which is [ fastJSON / fastBinaryJSON ] the serializer performs at least 2% faster and the deserializer at least 17% faster, with the greatest difference being with DataSet types which are a lot of rows of data.

Using the code

To use fastBinaryJSON you can use the following code samples:

byte[] bytes = fastBinaryJSON.BJSON.Instance.ToJSON(obj);
byte[] bytes = fastBinaryJSON.BJSON.Instance.ToJSON(obj, true, true); // optimized dataset, unicode strings

object obj = fastBinaryJSON.BJSON.Instance.ToObject(bytes);
SalesInvoice obj = fastBinaryJSON.BJSON.Instance.ToObject<SalesInvoice>(bytes); // type is known 

There are 3 parameters which control how the serialization is done which can be set on the Instance or can be done by a per call basis like the above examples :

UseOptimizedDatasetSchemaUse an optimized format to serialze the Dataset Schema (default = True)
ShowReadOnlyPropertiesSerialize ReadOnly Properties (default = False)
UseUnicodeStrings

Use Unicode encoding for strings (default = True)

Appendix v1.3.14.1 - Parametric Constructors

As of this version fastBinaryJSON can now handle deserializing parametric constructor classes without a default constructors, like:

public class pctor
{
      public pctor(int a) // pctor() does not exist
      {
      }
}

Now to do this fastBinaryJSON is using the FormatterServices.GetUninitializedObject(type) in the framework which essentially just allocates a memory region for your type and gives it to you as an object by passing all initializations including the constructor. While this is really fast, it has the unfortunate side effect of ignoring all class initialization like default values for properties etc. so you should be aware of this if you are restoring partial data to an object (if all the data is in json and matches the class structure then you are fine).

To control this you can set the ParametricConstructorOverride to true in the BJSONParameters.

Appendix v1.4.0 - Circular References & Breaking changes

As of this version I fixed a design flaw since the start which was bugging me, namely the removal of the BJSON.Instance singleton. This means you type less to use the library which is always a good thing, the bad thing is that you need to do a find replace in your code.

Also I found a really simple and fast way to support circular reference object structures. So a complex structure like the following will serialize and deserialize properly ( the unit test is CircularReferences()):

var o = new o1 { o1int = 1, child = new o3 { o3int = 3 }, o2obj = new o2 { o2int = 2 } };
o.o2obj.parent = o;
o.child.child = o.o2obj; 

To do this fastBinaryJSON replaces the circular reference with :

{"$i" : number } // number is an index for the internal reference
also a $circular : true is added to the top of the binary json for the deserializer to know.

Previous Versions

Below is a list of the previous versions of fastBinaryJSON.

History

  • Initial Release : 25th March 2012
  • Update : 26th March 2012
    • Added link to codeplex source control
  • Update v1.1 : 26th May 2012
    • bug fix datetime to local time
    • added BJSONParameters
    • added global types (reverse format from fastJSON to overcome size limits on name strings)
  • Update v1.2 : 24th July 2012
    • refactored reflection code into Reflection class
    • added support for top level struct object serialize/deserialize
  • Update v1.3 : 11th August 2012
    • bug fix reflection code
    • added unit tests
    • changed ArrayList to List<object>
    • updated code to be similar to fastJSON
    • deserialize ToObject< Dictionary<T,V> >
    • deserialize ToObject< List<T> >
    • added FillObject method
  • Update v1.3.1 : 16th August 2012
    • bug fix $types and arrays
    • optimize writing $types
  • Update v1.3.2 : 7th September 2012
    • null object serialize fixed
    • added sealed keyword to classes
    • bug fix SerializeNullValues=false and an extra comma at the end
    • FillObject nested types
  • Update v1.3.3 : 17th September 2012
    • bug fix deserialize zero length arrays
    • test for German locale numbers
  • Update v1.3.4 : 20th September 2012
    • singleton uses ThreadStatic for concurrency (thanks to Philip Jander)
    • bug fix extra comma in the output when only 1 property in the object (thanks to Philip Jander)
  • Update v1.3.5 : 16th November 2012
    • added support for root level DataSet and DataTable deserialize (you have to do ToObject<DataSet>(...) )
    • added dataset tests
    • added MonoDroid project
  • Update v1.3.7 : 20th April 2013
    • customtype is now builtin
    • added UseUTCTimes property for datetime
    • switched to properttype enum instead of booleans
    • using switch instead of linked if statements
    • unified DynamicMethod between silverlight and full .net
    • SafeDictionary lock fixes
  • Update v1.3.8 : 19th August 2013
    • added serialization of static fields and properties
    • fixed disabling extensions in the output
    • fixed serializing anonymous types
    • added support for dynamic objects
  • Update v1.3.9 : 27th August 2013
    • fix dynamic objects and lists
    • fix deserialize Dictionary<T, List<V>> and Dictionary<T, V[]>
    • added tests for dictionary with lists
  • Update v1.3.10 : 11th September 2013
    • fixed hastable deserialize
    • added test for hashtable
    • changed list of getters to array ~3% performance gain
    • removed unused code
  • Update v1.3.11 : 2nd November 2013
    • added signed assembly
    • version numbers will stay at 1.0.0.0 for drop in compatibility
    • file version will reflect the build number
    • bug fix deserializing to dictionaries instead of dataset when type is not defined
    • access inner property in arrays in dynamic types e.g. d.arr[1].a
  • Update v1.3.12 : 23rd November 2013
    • bug fix dynamic json and root arrays e.g. [1,2,3,4]
    • bug fix objects in array dynamic types e.g. [1,2,{"prop":90}]
    • added support for special collections : StringDictionary, NameValueCollection
  • Update v1.3.13 : 10th January 2014
    • fixed working with const properties and fields (i.e ignored)
  • Update v1.3.14 : 22nd March 2014
    • fixed create enum from value and string
    • replaced safedictionary with dictionary for some of the internals so no locks on read
    • added custom ignore attributes (Thanks to Jared Thirsk)
    • using IsDefined instead of GetCustomAttributes (Thanks to Andrew Rissing)
    • moved all the reflection code out of BJSON.cs
    • now you can deserialize non default constructor classes (Thanks to Anton Afanasyev)
  • Update v1.3.14.1 : 29th March 2014
    • added ParametricConstructorOverride parameter to control non default constructors
  • Update v1.4.0 : 7th April 2014
    • *breaking change* : removed the BJSON.Instance singleton
    • moved all the state from BJSON to the Reflection singleton
    • all of the BJSON interface is now static
    • added BJSONParameters overloads for ToObject()
    • support for circular referenced object structures
    • added circular test
  • Update v1.4.1 : 2nd May 2014
    • bug fix obj.List<List<object>> and obj.List<object[]>
    • added code intellisense help for methods
    • added ClearReflectionCache() to reset all internal structures

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

Mehdi Gholam
Architect
United Kingdom United Kingdom
Mehdi first started programming when he was 8 on BBC+128k machine in 6512 processor language, after various hardware and software changes he eventually came across .net and c# which he has been using since v1.0.
He is formally educated as a system analyst Industrial engineer, but his programming passion continues.
 
* Mehdi is the 5th person to get 6 out of 7 Platinums on CodeProject (13th Jan'12)

Comments and Discussions

 
GeneralMy vote of 5 PinmemberTejas Vaishnav21-Sep-12 2:57 
GeneralRe: My vote of 5 PinmemberStrange_Pirate29-Sep-12 0:17 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web03 | 2.8.140709.1 | Last Updated 2 May 2014
Article Copyright 2012 by Mehdi Gholam
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid