Click here to Skip to main content
Click here to Skip to main content

fastBinaryJSON

, 25 May 2012
Rate this:
Please Sign up or sign in to vote.
A binary JSON serializer based on fastJSON
This is an old version of the currently published article.

Preface 

The code is now on CodePlex under Git source control (http://fastbinaryjson.codeplex.com/).

Introduction 

fastBinaryJSON is based on my fastJSON article (http://www.codeproject.com/Articles/159450/fastJSON) and code which is a polymorphic object serializer. The main purpose for fastBinaryJSON is speed in serializing and deserializing data for the use in data transfer and storage to disk. It was created for my upcoming RaptorDB -  Document Database engine, for performance.

Features

fastBinaryJSON has the following feature list: 

  • Based on fastJSON code base (very fast, polymorphic)
    • Supports : HashTables, Dictionary, Generic Lists, Datasets, ...
  • Typically 2-10% faster on serialize, 17%+ faster on deserialize. 

Why?

Why another serializer you may ask, why not just use fastJSON? The answer to this is simple : performance. JSON while a great format has the following problem:

  • JSON is a text format, so you loose type information on serializing which makes deserializing the data again time consuming. 

Why not BSON?

Why not just use BSON (http://bsonspec.org/) you may ask? The answer is the following:

  • Looking at the specifications on the above site, you feel overwhelmed as it is hard to follow.
  • You feel that the specs have evolved over time and a lot of the coding parts have been deprecated.
  • BSON encodes lengths into the stream which inflate the data, this might be fine for the use case the authors envisioned, but for data transfer and storage it just makes things larger than they need to be.
  • Because of the length prefixes, the encoding of the data object must be done in two passes, once to output the data, and a second time to set the length prefixes.

I initially started off by doing a BSON conversion on fastJSON but it got too complicated, so it was scrapped.  

How is data encoded in fastBinaryJSON?  

JSON is an extremely simple format, so fastBinaryJSON takes that simplicity and add the needed parts to do binary serialization. fastBinaryJSON follows the same rules as the JSON specification (http://json.org) with the following table showing how data is encoded: 

As you can see  from the above all the encoding rules are the same as JSON and primitive data types have been given 1 byte tokens for encoding data. So the general format is :

TOKEN,  { DATA } : where DATA can be 0 or more bytes

Strings can be encoded in 2 ways, as UTF8 or Unicode, where UTF8 is more space efficient and Unicode is faster.

String keys or property names are encoded as a special UTF8 stream which is limited to 255 bytes in length to save space (you should not have a problem with this as most property names are short in length).

Performance tests

To get a sense of the performance differences in fastBinaryJSON against fastJSON the following tests were performed, times are in milliseconds, each test was done on 1000 objects and repeated 5 times, the AVG column is the average of the test excluding the first which is skewed by initialization times:


As you can see in the DIFF column which is [ fastJSON / fastBinaryJSON ] the serializer performs at least 2% faster  and the deserializer at least 17% faster, with the greatest difference being with DataSet types which are a lot of rows of data.

Using the code 

To use fastBinaryJSON you can use the following code samples:

byte[] bytes = fastBinaryJSON.BJSON.Instance.ToJSON(obj);
byte[] bytes = fastBinaryJSON.BJSON.Instance.ToJSON(obj, true, true); // optimized dataset, unicode strings

object obj = fastBinaryJSON.BJSON.Instance.ToObject(bytes);
SalesInvoice obj = fastBinaryJSON.BJSON.Instance.ToObject<SalesInvoice>(bytes); // type is known 

There are 3 parameters which control how the serialization is done which can be set on the Instance or can be done by a per call basis like the above examples :

UseOptimizedDatasetSchemaUse an optimized format to serialze the Dataset Schema (default = True)
ShowReadOnlyPropertiesSerialize ReadOnly Properties (default = False)
UseUnicodeStringsUse Unicode encoding for strings (default = True)

History    

  • Initial Release : 25th March 2012
  • Update : 26th March 2012
    • Added link to codeplex source control
  • Update v1.1 : 26th May 2012
    • bug fix datetime to local time
    • added BJSONParameters
    • added global types (reverse format from fastJSON to overcome size limits on name strings)

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

Mehdi Gholam
Architect
United Kingdom United Kingdom
Mehdi first started programming when he was 8 on BBC+128k machine in 6512 processor language, after various hardware and software changes he eventually came across .net and c# which he has been using since v1.0.
He is formally educated as a system analyst Industrial engineer, but his programming passion continues.
 
* Mehdi is the 5th person to get 6 out of 7 Platinums on CodeProject (13th Jan'12)

Comments and Discussions


Discussions posted for the Published version of this article. Posting a message here will take you to the publicly available article in order to continue your conversation in public.
 
QuestionAbility to read document from stream at position x Pinmemberronnyek3-Jul-14 5:21 
AnswerRe: Ability to read document from stream at position x Pinmemberronnyek3-Jul-14 5:46 
AnswerRe: Ability to read document from stream at position x PinmvpMehdi Gholam3-Jul-14 7:00 
BugAccessing internally visible members Pinmemberbetovene14-May-14 7:03 
GeneralRe: Accessing internally visible members PinmvpMehdi Gholam14-May-14 7:08 
GeneralMy vote of 5 PinpremiumVolynsky Alex22-Mar-14 8:50 
QuestionHave you seen MessagePack? PinmemberJaredThirsk10-Mar-14 3:58 
AnswerRe: Have you seen MessagePack? PinmvpMehdi Gholam10-Mar-14 4:13 
Questionexcellent: combining fastBinaryJSON with the MiniLZO class from Mehdi's Raptor DB produces great results ! PinmemberBillWoodruff15-Jul-13 3:56 
AnswerRe: excellent: combining fastBinaryJSON with the MiniLZO class from Mehdi's Raptor DB produces great results ! PinmvpMehdi Gholam15-Jul-13 5:30 
GeneralRe: excellent: combining fastBinaryJSON with the MiniLZO class from Mehdi's Raptor DB produces great results ! PinmemberBillWoodruff15-Jul-13 18:42 
GeneralRe: excellent: combining fastBinaryJSON with the MiniLZO class from Mehdi's Raptor DB produces great results ! PinmvpMehdi Gholam15-Jul-13 18:50 
GeneralRe: excellent: combining fastBinaryJSON with the MiniLZO class from Mehdi's Raptor DB produces great results ! PinmemberBillWoodruff17-Jul-13 3:31 
Questionfyi: the default settings for BJSON.Instance.Parameters, and a test example's Byte[] size results [modified] PinmemberBillWoodruff15-Jul-13 1:51 
AnswerRe: fyi: the default settings for BJSON.Instance.Parameters, and a test example's Byte[] size results PinmvpMehdi Gholam15-Jul-13 1:59 
Questiongood news ... and ... correction to syntax of using FBJ in current article ? PinmemberBillWoodruff9-Jul-13 21:43 
GeneralSerialize more types [modified] Pinmemberchprogmer6-Jun-13 8:04 
GeneralRe: Serialize more types PinmvpMehdi Gholam6-Jun-13 20:13 
Suggestiona very simple (de)serialization filter Pinmemberchprogmer6-Jun-13 23:50 
GeneralRe: Serialize more types [modified] PinmemberBillWoodruff10-Jul-13 18:26 
GeneralRe: Serialize more types Pinmemberchprogmer11-Jul-13 20:54 
GeneralRe: Serialize more types PinmemberBillWoodruff13-Jul-13 17:34 
GeneralRe: Serialize more types PinmemberChristophe Bertrand13-Jul-13 2:43 
GeneralRe: Serialize more types PinmemberBillWoodruff13-Jul-13 17:38 
Suggestionsuggestion: add a parameter, MissingConstructorFaultTolerent Pinmemberchprogmer4-Jun-13 8:32 
GeneralRe: suggestion: add a parameter, MissingConstructorFaultTolerent PinmvpMehdi Gholam4-Jun-13 21:27 
SuggestionRe: suggestion: add a parameter, MissingConstructorFaultTolerent Pinmemberchprogmer4-Jun-13 22:28 
GeneralRe: suggestion: add a parameter, MissingConstructorFaultTolerent PinmvpMehdi Gholam22-Mar-14 7:53 
GeneralMy vote of 5 PinmemberTejas Vaishnav21-Sep-12 2:57 
GeneralRe: My vote of 5 PinmemberStrange_Pirate29-Sep-12 0:17 
GeneralGlad to see that some still think performance is important PinmvpEspen Harlinn16-Aug-12 12:47 
GeneralRe: Glad to see that some still think performance is important PinmvpMehdi Gholam16-Aug-12 20:39 
Questionmention of your work on Scott Hanselman's blog PinmemberBillWoodruff11-Aug-12 6:30 
AnswerRe: mention of your work on Scott Hanselman's blog PinmvpMehdi Gholam11-Aug-12 7:26 
AnswerRe: mention of your work on Scott Hanselman's blog [modified] Pinmembercycnus11-Aug-12 17:02 
GeneralMy vote of 5 PinmemberJF201511-Aug-12 3:06 
GeneralMy vote of 5 PinmemberAlluvialDeposit23-Jul-12 23:52 
GeneralRe: My vote of 5 PinmvpMehdi Gholam23-Jul-12 23:56 
QuestionDateTime Parse Pinmembercqwydz20-May-12 6:24 
AnswerRe: DateTime Parse Pinmembercqwydz20-May-12 6:31 
AnswerRe: DateTime Parse PinmvpMehdi Gholam20-May-12 7:33 
QuestionParseArray Bug? Pinmembercqwydz19-May-12 2:41 
AnswerRe: ParseArray Bug? Pinmembercqwydz19-May-12 2:49 
AnswerRe: ParseArray Bug? PinmvpMehdi Gholam19-May-12 4:00 
GeneralRe: ParseArray Bug? Pinmembercqwydz20-May-12 6:56 
GeneralRe: ParseArray Bug? PinmvpMehdi Gholam20-May-12 7:34 
Questionjavascript ? Pinmemberchristoph braendle30-Apr-12 1:20 
AnswerRe: javascript ? PinmvpMehdi Gholam30-Apr-12 1:31 
GeneralRe: javascript ? Pinmemberchristoph braendle30-Apr-12 2:21 
QuestionIDictionary<string, object>? PinmemberHuisheng Chen17-Apr-12 15:14 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.140721.1 | Last Updated 26 May 2012
Article Copyright 2012 by Mehdi Gholam
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid