Click here to Skip to main content
Click here to Skip to main content
Go to top

Named Binary Tag serialization

, 25 May 2013
Rate this:
Please Sign up or sign in to vote.
This article describes the file format NBT and shows how can be implemented in a real application to store data.

Introduction

The Named Binary Tag (also known as NBT) is a very simple file format that use binary tags to store data. This file format was created by Markus Persson for storing game data MineCraft.

Unfortunately, the NBT format is present in several compression formats:

  1. Uncompressed.
  2. Compressed with GZIP.
  3. Compressed with ZLIB.

The library that i designed, includes the first two cases.

Standard tag types

TagIDNamePayload size (Bytes)Description
0TagEnd0The propouse of this tag is indicates the end of the opened TagCompound.
1TagByte1A single unsigned byte.
2TagShort2A single signed short.
3TagInt4A single signed integer.
4TagLong8A single signed long.
5TagFloat4A single signed float.
6TagDouble8A single signed double.
7TagByteArrayVariableByte array. This tag is prefixed with a single signed integer that indicates the size of array.
8TagStringVariableA UTF-8 string. The string is prefixed with a single unsigned short that indicates the size of string.
9TagListVariableList of nameless tags, all tags must be same tag type. The list is prefixed with the TagID of the items it contains (just one byte), and the length of the list with a single signed integer. This list is sortable.
10TagCompoundVariableList of named tags. This list is not sortable and his size is variable (without prefixed length)
11TagIntArrayVariableInteger Array. This tag is prefixed with a single integer that indicates the size of array.

My custom tag types

To give more options to NBT file format, i created new tag types.

252TagImageVariableFor storing images. (System.Drawing.Image)
253TagIPVariableFor storing a IPAddress.
254TagMACVariableFor storing a Physical Address.
251TagSByte1For storing a signed Byte.
250TagUShort2For storing a unsigned Short.
249TagUINT4For storing a unsigned Integer.
248TagULong8For storing a unsigned Long.
247TagShortArrayVariableShort array. This tag is prefixed with a single integer that indicates the size of array.
246TagDateTime8For storing a date time value.
245TagTimeSpan8For storing a time span value.
244TagLongArrayVariableLong array. This tag is prefixed with a single integer that indicates the size of array.
243TagFloatArrayVariableFloat array. This tag is prefixed with a single integer that indicates the size of array.
242TagDoubleArrayVariableDouble array. This tag is prefixed with a single integer that indicates the size of array.
241TagSByteArrayVariableSByte array. This tag is prefixed with a single integer that indicates the size of array.
240TagUShortArrayVariableUShort array. This tag is prefixed with a single integer that indicates the size of array.
239TagUIntArrayVariableUInt array. This tag is prefixed with a single integer that indicates the size of array.
238TagULongArrayVariableULong array. This tag is prefixed with a single integer that indicates the size of array.
237TagImageArrayVariableImage array. This tag is prefixed with a single integer that indicates the size of array.

File format rules

  1. Everything is in big-endian.
  2. All NBT files must begin with TagCompound.
  3. All tags begin with a single byte that indicates his tag type.
  4. All tags (except TagEnd and the items in TagList), begins with a TagString.
  5. All tags of TagCompound must be closed by TagEnd.

Format specifications and samples

The tags contains the following format:

TagType (TagID)TagString (Name)Payload

The following sample show how this format store TagShort inside a TagCompound

Theory:

TagCompound: ('Test')
{
TagShort: ('sample') : 123
}

Data on disk (hex format):

(1) 10

(2) 00 04

(3) 54 65 73 74

(4) 02

(5) 00 06

(6) 73 61 6D 70 7C 65

(7) 00 7B(8) 00

(1) ID of TagCompound.

(2) Length of the TagCompound name.

(3) UTF-8 String ("Test").

(4) Tag ID, in this case 2 because our tag is a TagShort.

(5) Length of his name.

(6) UTF-8 String ("sample").

(7) Payload.

(8) TagEnd (indicates the end of the TagCompound).

Using the code

To use my library, it's necessary to import the following name space:

  1. NBT.IO (This name space contains everything to do with the file and its compression)
  2. NBT.Tags (Contains all supported tag types)

There are two namespaces (NBT.Exceptions and NBT.Compression).

NBT.Exceptions contains all exception that can throw the library, and NBT.Compression contains all related with the compression.

Inside the library - Part 1 (NBT.IO)

The namespace NBT.IO is responsible for matters relating to the treatment of the file, reading, writing, exceptions, ...

to read a file is necessary to create an instance of the class NBTFile. The NBTFile class provide the main methods for the administration of the file. It also detects automatically the compression format.

//
// Part of the code from NBTCompression Headers
//
public static NBTCompressionTypes.enumNBTCompressionTypes CompressionType(string filePath)
{
    NBTCompressionTypes.enumNBTCompressionTypes result = 
                NBTCompressionTypes.enumNBTCompressionTypes.Uncompressed;
    //We open the file and check if file have the header of GZIP
    using (Stream fStream = new FileStream(filePath, FileMode.Open, FileAccess.Read))
    {
        result = NBTCompressionHeaders.CompressionType(fStream);
    }
    return result;
}

public static bool IsGzipStream(Stream stream)
{
    bool result = false;
    if (stream == null)
    {
        throw new NBT_InvalidArgumentNullException();
    }
    if (stream.CanSeek == false)
    {
        throw new NBT_IOException("Can't seek in the stream");
    }
    //we keep the current position within the stream.
    long initialOffset = stream.Seek(0, SeekOrigin.Current);
    stream.Seek(0, SeekOrigin.Begin);
    //Check if the stream is a gzip stream.
    if ((stream.ReadByte() == GZIP_Header[0]) && (stream.ReadByte() == GZIP_Header[1]))
    {
        result = true;
    }
    //Set the position to the initial position
    stream.Seek(initialOffset, SeekOrigin.Begin);
    return result;
}
//
// Part of the code from NBTFile
//
public void Load(Stream stream)
{
    try
    {
        //Indicates if the stream will be closed after this function.
        bool closeAuxStream = false;
        if (stream == null)
        {
            throw new NBT_IOException();
        }
        //Determine the compression
        NBTCompressionType fileCompression = NBTCompressionHeaders.CompressionType(stream);
        Stream auxStream = null;
        switch (fileCompression)
        {
            case NBTCompressionType.Uncompressed:
                {
                    auxStream = stream;
                    closeAuxStream = false;
                    break;
                }
            case NBTCompressionType.GZipCompression:
                {
                    auxStream = new GZipStream(stream, CompressionMode.Decompress, true);
                    closeAuxStream = true;
                    break;
                }
        }
        if (auxStream == null)
        {
            throw new NBT_IOException();
        }
        byte firstTag = (byte)auxStream.ReadByte();
        if (firstTag != TagTypes.TagCompound)
        {
            throw new NBT_IOException("The first tag must be a TagCompound");
        }
        this.fileType = fileCompression;
        this.rootTagName = TagString.ReadString(auxStream);
        this.rootTagValue.readTag(auxStream);
        if (closeAuxStream == true)
        {
            //Close the auxStream, but the original stream still opened
            auxStream.Close();
        }
    }
    catch (Exception ex)
    {
        throw new NBT_IOException("Load exception", ex);
    }
}
public void Load(string filePath)
{
    try
    {
        if (File.Exists(filePath) != true)
        {
            throw new NBT_IOException("File not found");
        }
        using (Stream stream = File.OpenRead(filePath))
        {
            this.Load(stream);
            this.filePath = filePath;
        }
    }
    catch (Exception ex)
    {
        throw new NBT_IOException("Load exception", ex);
    }
}

Inside the library - Part 2 (NBT.Tags)

This namespace contains all tag types available. The main idea is that all tags inherit from the abstract class Tag.

Is so because the Tag class provide the minimum functions that must have all tags.

Because TagCompound is the first tag of a NBT file, NBTFile.Load() call a readTag (this function is in the TagCompound class).

Here is the explanation and the code:

internal override void readTag(Stream stream)
{
    if (stream == null)
    {
        throw new NBT_InvalidArgumentNullException();
    }
    //Clear the current content in the dictionary
    this.Clear();
    bool exit = false;
    while (exit != true)
    {
        //Read the tag ID
        byte id = TagByte.ReadByte(stream);
        //If tagID = 0 (TagEnd), is the end of the list and we close the list.
        if (id == TagTypes.TagEnd)
        {
            exit = true;
        }
        if (exit != true)
        {
            //Read the Key (unique name of the tag in this TagCompound list)
            string tagEntry_Key = TagString.ReadString(stream);
            //Read the value (the tag)
            //See bellow to see the ReadTag code
            Tag tagEntry_Value = Tag.ReadTag(stream, id);
            //Add the tag with its key to the dictionary inside the TagCompound
            this.value.Add(tagEntry_Key, tagEntry_Value);
        }
    }
}

This function is in the abstract class Tag, and his function is simply, create a instance of the tag that match with the id parameter.

internal static Tag ReadTag(Stream stream, byte id)
{
    switch (id)
    {
        case TagTypes.TagEnd:
            return new TagEnd();

        case TagTypes.TagByte:
            return new TagByte(stream);

        case TagTypes.TagShort:
            return new TagShort(stream);
                  .
                  .
                  .
                  .
    }
}

The idea is simple, each tag is responsible for loading your data from the input stream, and also to save them.

Sample code

frmMain.cs

//
// the following sample show how you can store
// a undefined number of TagStrings into the main TagCompound.
//

//We need import the library.
using NBT.Tags;
using NBT.IO;

public partial class frmMain : Form
{
    //We create a instance of NBTFile to manage the data file.
    NBTFile nbtFile = new NBTFile();
    //Path where we found the nbt file.
    string filePath = Application.StartupPath + @"\test.nbt";

    public frmMain()
    {
        InitializeComponent();
    }

    private void frmMain_Load(object sender, EventArgs e)
    {
        if (File.Exists(this.filePath) == true)
        {
            //We open the file using the function LoadFromFile, if you don't use
            //a file, because you use a stream, you can use the function LoadFromStream.
            this.nbtFile.Load(this.filePath);
            //Reload the list
            this.ReloadList();
        }
    }

    private void btnSave_Click(object sender, EventArgs e)
    {
        //Save the current data into the specified file.
        this.nbtFile.Save(this.filePath);
    }

    private void btnAdd_Click(object sender, EventArgs e)
    {
        //Save a new TagString into the main TagCompound. (The key must be unique)
        this.nbtFile.RootTag.Add(this.txtKey.Text, new TagString(this.txtValue.Text));
        this.ReloadList();
    }
    private void ReloadList()
    {
        this.lstItems.Items.Clear();
        //Retrieve all items stored in the main TagCompound
        foreach (KeyValuePair<string,> item in this.nbtFile.RootTag)
        {
            //Check if the tag is a TagString to retrieve its value,
            //but it isn't necessary if you use ToString()
            if (item.Value.GetType() == typeof(TagString))
            {
                ListBoxItem lstItem = new ListBoxItem();
                lstItem.Text = ((TagString)item.Value).value;
                lstItem.Tag = item.Key;
                this.lstItems.Items.Add(lstItem);
            }
        }
    }

    private void btnDelete_Click(object sender, EventArgs e)
    {
        if (this.lstItems.SelectedItems.Count > 0)
        { 
            ListBoxItem selectedItem = (ListBoxItem)this.lstItems.SelectedItem;
            //Delete the selected key
            this.nbtFile.RootTag.Remove((string)selectedItem.Tag);
            this.ReloadList();            
        }
    }
}
ListBoxItem.cs
public class ListBoxItem
{
    private string visibleText = "";
    private object itemTag = null;
    public string Text
    {
        get
        {
            return this.visibleText;
        }
        set
        {
            this.visibleText = value;
        }
    }
    public object Tag
    {
        get
        {
            return this.itemTag;
        }
        set
        {
            this.itemTag = value;
        }
    }
    public ListBoxItem()
    {

    }
    public override string ToString()
    {
        return this.visibleText;
    }
}

My free graphical tool to edit any NBT file.

You can download directly following this link to test your own NBT files. Download NBT Maker from my Skydrive (It's freeware)

Possible usages

I recently made a number of programs that use this format to store data. Among them a wake on lan program, that store the computers in directories using the TagCompound.

Conclusion

I think this format, although very simple, is quite powerful because it allows any data store. Furthermore, the fact that it is organized by name and sub ​​directories is a great feature that should be taken present to store data hierarchically.

History

  • 27 July, 2012:
    • Initial release
  • 11 August, 2012:
    • Added new 4 tag types (TagSByte | TagUShort | TagUInt | TagULong)
  • 13 October, 2012:
    • Added new 3 tag types (TagShortArray | TagDateTime | TagTimeSpan)
    • The Load/Save functions are overloaded
  • 26 December, 2012
    • Added new 8 tag types: (TagLongArray | TagFloatArray | TagDoubleArray | TagSByteArray | TagUShortArray | TagUIntArray | TagULongArray | TagImageArray)
    • Fixed minor bugs
  • 26 January, 2013:
    • New sample added (NBT Phonebook - with contact image support)
  • 31 March, 2013:
    • Fixed the TagImageArray bug
  • 25 May, 2013:
    • Added the equality function
    • Added null protection in each tag array while writing in the nbt file

Related links

License

This article, along with any associated source code and files, is licensed under The Creative Commons Attribution-Share Alike 3.0 Unported License

Share

About the Author

Alberto Molero
Software Developer
Spain Spain
No Biography provided

Comments and Discussions

 
GeneralMy vote of 5 Pinmemberibrahim ragab6-Apr-13 10:03 
GeneralNBT Maker - temporarily unavailable PinmemberAlberto Molero6-Apr-13 0:45 
GeneralMy vote of 5 PinmemberTechnoGeek00131-Mar-13 18:24 
GeneralMy vote of 5 PinmemberPrasad Khandekar31-Mar-13 10:57 
QuestionHow about binary serialization? Pinmemberibrahim ragab31-Mar-13 4:58 
AnswerRe: How about binary serialization? PinmemberAlberto Molero31-Mar-13 7:57 
GeneralMessage Removed Pinmemberibrahim ragab6-Apr-13 4:38 
GeneralRe: How about binary serialization? PinmemberAlberto Molero6-Apr-13 8:28 
GeneralRe: How about binary serialization? Pinmemberibrahim ragab6-Apr-13 10:01 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.140905.1 | Last Updated 25 May 2013
Article Copyright 2012 by Alberto Molero
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid