65.9K
CodeProject is changing. Read more.
Home

Named Binary Tag serialization

Alberto M.
starIconstarIconstarIconstarIcon
emptyStarIcon
starIcon

4.93/5 (23 votes)

Jul 26, 2012

CC (ASA 3U)

6 min read

viewsIcon

69228

downloadIcon

2067

This article describes the file format NBT and shows how can be implemented in a real application to store data.

 

Introduction

The Named Binary Tag (also known as NBT) is a very simple file format that use binary tags to store data. This file format was created by Markus Persson for storing game data MineCraft.

Unfortunately, the NBT format is present in several compression formats:

  1. Uncompressed.
  2. Compressed with GZIP.
  3. Compressed with ZLIB.

 

Standard tag types

TagID Name Payload size (Bytes) Description
0 TagEnd 0 The propouse of this tag is indicates the end of the opened TagCompound.
1 TagByte 1 A single unsigned byte.
2 TagShort 2 A single signed short.
3 TagInt 4 A single signed integer.
4 TagLong 8 A single signed long.
5 TagFloat 4 A single signed float.
6 TagDouble 8 A single signed double.
7 TagByteArray Variable Byte array. This tag is prefixed with a single signed integer that indicates the size of array.
8 TagString Variable A UTF-8 string. The string is prefixed with a single unsigned short that indicates the size of string.
9 TagList Variable List of nameless tags, all tags must be same tag type. The list is prefixed with the TagID of the items it contains (just one byte), and the length of the list with a single signed integer. This list is sortable.
10 TagCompound Variable List of named tags. This list is not sortable and his size is variable (without prefixed length)
11 TagIntArray Variable Integer Array. This tag is prefixed with a single integer that indicates the size of array.

My custom tag types

To give more options to NBT file format, i created new tag types.

252 TagImage Variable For storing images. (System.Drawing.Image)
253 TagIP Variable For storing a IPAddress.
254 TagMAC Variable For storing a Physical Address.
251 TagSByte 1 For storing a signed Byte.
250 TagUShort 2 For storing a unsigned Short.
249 TagUINT 4 For storing a unsigned Integer.
248 TagULong 8 For storing a unsigned Long.
247 TagShortArray Variable Short array. This tag is prefixed with a single integer that indicates the size of array.
246 TagDateTime 8 For storing a date time value.
245 TagTimeSpan 8 For storing a time span value.
244 TagLongArray Variable Long array. This tag is prefixed with a single integer that indicates the size of array.
243 TagFloatArray Variable Float array. This tag is prefixed with a single integer that indicates the size of array.
242 TagDoubleArray Variable Double array. This tag is prefixed with a single integer that indicates the size of array.
241 TagSByteArray Variable SByte array. This tag is prefixed with a single integer that indicates the size of array.
240 TagUShortArray Variable UShort array. This tag is prefixed with a single integer that indicates the size of array.
239 TagUIntArray Variable UInt array. This tag is prefixed with a single integer that indicates the size of array.
238 TagULongArray Variable ULong array. This tag is prefixed with a single integer that indicates the size of array.
237 TagImageArray Variable Image array. This tag is prefixed with a single integer that indicates the size of array.

File format rules

  1. Everything is in big-endian.
  2. All NBT files must begin with TagCompound.
  3. All tags begin with a single byte that indicates his tag type.
  4. All tags (except TagEnd and the items in TagList), begins with a TagString.
  5. All tags of TagCompound must be closed by TagEnd.

Format specifications and samples

The tags contains the following format:

TagType (TagID) TagString (Name) Payload

The following sample show how this format store TagShort inside a TagCompound

Theory:

TagCompound: ('Test')
{
TagShort: ('sample') : 123
}

Data on disk (hex format):

(1) 10

(2) 00 04

(3) 54 65 73 74

(4) 02

(5) 00 06

(6) 73 61 6D 70 7C 65

 (7) 00 7B (8) 00

(1) ID of TagCompound.

(2) Length of the TagCompound name.

(3) UTF-8 String ("Test").

(4) Tag ID, in this case 2 because our tag is a TagShort.

(5) Length of his name.

(6) UTF-8 String ("sample").

(7) Payload.

(8) TagEnd (indicates the end of the TagCompound).

Using the code

To use my library, it's necessary to import the following name space:

  1. NBT.IO (This name space contains everything to do with the file and its compression)
  2. NBT.Tags (Contains all supported tag types)

There are two namespaces (NBT.Exceptions and NBT.Compression).

NBT.Exceptions contains all exception that can throw the library, and NBT.Compression contains all related with the compression.

Inside the library - Part 1 (NBT.IO)

The namespace NBT.IO is responsible for matters relating to the treatment of the file, reading, writing, exceptions, ...

to read a file is necessary to create an instance of the class NBTFile. The NBTFile class provide the main methods for the administration of the file. It also detects automatically the compression format.

//
// Part of the code from NBTCompression Headers
//
public static NBTCompressionTypes.enumNBTCompressionTypes CompressionType(string filePath)
{
    NBTCompressionTypes.enumNBTCompressionTypes result = 
                NBTCompressionTypes.enumNBTCompressionTypes.Uncompressed;
    //We open the file and check if file have the header of GZIP
    using (Stream fStream = new FileStream(filePath, FileMode.Open, FileAccess.Read))
    {
        result = NBTCompressionHeaders.CompressionType(fStream);
    }
    return result;
}

public static bool IsGzipStream(Stream stream)
{
    bool result = false;
    if (stream == null)
    {
        throw new NBT_InvalidArgumentNullException();
    }
    if (stream.CanSeek == false)
    {
        throw new NBT_IOException("Can't seek in the stream");
    }
    //we keep the current position within the stream.
    long initialOffset = stream.Seek(0, SeekOrigin.Current);
    stream.Seek(0, SeekOrigin.Begin);
    //Check if the stream is a gzip stream.
    if ((stream.ReadByte() == GZIP_Header[0]) && (stream.ReadByte() == GZIP_Header[1]))
    {
        result = true;
    }
    //Set the position to the initial position
    stream.Seek(initialOffset, SeekOrigin.Begin);
    return result;
}
//
// Part of the code from NBTFile
//
public void Load(Stream stream)
{
    try
    {
        //Indicates if the stream will be closed after this function.
        bool closeAuxStream = false;
        if (stream == null)
        {
            throw new NBT_IOException();
        }
        //Determine the compression
        NBTCompressionType fileCompression = NBTCompressionHeaders.CompressionType(stream);
        Stream auxStream = null;
        switch (fileCompression)
        {
            case NBTCompressionType.Uncompressed:
                {
                    auxStream = stream;
                    closeAuxStream = false;
                    break;
                }
            case NBTCompressionType.GZipCompression:
                {
                    auxStream = new GZipStream(stream, CompressionMode.Decompress, true);
                    closeAuxStream = true;
                    break;
                }
        }
        if (auxStream == null)
        {
            throw new NBT_IOException();
        }
        byte firstTag = (byte)auxStream.ReadByte();
        if (firstTag != TagTypes.TagCompound)
        {
            throw new NBT_IOException("The first tag must be a TagCompound");
        }
        this.fileType = fileCompression;
        this.rootTagName = TagString.ReadString(auxStream);
        this.rootTagValue.readTag(auxStream);
        if (closeAuxStream == true)
        {
            //Close the auxStream, but the original stream still opened
            auxStream.Close();
        }
    }
    catch (Exception ex)
    {
        throw new NBT_IOException("Load exception", ex);
    }
}
public void Load(string filePath)
{
    try
    {
        if (File.Exists(filePath) != true)
        {
            throw new NBT_IOException("File not found");
        }
        using (Stream stream = File.OpenRead(filePath))
        {
            this.Load(stream);
            this.filePath = filePath;
        }
    }
    catch (Exception ex)
    {
        throw new NBT_IOException("Load exception", ex);
    }
}

Inside the library - Part 2 (NBT.Tags)

This namespace contains all tag types available. The main idea is that all tags inherit from the abstract class Tag.

Is so because the Tag class provide the minimum functions that must have all tags.

Because TagCompound is the first tag of a NBT file, NBTFile.Load() call a readTag (this function is in the TagCompound class).

Here is the explanation and the code:

internal override void readTag(Stream stream)
{
    if (stream == null)
    {
        throw new NBT_InvalidArgumentNullException();
    }
    //Clear the current content in the dictionary
    this.Clear();
    bool exit = false;
    while (exit != true)
    {
        //Read the tag ID
        byte id = TagByte.ReadByte(stream);
        //If tagID = 0 (TagEnd), is the end of the list and we close the list.
        if (id == TagTypes.TagEnd)
        {
            exit = true;
        }
        if (exit != true)
        {
            //Read the Key (unique name of the tag in this TagCompound list)
            string tagEntry_Key = TagString.ReadString(stream);
            //Read the value (the tag)
            //See bellow to see the ReadTag code
            Tag tagEntry_Value = Tag.ReadTag(stream, id);
            //Add the tag with its key to the dictionary inside the TagCompound
            this.value.Add(tagEntry_Key, tagEntry_Value);
        }
    }
}

This function is in the abstract class Tag, and his function is simply, create a instance of the tag that match with the id parameter.

internal static Tag ReadTag(Stream stream, byte id)
{
    switch (id)
    {
        case TagTypes.TagEnd:
            return new TagEnd();

        case TagTypes.TagByte:
            return new TagByte(stream);

        case TagTypes.TagShort:
            return new TagShort(stream);
                  .
                  .
                  .
                  .
    }
}

The idea is simple, each tag is responsible for loading your data from the input stream, and also to save them.

Sample code

frmMain.cs

//
// the following sample show how you can store
// a undefined number of TagStrings into the main TagCompound.
//

//We need import the library.
using NBT.Tags;
using NBT.IO;

public partial class frmMain : Form
{
    //We create a instance of NBTFile to manage the data file.
    NBTFile nbtFile = new NBTFile();
    //Path where we found the nbt file.
    string filePath = Application.StartupPath + @"\test.nbt";

    public frmMain()
    {
        InitializeComponent();
    }

    private void frmMain_Load(object sender, EventArgs e)
    {
        if (File.Exists(this.filePath) == true)
        {
            //We open the file using the function LoadFromFile, if you don't use
            //a file, because you use a stream, you can use the function LoadFromStream.
            this.nbtFile.Load(this.filePath);
            //Reload the list
            this.ReloadList();
        }
    }

    private void btnSave_Click(object sender, EventArgs e)
    {
        //Save the current data into the specified file.
        this.nbtFile.Save(this.filePath);
    }

    private void btnAdd_Click(object sender, EventArgs e)
    {
        //Save a new TagString into the main TagCompound. (The key must be unique)
        this.nbtFile.RootTag.Add(this.txtKey.Text, new TagString(this.txtValue.Text));
        this.ReloadList();
    }
    private void ReloadList()
    {
        this.lstItems.Items.Clear();
        //Retrieve all items stored in the main TagCompound
        foreach (KeyValuePair<string,> item in this.nbtFile.RootTag)
        {
            //Check if the tag is a TagString to retrieve its value,
            //but it isn't necessary if you use ToString()
            if (item.Value.GetType() == typeof(TagString))
            {
                ListBoxItem lstItem = new ListBoxItem();
                lstItem.Text = ((TagString)item.Value).value;
                lstItem.Tag = item.Key;
                this.lstItems.Items.Add(lstItem);
            }
        }
    }

    private void btnDelete_Click(object sender, EventArgs e)
    {
        if (this.lstItems.SelectedItems.Count > 0)
        { 
            ListBoxItem selectedItem = (ListBoxItem)this.lstItems.SelectedItem;
            //Delete the selected key
            this.nbtFile.RootTag.Remove((string)selectedItem.Tag);
            this.ReloadList();            
        }
    }
}
ListBoxItem.cs
public class ListBoxItem
{
    private string visibleText = "";
    private object itemTag = null;
    public string Text
    {
        get
        {
            return this.visibleText;
        }
        set
        {
            this.visibleText = value;
        }
    }
    public object Tag
    {
        get
        {
            return this.itemTag;
        }
        set
        {
            this.itemTag = value;
        }
    }
    public ListBoxItem()
    {

    }
    public override string ToString()
    {
        return this.visibleText;
    }
}

My free graphical tool to edit any NBT file.

You can download directly following this link to test your own NBT files. Download NBT Maker from my Skydrive (It's freeware)

Possible usages

I recently made a number of programs that use this format to store data. Among them a wake on lan program, that store the computers in directories using the TagCompound.

Conclusion

I think this format, although very simple, is quite powerful because it allows any data store. Furthermore, the fact that it is organized by name and sub ​​directories is a great feature that should be taken present to store data hierarchically.

History

  • 27 July, 2012:
    • Initial release
  • 11 August, 2012:
    • Added new 4 tag types (TagSByte | TagUShort | TagUInt | TagULong)
  • 13 October, 2012:
    • Added new 3 tag types (TagShortArray | TagDateTime | TagTimeSpan)
    • The Load/Save functions are overloaded
  • 26 December, 2012
    • Added new 8 tag types: (TagLongArray | TagFloatArray | TagDoubleArray | TagSByteArray | TagUShortArray | TagUIntArray | TagULongArray | TagImageArray)
    • Fixed minor bugs
  • 26 January, 2013:
    • New sample added (NBT Phonebook - with contact image support)
  • 31 March, 2013:
    • Fixed the TagImageArray bug
  • 25 May, 2013:
    • Added the equality function
    • Added null protection in each tag array while writing in the nbt file
  • 18 Oct, 2014:
    • Fixed TagCompound clone
    • Fixed TagEnd equals
    • Updated to Microsoft .Net 4.5.1
    • Added ZLib Compression
  • 16 Feb, 2015:
    • Fixed problem when compressing data using ZLib with an emtpy tagString
    • Some minor changes

Related links