Parallelizable File Formats

Introduction

I have been recently thinking about possible improvements of the game development process, particularly content creation. While content creation in principle is well parallelizable, some unnecessary blocks happen because of data formats, which are usually created for runtime efficiency.

I understand the reasoning behind ultra-fast "memory dump" binary formats which can be quickly loaded without massaging data in memory. However, while these formats are perfect for the final product, they serialize development, because in practice only a single person can modify resource stored in such a format.

Distinct, "decomposed" formats are unavoidable while the game is being developed. Of course, this has its cost in iteration times for a single developer (either longer loading times or need to bake the "editable" format into "deployable" one before each run), but hopefully we can make up for this by enabling several artists to work on a single resource at once.

Here are just my suggestions and ideas on how to push this approach further:

1. Logical structure of resource should be reflected in its externally visible file structure, creating a "meta-format" consisting of multiple files and directories.
   
   
   • If your data consists of linked list of objects, store them (or pointers to them) in a linked list of files, so adding/removing an object will only involve modifying its neighbour file(s). Such files should be text, so "pointers" are both human-readable and human-changeable (in case of conflicts).
   
   
   • If your data is hierarchically organized, have your editor create appropriate directory structure reflecting this hierarchy, so people may check out and work on different sub-areas independently.
   
   
   • Loosely related components of a single resource should be split into multiple files, so parts that can be - within reason - modified separately are also stored separately.


2. Fixed properties of objects should be stored in human-readable (more: non-technical human-readable) form in text files, so merging/diffing is possible.


3. Validation should be introduced to check resulting resource (in this "meta-format" with multiple files) after resolving conflicts (just as coders make sure that sources still compile). Ability to fix some common problems or suggest solutions is also helpful.


4. Ideally, visual diffing and merging tools need to be written for the most common conflicts.

The basic idea is to exploit existing parallelism of versioning tools, which already operate on a hierarchically structured data (file system). It is similar to the approach used for program source - I don't think that there are many programmers who'd like to have their sources stored in a single large file featuring one giant main() function (although the compiler might like this), but we're basically forcing artists to work in such conditions.

Of course, all this comes at a cost of runtime efficiency (iteration times) and possibly even editor performance (although this may be alleviated by SSD). However, I'm not suggesting the all-or-nothing approach, some of these ideas can be applied to existing formats if bottlenecks in content creation are identified.