Binary Serialization in Df_game_data

This describes df_game_data, an open-source MIT-licensed game data middleware that provides a flexible, efficient binary serialization format and a schema-driven toolchain. It tackles the core problem of mixing fixed-size and variable-size fields by separating fixed data from dynamic data and writing offsets that are later fixed up into pointers when the file is loaded. To avoid O(n) indexing through variable-length entries, dynamic elements (strings and dynamic arrays) are written out-of-line and referenced by offsets, and a table of deferred writes records where those offsets must be patched. To support nested cases - like arrays of objects that contain strings - the system uses a stack of data buffers so each nesting level has its own fixed/dynamic regions; deferred entries record both source and target stack locations. An offset value that translates to file position zero is treated as null, enabling optional fields.

The toolchain uses a C/C++-style .def schema to produce a .bin data file and a header-only loader (.h) in C or C++. Struct fields support booleans, various integer sizes, floats, strings, enums, structs, unions, links, and static or dynamic arrays with no special placement rules. Files include a fourcc for type and endian checks, a three-byte version, a uint64 schema hash (computed from the in-memory schema representation so nonsemantic edits don’t break compatibility), and the serialized data where pointers are stored as uint64 offsets. The design aims for O(1) access, portability, optional features like hot-reload and name lookup, and a more permissive model than alternatives like flatbuffers.