Packing Binary Is Fun

An engineer builds a custom binary schema language called jBin to replace verbose JSON with a compact, schema-driven binary format and reports payload size reductions around 80%. The write-up motivates the move from text to binary, then walks through the design trade-offs that lead to a schema-driven encoder: avoid per-value type headers, treat integers/bools as varints and floats as fixed-width, and use delimited blocks for strings. The implementation goal is to let a schema describe structure so encodings are predictable and compact, and tooling handles the tedious packing and unpacking.

The technical core explains a varint encoding using 7-bit groups with an MSB continuation bit (little-endian grouping) and how wire type and field number are packed together by shifting the field number left two bits and OR-ing the 2-bit type. That permits a single varint header to convey field id, type and length metadata. Schema syntax is numeric-field oriented (e.g., "1. name: type"), supports messages, enums, lists, maps, unions (with per-type defaults), self-references, packages/imports, and optional defaults. The write-up also covers building the compiler toolchain - lexer, parser, AST, semantic analyzer, dynamic packer, codegen, visitor pattern and a CLI - showing a full end-to-end system for schema-driven binary packing.