Scaling Hierarchical Networks

Hierarchical Networks (H-Nets) aim to replace fixed language-model tokenizers with learned, data-dependent chunking. Rather than compressing bytes into tokens offline, an H-Net learns to group raw inputs into progressively larger units inside the model, such as bytes into words and words into phrases. Its encoder and chunking stage compress the sequence for a main network; a decoder then restores the representations to byte-level resolution. Chunking can be nested to create multiple abstraction levels. The central argument is that learning directly from raw data better follows AI’s “bitter lesson” than relying on human-designed preprocessing, and could support models across modalities, including continuous data.

The reported experiments compare H-Nets with BPE Transformers using a controlled training setup: the same text and document boundaries, a mixture of 70% English, 20% code, 5% math, and 5% multilingual data, and bits-per-byte metrics to normalize loss across tokenizations. The team also trained a BPE tokenizer on the comparison corpus. In scaling tests, one-stage H-Nets reportedly improve validation performance and deliver compute-efficiency gains of up to 3-5 times; gains are especially strong on code and out-of-distribution evaluation. A larger, two-stage H-Net is reported to beat a matched open-source model using ten times less compute and data. H-Nets also learn varied behaviors, including in-context tokenization, and early applications beyond language are presented as promising.