hn.today

Latent Lattice Language Models

shivamduggal4.github.io55 points5 comments
Screenshot of Latent Lattice Language Models

Latent Lattice Language Models propose increasing a language model’s computational capacity by maintaining more active states, rather than relying only on deeper networks, extra reasoning steps, or modified residual connections. Each layer gains horizontal latent streams that recur across tokens alongside the usual vertical residual stream, which carries information through the model’s depth. These persistent latents are not just sequence memory: they directly influence attention, key-value cache writes, MLP computation, and decoding. The approach is presented as complementary to residual-stream enhancements and to recurrent sequence-mixing architectures.

In experiments on 20 billion FineWeb-Edu training tokens, a 406-million-parameter Lattice model achieved validation perplexity of 14.40, compared with 15.41 for a parameter-matched Transformer. Combining the Lattice with hyper-connections and input residuals lowered perplexity further, to 13.61, versus 14.22 for the Transformer with those enhancements. The gains also held across Gated DeltaNet, Mamba-2, Triadic GDN, and hybrid recurrent backbones at every tested native state size; Lattice-GDN variants even surpassed pure baselines with four times larger native states. The experiments also examine training scale, component ablations, and inference computation. Lattice models using 22 rather than 40 layers can retain the same attention width while reducing KV-cache size by 45%, suggesting persistent per-layer states offer another way to scale model computation.

Read on shivamduggal4.github.io5 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.