The architecture of open-weight LLMs: MoE, hybrid layers, ternary weights

The study compares twelve open-weight language models, from 1.7B to 35B total parameters, selected to run on a computer with an 8 GB GPU. Its central argument is that parameter count alone does not predict local performance: architecture determines how much memory a model needs, how much work it performs per token, and how context length affects its speed. Dense transformers must read all their weights for each token, while attention also builds a KV cache that grows with the conversation. Grouped-query attention, which shares key/value heads, and sliding-window attention reduce that cache; the example Llama 3.2 3B still needs about 3.5 GiB for a 32K-token cache.

Mixture-of-experts models separate total capacity from computation: a router activates only some experts per token, so models with 20-35B total parameters can use compute comparable to a roughly 3B dense model, though they still need memory for all weights and may depend on RAM bandwidth. Recurrent layers and short convolutions further reduce context-related costs, but can make exact recall of distant details harder. Hybrid models combine these efficient layers with some full-attention layers to preserve retrieval. The comparison also highlights per-layer embeddings in Gemma 4 E4B and low-bit or ternary weights, such as Ternary Bonsai 2’s roughly 1.6-bit representation, as ways to fit more capacity into limited memory. Together, these designs trade off memory, computation, and recall in different ways.