Breaking the 1.58-bit Barrier for Ternary LLMs

Ternary Large Language Models store weights as -1, 0, or +1 and are commonly treated as costing log2(3) ≈ 1.585 bits per weight; practical deployments pack five ternary weights into a byte (five-trit packing), producing an effective 1.625 bits per weight because of power-of-two grouping and an implicit assumption of equiprobable symbols. A systematic measurement across 29 ternary LLMs shows symbol distributions are far from uniform, with zeros comprising up to 51.5% of weights. This nonuniformity motivates a redistribution of storage: by explicitly encoding the presence of nonzero entries and compacting their signs, one can break the conventional bit barrier.

BITCOS is a distribution-adaptive layout that stores a dense presence bitmap plus a compacted sign vector, achieving an average cost of 2 − z bits per weight given zero density z. BITCOS compresses 26 of the 29 tested models more compactly than five-trit packing and reaches 1.485 bits per weight on the sparsest model. The layout supports efficient unpacking on modern CPUs and Intel Xe2 GPUs; optimized sequences for AVX-512, AVX2 and Xe2 are provided. Against state-of-the-art ternary matrix-vector kernels, BITCOS yields up to 1.28× kernel speedups and end-to-end LLM decode throughput improvements up to 1.18× on CPUs and 1.27× on GPUs.