LittleBit addresses the memory and compute burden of deploying large language models by pushing weight quantization below one bit per weight, where preserving model quality is especially difficult. It compresses weights through low-rank latent matrix factorization, then binarizes the resulting factors. To recover information lost at such low precision, the method learns multi-scale importance parameters across rows, columns, and latent dimensions, and uses residual compensation to reduce approximation error. Its quantization-aware training is initialized with Dual Sign-Value-Independent Decomposition (Dual-SVID), designed to make training effective despite the severe compression.
Experiments report that LittleBit reaches 0.1 bits per weight, roughly a 31-fold memory reduction, and compresses Llama 2 13B to less than 0.9 GB. On Llama 2 7B, the authors report that LittleBit at 0.1 bits per weight outperforms leading methods using 0.7 bits per weight, suggesting that latent factorization and compensation can make extreme compression more practical without an equivalent loss of capability. The approach offers a new size-performance trade-off for resource-constrained deployment; the authors estimate potential inference speedups of up to 11.6 times compared with FP16. The work’s central claim is that careful factorization, initialization, and learned error correction can make sub-one-bit LLM quantization useful rather than merely small.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.