hn.today

Inkling-2-Mini in 5.4 GB Memory

github.com494 points124 comments
Screenshot of Inkling-2-Mini in 5.4 GB Memory

LookAtMyQuant is an open-source quantization compiler that measures how compressing individual model components changes their outputs, then allocates limited storage to minimize overall damage. Its techniques include trellis coding, Fisher-information-based bit allocation by row, calibration on representative data, and post-quantization fine-tuning. The repository reports results across several models, with its main example here being Inkling-2-Mini: a model with 26.4 billion physical parameters and 5.4 billion active parameters, whose weights are compressed from 18.89 GB to 5.38 GB or 8.87 GB. The recipe uses hashed block-vector and row trellis codes, plus proximal fine-tuning; the vision tower remains uncompressed.

In the reported tests against the original BF16 model, the 5.38 GB version scores 134/200 on GSM8K and 60/164 on HumanEval, compared with 147 and 66 for the original. It beats Unsloth’s 9.14 GB UD-Q2_K_XL on both tasks while using 41% less space. The 8.87 GB version matches the original’s GSM8K score and has lower text KL divergence than that Unsloth model across English, chat, code, and 13 languages. Both versions run on Apple Silicon through custom Metal kernels, with reported decode speeds up to 73.2 tokens per second for the smaller model. The repository provides runtime installation and usage instructions, model files, and a compiler workflow for measuring and allocating compression.

Read on github.com124 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.