I'm making LLMs work on small and old devices

SMOLMAXXING is an open-source project exploring how far language models can be pushed on inexpensive, aging hardware, including devices with about 1 GB of usable memory. Its main test platform is a Xiaomi TV box with a quad-core Cortex-A55 processor, Android 14, and no root access; a jailbroken 2013 iPad Air provides a second test. The project measures performance on the devices themselves and documents 160 experiments, focusing on model size, generation speed, memory use, and output quality.

The TV box runs models from 260,000 parameters up to 16 billion. Its best results include a 2B model at 7.7 tokens per second in chat, an 8B mixture-of-experts model at 7.2, and a 16B mixture-of-experts model at 5.1. The iPad runs a 1.7B model at 24-26 tokens per second and a 16B model at 2.6-4.5. These gains come from low-bit and ternary quantization, specialized ARM kernels, reduced memory overhead, and streaming weights or paging selected experts from flash. The findings show that model size alone does not determine usefulness: the 2B model is the TV box’s best speed-quality balance, while a larger 16B model performs worse per byte. Results also depend on device-specific constraints, and some oversized dense models remain extremely slow.