A prototype inference engine runs GPT-OSS-120B and Qwen3.5-122B-A10B on a 32GB M1 Max while enforcing a 16 GiB process-memory budget. It keeps most mixture-of-experts weights on the SSD and loads what inference needs, aiming not only to fit large models but to keep macOS and other applications responsive. In a controlled comparison using matching GPT-OSS weights, it averaged 7.84 tokens per second versus 5.78 for TUFF, about 36% faster. With another process using 4 GiB of RAM, its speed advantage grew: the prototype slowed about 3%, while TUFF slowed about 23%. The prototype also kept model data out of the macOS file cache and compressed memory near idle levels. A competing process’s 4 GiB page-touch pass took a median 4 ms with the prototype running, compared with 352 ms with TUFF.
The results emerged after correcting misleading benchmarks caused by cached model data and bugs in numerical handling and prefix-cache reuse. In current tests, all nine reference cases for each 120B-class model matched generated token IDs and logit hashes; additional GPT-OSS stress and cancel-and-retry tests also found no corruption. These checks do not establish correctness for every input. The comparison covers one laptop, one request at a time, and mostly decoding; TUFF prefills a long prompt faster in an earlier test. The engine’s code is not yet released, and Qwen’s comparative speed results remain to be redone. The work’s central claim is therefore a promising, carefully measured approach to predictable-memory MoE inference on Apple Silicon - not a general performance ranking or a claim to have invented expert offloading.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.
Researchers tested 16 LLMs on all 55 TIS-100 puzzles, giving each one attempt and time to refine its code. The models found 133 new performance tradeoffs across 18 puzzles and set three outright cycle records, though human players improved on most solutions within days. (tis-100.wincon.dev)
Australia is considering a new criminal offence for AI companies that fail to take reasonable steps to prevent harm caused by their models. The government is also weighing an AI safety watchdog with power to impose significant fines, following a hack of a Medicare statistics portal involving OpenAI. (abc.net.au)
AI is increasingly becoming a gateway through which people find information. Audiences still resist having AI produce their news. (niemanlab.org)
Nvidia holds a central position in the AI industry, making disruption to the company potentially catastrophic for the wider ecosystem. The race for advanced AI is also intensifying economic and geopolitical conflict, while concentrated corporate power and infrastructure face growing political resistance. (france24.com)
LLM inference can require reading about 26 GB of model weights to generate each token from a 13-billion-parameter model. The explanation shows how model size drives memory bandwidth demands during text generation. (youtube.com)
Deletion Tuning matches full fine-tuning across three LLM post-training methods by zeroing fewer than 0.05% of model weights. Its compact masks also preserve prior-task accuracy better than full fine-tuning, LoRA, and random sparse tuning. (arxiv.org)
Today's best Hacker News stories, summarized and screenshotted, one email a day.