A high-performance open-source engine that runs a 125B mixture-of-experts Qwen3.8-Flash-Next (NVFP4) on two 16 GB consumer GPUs by tensor-parallelizing most computation across the cards and offloading the routed experts that don’t fit in VRAM to host RAM/CPU. The design uses PCIe tensor-parallelism without peer-to-peer, a VRAM cache partitioned by expert id, and an AVX-512 NVFP4 CPU kernel to compute missing experts; it supports MTP speculative decoding (K=3), INT8 KV with an elastic KV/expert-slot budget up to 262,144 tokens, prefix caching across turns, and an OpenAI-/Anthropic-compatible server. The same binary also serves a dense Qwen3.8-27B. It is a two-GPU fork of ninfer-tp2 (renamed capy-flash) and ships with install scripts, serving profiles, and measured development logs.
Benchmarks on a Ryzen 9 9900X with 2×RTX 5070 Ti (16 GB) show strong throughput and responsiveness: prefill speeds reach ~3.8k-4.0k tok/s for long prompts; agentic replay median time-to-first-token 1.82 s and decode 100.8 tok/s (total 243 s) versus Strata NVFP4 at 2.42 s and 78.3 tok/s (314 s). Accuracy is comparable to an NVIDIA GB300 NVFP4 reference on GSM8K and MMLU-Pro. Agentic task runs solved 20/20 cases with no malformed tool calls and produced ~42.9k generated tokens per run. Requirements include Linux, CUDA 13.1, two sm_120 GPUs (RTX 50-series 16 GB), an AVX-512 CPU, ~70 GiB free RAM and ~135 GB disk (64.6 GiB page-locked host allocation for experts). Versioned release: v0.5.4.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.
Elon Musk has made numerous predictions about Tesla achieving full autonomous driving, with timelines ranging from 2018 to 2023. As of 2024, Tesla's FSD system remains classified as Level 2, and full autonomy has not yet been achieved. (en.wikipedia.org)
AI has contributed to a doubling of modestly successful software projects, such as plugins, mods, and extensions, since mid-2025. These projects mainly serve everyday user needs, indicating AI's role in enabling hobbyist and utility software development. (seangoedecke.com)
Open weight models now process over 71.5% of tokens and account for nearly 23% of AI Gateway spend, quadrupling their share since June. Spending on open weight models has increased more than four times in three months, reflecting rapid adoption and shifting AI usage patterns. (vercel.com)
An AI system has successfully designed 16 new viruses, including bacteriophages. This breakthrough raises concerns about potential biosecurity risks. (cnn.com)
Flux 3 Image Klein is a 5.6 billion parameter open weights model for text-to-image generation, editing, and multi-reference creation. It offers high-quality, ground-connected images with efficient performance on consumer GPUs, enabling local and on-device use. (bfl.ai)
Yum! Brands has implemented AI in its restaurants, such as automating drive-thru orders, which has improved employee retention by making jobs easier. Despite public criticism of AI, company leaders see its benefits in increasing workplace satisfaction and efficiency. (fortune.com)
Today's best Hacker News stories, summarized and screenshotted, one email a day.