hn.today

Field Guide: Running Qwen3.8 Locally

igorstechnoclub.com5 points1 comments
Screenshot of Field Guide: Running Qwen3.8 Locally

The guide explains how to run Qwen models locally with llama.cpp, from installing or building the software to launching a model in a terminal or serving it to coding tools through an OpenAI-compatible API. Its central claim is that local models have become useful for real coding work: practitioners report that Qwen3.8-27B can act as a coding agent when given a clear plan and sufficient context, though this is experience-based rather than a controlled benchmark. The guide matches models to hardware: Qwen3.5-4B suits small GPUs and laptops for simple tasks; Qwen3.6-35B-A3B is a faster mixture-of-experts option for 16GB GPUs or 32GB Macs; Qwen3.8-27B is the higher-quality choice with roughly 24GB of VRAM; and Flash Next targets machines with 128GB of memory. It also gives example launch commands and approximate memory and speed figures.

Practical advice focuses on avoiding common setup problems and improving performance. Recent llama.cpp builds are recommended, with platform-specific notes for NVIDIA CUDA, AMD Vulkan or ROCm, and CUDA under WSL. The guide recommends limiting excessive reasoning, quantizing the KV cache to reduce context-memory use, and trying speculative decoding; MoE models can benefit from keeping experts in system RAM and increasing micro-batch size. For Mac users, it notes that the default GPU memory limit may need raising. The server should remain bound to localhost unless access is secured, since it has no authentication by default. The overall message is that model choice and configuration now determine whether local use is practical: smaller models handle lightweight work, while stronger coding-agent use requires more capable hardware and careful setup.

Read on igorstechnoclub.com1 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.