Show HN: DeepSeek-Coder-V2-Lite Under 5GB: 65 TPS 100% In-VRAM on 6GB GPU

This is a quantized, pruned version of DeepSeek-Coder-V2-Lite-Instruct designed to run entirely in the memory of a 6GB laptop GPU. Its MAPLE method retains 54 of the original 64 routed experts in each of 26 mixture-of-experts blocks, while keeping the shared experts and attention architecture intact. The model card reports a reduction from 15.7 billion to 13.5 billion total parameters; the number active per token remains 2.4 billion. The resulting GGUF file is 4.50GB, intended to leave room for GPU overhead and the model’s KV cache rather than relying on slower CPU offloading.

Benchmarks reported on an RTX 4050 laptop GPU show all 27 layers resident in VRAM, with generation speeds around 65 tokens per second and prompt processing between 236 and 370 tokens per second. The card contrasts this with 4.5-10 tokens per second for larger quantizations split between CPU and GPU, though the results come from one specific system and should not be treated as universal. Example coding tasks include a C++ lock-free ring buffer, a Python graph solver, and a Redis Lua rate limiter. The model can be run through llama.cpp, Ollama, vLLM, Docker Model Runner, and other listed tools, with commands provided for local inference and serving an API.