DeepSeek Ultrafast

DeepSeek Ultrafast is an open-source Rust inference engine and serving stack for DeepSeek-V4.1-Flash, designed for NVIDIA B200, B300, and GB300 systems. Its central claim is that speculative decoding and optimized GPU kernels can deliver very high generation rates while supporting context lengths up to one million tokens. Reported tests show 1,740 tokens per second on four GB300 GPUs and 2,060 on eight B300 GPUs, with average decode steps of 3.30 and 2.71 milliseconds, respectively. Each step produces about 5.6-5.7 accepted tokens. For prefill, the system processes one million tokens in 13.2 seconds on four GB300s or 11.7 seconds on eight B300s.

The repository explains how to build and run the server, which exposes an OpenAI-compatible API, and how to reproduce its Tetris-generation benchmark. Running it requires CUDA 13, model files, supported GPUs, and - when coordinating multiple servers - a Redis instance. The design keeps conversation KV caches in host memory and uses RDMA to transfer them directly between servers when later requests reach a different replica. Redis tracks cache ownership, so deployments can balance requests without routing users to a server based on prompt prefixes; the compressed caches are intended to move quickly enough to make this practical. The documentation also includes a Kubernetes deployment example and identifies libraries used for tokenization, image handling, and cache transfer.