6-11X speedup running Qwen3.6 on my RTX 4090, full repo.

Lightreason is a local inference and benchmarking project for running Qwen models on an RTX 4090, with an interactive browser-based Studio and extensive experimental code. It reports speedups by comparing optimized runners with an initial runner on the same prompts: Qwen2.5-3B reaches 2.22× on a target-only test suite, 9.00× on a complete matching weather response, and 11.40× on a matching 128-token Fibonacci segment. For the much larger Qwen3.6-35B-A3B text model, it reports 6.01× on a weather prompt and 6.44× on a capped Python prompt. These figures depend on the workload and whether outputs match; the 35B Python response first differs from the reference at token 116.

The techniques include CUDA Graphs and compact key-value caching for the 3B model, plus keeping most 35B expert weights in system RAM and loading the experts used by each request. Drafted tokens are checked together by the original BF16 Qwen model, which makes the final token decisions. The repository documents experiments, benchmarks, output comparisons, and slower or output-changing configurations, while building on CUDA Graphs, PARD2, DFlash2, and FreeToken. It includes a Studio for comparing standard and optimized generation, but no model checkpoints or adapter weights; these must be downloaded separately. Reproducing the public 3B Studio setup requires Linux, an RTX 4090, and a CUDA 12.8-compatible driver, while the 35B experiments have separate hardware and memory needs.