Why Is Speculative Decoding Fast?

Speculative decoding uses a small draft model to propose multi-token continuations that a large model verifies in a single forward pass. The speedup is not because verification is cheaper in FLOPs than generation - total FLOPs typically increase - but because the GPU is doing a different kind of work. Single-token autoregressive decoding is heavily memory-bound: each step streams model parameters from global memory, leaving many ALUs idle. Verifying many tokens at once (or batching many sequences) shifts the workload to being compute-bound, better utilizing GPU arithmetic and thus reducing wall-clock latency at low batch sizes.

The core tradeoff is breadth versus depth: given a limit on useful tokens per forward pass, one can advance several sequences by one token (breadth) or advance one sequence many tokens via speculation (depth). At small batch sizes, speculative decoding uses otherwise unused compute to speed individual sequences. At high throughput, however, speculation can waste compute - especially when many draft tokens are rejected - so adaptive controls matter. Practical systems expose options to disable speculation by batch size and researchers are exploring dynamic per-step depths (e.g., dSpark) to balance utilization and rejection overhead.