LLM inference engineering can be understood through an efficient frontier: the best available combinations of latency and throughput, which also shape serving cost. Some techniques move a deployment along that frontier, trading faster individual responses for higher throughput or vice versa; others improve efficiency across the board, letting teams allocate gains to speed, capacity, or both. The frontier is jagged rather than smooth, so the best configurations often emerge from empirical testing. The discussion assumes large models used for agentic coding, with KV-cache reuse and cache-aware routing.
Batch size is a direct latency-throughput tradeoff: smaller batches favor responsiveness, while larger ones generate more tokens per GPU at lower cost. Parallelism choices also matter: tensor parallelism can reduce latency, while broad expert parallelism and attention data parallelism can favor throughput. Quantization can improve both serving speed and throughput, often with little quality loss using formats such as MXFP4 or NVFP4, though quality and efficiency remain in tension. Other techniques push the frontier outward: optimized GPU kernels and runtimes reduce the resources needed per token; speculative decoding can skip forward passes, particularly effectively for predictable code generation; and separating prefill from decode lets operators tune workers and capacity to traffic patterns. These gains can compound, creating more room to improve latency, throughput, or cost.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.