hn.today

The efficient frontier of LLM inference

baseten.co11 points0 comments
Screenshot of The efficient frontier of LLM inference

LLM inference engineering can be understood through an efficient frontier: the best available combinations of latency and throughput, which also shape serving cost. Some techniques move a deployment along that frontier, trading faster individual responses for higher throughput or vice versa; others improve efficiency across the board, letting teams allocate gains to speed, capacity, or both. The frontier is jagged rather than smooth, so the best configurations often emerge from empirical testing. The discussion assumes large models used for agentic coding, with KV-cache reuse and cache-aware routing.

Batch size is a direct latency-throughput tradeoff: smaller batches favor responsiveness, while larger ones generate more tokens per GPU at lower cost. Parallelism choices also matter: tensor parallelism can reduce latency, while broad expert parallelism and attention data parallelism can favor throughput. Quantization can improve both serving speed and throughput, often with little quality loss using formats such as MXFP4 or NVFP4, though quality and efficiency remain in tension. Other techniques push the frontier outward: optimized GPU kernels and runtimes reduce the resources needed per token; speculative decoding can skip forward passes, particularly effectively for predictable code generation; and separating prefill from decode lets operators tune workers and capacity to traffic patterns. These gains can compound, creating more room to improve latency, throughput, or cost.

Read on baseten.co0 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

ChatGPT Wallet

ChatGPT Wallet

ChatGPT Wallet lets ChatGPT agents make purchases using tokenized card details, with users reviewing each purchase and approving it through a security check. Launching first for US ChatGPT Pro subscribers, it is built with major payment networks and preserves eligible card rewards and protections. (openai.com)

Friends at Frontier Labs

Friends at Frontier Labs

Working at frontier AI labs offers friends the author cares about prestige, purpose, and collaboration, while raising serious concerns about the labs’ work. The author encourages them to keep reassessing whether to stay, recognizing that stress, loyalty, and sharp criticism make the choice difficult. (jenn.site)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.