Why harnesses matter for energy (and token) efficiency

This explains how the client-side harness that builds and edits prompts drives GPU work, energy use, and latency by affecting prompt-cache hit rates. Measured across many customers, uncached input ranged from about 1.6% to 11.1% by client; a drop from 97% to 92% cached increases prefill work by ~2.7×. Requests with little cached context use roughly twice the energy of fully cached ones (about 2-2.5× on the models tested), and wasted prefill also contends with generation on the same GPUs, slowing responses for others. The piece emphasizes that small differences in harness behavior therefore have outsized operational cost and latency implications.

Prompt caching requires an exact prefix, so any harness edit to previously sent content forces recomputation from the first difference. In controlled replay sessions of popular agents, 88% of turns were pure appends and the remaining breaks all came from harness edits (examples: a plan-mode reminder that toggles, omp’s read-pruning, inserting tools/skills, date changes, compaction, or changing reasoning effort). Deep sessions amplify the damage: past ~128k tokens, rare turns that re-read most of the prompt account for 75% of re-read tokens and one such turn re-reads ~230k tokens - equivalent to about 600 normal turns. Many cache breaks are fixable by changing harness behavior; newer client designs already reduce several common causes.