How Uber Protects Against Retry Storms

This describes how Uber prevents retry storms - cascading amplifications of traffic caused when many services uniformly retry failed calls down a deep dependency chain. It quantifies amplification with R^d × η (R = retries, d = depth) and shows that simple retry budgets ((1+B)^d × η) substantially reduce load amplification; a 10% budget limits exponential growth in their examples. Because retry behavior today lacks context, retries against a degraded downstream can worsen overload. The proposed fix is "error ownership": use a service dependency analysis that correlates inbound failures with outbound failures and marks responses with an x-uber-error-claim header to indicate whether the returning error is a symptom of a downstream or an originating failure. If a service’s outbound failed, it claims the error; if its outbounds succeeded but it still errors, it owns the error.

Callers consult the claim/unclaim logic and a decision matrix to decide whether to retry, and the system unclaims missing headers at the first upstream to limit disturbance. The design also handles coincidental failures by preferring downstream attribution and using historical failure-memory to avoid suppressing legitimate retries. After six months of analysis the approach shows rare misattribution risk and has stopped millions of spurious retries (notably ~9.5M), reducing cross-service amplification while preserving necessary retry paths.