Retry amplification cascade

On-call incidentMedium35 minGoogle style

Your task

A request flows A to B to C. C has intermittently slow calls (its p99 is 6s). A's client timeout to B is 2s with 2 retries; B's client timeout to C is 5s with 2 retries. When C is slow, you observe: A times out at 2s and retries, but B is still happily working on the first request (and will keep going up to 5s × retries), and the backend ends up doing far more work for C than the user ever waits for. C's load climbs even though user traffic is flat. Triage and explain the timeout/retry misconfiguration.

It covers

  • Timeout misconfiguration
  • Nested timeouts
  • Work amplification