Retry amplification cascades through hops

On-call incidentHard40 minGoogle style

Your task

A user-facing request fans out through 4 service hops (gateway to A to B to C). At 17:10 you see massive load on service C and timeouts at the gateway. Dashboards: each hop has its own 2s timeout and retries twice on timeout; C's actual p99 is 1.8s but it's receiving ~8x its normal request volume; user traffic is flat; the gateway's own deadline is 2s. A latency bump on C started at 17:05 (a slow query). How do you triage and mitigate?

It covers

  • Timeout budget
  • Deadline propagation
  • Retry amplification