Tail latency triggers cascading timeouts

On-call incidentHard40 minGoogle style

Your task

A request flows gateway to A to B to C. Each hop has a 3s client timeout. At 19:00, C develops a mild tail: most calls are fast but p99.9 hits 2.8s. You'd expect this to be harmless, 2.8s is under every 3s timeout. Instead, the gateway's error rate climbs to 8% and you see wasted work: C completes requests successfully that the gateway has already given up on, and B and A both retry. Dashboards: C's success rate is ~100% (it eventually answers), but its load is climbing; the gateway sees timeouts. No deploy on C; it's just a tail. How do you triage and mitigate?

It covers

  • Timeout tuning
  • Tail latency
  • No deadline budget