Timeout reduction triggers retry storm

On-call incidentMedium30 min

Your task

An SRE lowers the HTTP client timeout to a downstream auth service from 2s to 200ms via a config push at 16:40 (intended to fail faster). At 16:42 the auth service's request rate triples, its CPU saturates, and the calling API's success rate drops from 99.9% to 88%. The auth service itself didn't change and its p50 latency is 150ms but p99 is 400ms. The client retries failed auth calls up to 3 times. Triage and mitigate.

It covers

  • Timeout
  • Retry storm
  • Dependency