Timeout reduction triggers retry storm
Your task
An SRE lowers the HTTP client timeout to a downstream auth service from 2s to 200ms via a config push at 16:40 (intended to fail faster). At 16:42 the auth service's request rate triples, its CPU saturates, and the calling API's success rate drops from 99.9% to 88%. The auth service itself didn't change and its p50 latency is 150ms but p99 is 400ms. The client retries failed auth calls up to 3 times. Triage and mitigate.
It covers
- Timeout
- Retry storm
- Dependency