Hedged request amplification
Your task
To cut tail latency you enabled hedged requests on a hot internal RPC: if the first attempt doesn't respond within the p95 (~15ms), the client fires a second request to another replica and takes whichever returns first. It worked beautifully and shaved the tail. Then at peak this morning the backend tier tipped over: when the backend got mildly slow, request volume to it suddenly nearly DOUBLED, pushing it from 80% to 100% CPU, which made more requests cross the 15ms threshold, which fired more hedges, a runaway. There was no traffic spike from real users. How do you triage and mitigate, and how should hedging be configured safely?
It covers
- Retry amplification
- Hedging
- Retry budget