Service recovery cascades into overload
Your task
A core internal service briefly hiccups at 20:00 (a 5-second blip during a leader election). Instead of recovering, it stays down: it comes back up, gets instantly flooded to many times its normal request volume, falls over again, and this cycle repeats every ~30 seconds. Five upstream services all call it; each retries failed calls 3x immediately, and several have their own callers that also retry. Request volume to the struggling service is now ~8x baseline even though end-user traffic is normal. Triage and break the cycle.
It covers
- Retry storm
- Cascading failure
- Jitter