Retry amplification after vendor blip

On-call incidentHard35 min

Your task

Your service calls a cloud vendor API with a per-minute quota you normally use ~30% of. At 11:40 you start getting 429 'quota exceeded' from the vendor and the affected feature errors at ~50%. Dashboards: your OUTBOUND call rate to the vendor jumped 4x starting at 11:38, but inbound user traffic is flat. Around 11:37 the vendor had a 90-second blip of 503s (now recovered). Your client retries failed vendor calls up to 5 times with a fixed 200ms backoff. No deploy. How do you triage and mitigate?

It covers

  • Retry storm
  • Amplification
  • Quota exhaustion