Retry storm amplifies upstream slowdown

On-call incidentHard40 min

Your task

Your recommendations service calls an upstream feature-store service. At 18:30 the feature-store's p99 creeps from 50ms to 400ms (a mild, real slowdown). Within two minutes the feature-store is fully down, 100% errors, and your recommendations service is also down. Dashboards: feature-store inbound request rate is 6x normal even though end-user traffic is flat; its CPU is pegged; your service shows massive retry counts. Recent context: a config change last sprint set client retries to '3 retries, no backoff' on the recommendations to feature-store call. How do you triage and mitigate?

It covers

  • Retries
  • Retry storm
  • Thundering herd