Checkout latency spikes

On-call incidentHard35 min

Your task

A non-critical 'related recommendations' enrichment dependency degraded to ~10s response times (it's slow, not down). Within minutes your primary checkout API, which only calls recommendations as an optional sidebar enrichment, saw its own p99 climb to 10s, threads pile up, and it starts failing health checks and getting killed/restarted, taking down checkout entirely. The recommendations call has no explicit timeout and is made inline in the request path. How do you triage and mitigate?

It covers

  • Timeout cascade
  • Missing timeout
  • Fallback