Dependency latency spike

On-call incidentMedium30 min

Your task

Your real-time inference service enriches each request by calling a third-party 'identity-risk' API synchronously to build one feature before scoring. At 12:40 your service's p99 latency triples and error rate hits 4%. Tracing shows the time and errors are all in the outbound call to the third-party API, which is returning slow responses and intermittent 503s; their status page later confirms a partial outage. Your own model and feature store are healthy. Triage and mitigate.

It covers

  • Dependency
  • Third party api
  • Timeout
  • Graceful degradation