Downstream service degraded

On-call incidentMedium20 min

Your task

Your service's latency dashboard shows requests getting slow and some returning 504 Gateway Timeout. Drilling in, the slowness is concentrated on endpoints that call one downstream service; endpoints that don't call it are fast. The downstream's own status page shows it's degraded. You didn't deploy. Your service's threads/connections are filling up waiting on it. How do you triage and protect your service?

It covers

  • Timeout
  • Cascade
  • Third party