Payment provider degradation

On-call incidentMedium30 minStripe style

Your task

It's a Tuesday afternoon and your checkout service's p99 latency jumps from 180ms to 9s. Error rate on /checkout climbs to 22%. Dashboards show your payment provider's API (Stripe charge creation) returning a mix of HTTP 503s and connections that hang until your 10s client timeout fires. Your own CPU, memory, and DB metrics are flat-to-low. There was no deploy in the last 6 hours. The Stripe status page just flipped one component to 'degraded.' Inbound checkout traffic is normal for the time of day. How do you triage and mitigate?

It covers

  • Payments
  • Graceful degradation
  • Circuit breaker