Checkout p99 holds at 1 second while customers wait

On-call incidentMedium25 min

Your task

It is 11:05. Checkout calls a fraud-check service before it charges a card. Since 10:20 fraud-check has been slow, and its team is working on it. This alert fired at 10:31:

CheckoutLatencyP99: p99 above 800 ms for 10 minutes (current 1.00 s)

What you see:

  • p99, the time 99% of requests finish within, was about 300 ms before 10:20. Since 10:22 the checkout dashboard has shown it at exactly 1.00 s, a flat line. p95, the time 95% finish within, also reads 1.00 s.
  • Checkout's /metrics lists checkout_request_seconds_bucket with le values of 0.1, 0.25, 0.5, 1 and +Inf.
  • The load balancer's p99 for checkout is 7.4 s. Customers say the Pay button spins for 5 to 10 seconds, and some give up.
  • The incident lead says checkout is still inside its objective, 99% of requests under 2 s, so this stays a minor incident with no status page update.

What do you tell the incident lead, and what do you change so the dashboard shows how slow checkout really is?

It covers

  • Prometheus
  • Latency histograms
  • Service objectives