Checkout p99 holds at 1 second while customers wait
Your task
It is 11:05. Checkout calls a fraud-check service before it charges a card. Since 10:20 fraud-check has been slow, and its team is working on it. This alert fired at 10:31:
CheckoutLatencyP99: p99 above 800 ms for 10 minutes (current 1.00 s)
What you see:
- p99, the time 99% of requests finish within, was about 300 ms before 10:20. Since 10:22 the checkout dashboard has shown it at exactly 1.00 s, a flat line. p95, the time 95% finish within, also reads 1.00 s.
- Checkout's
/metricslistscheckout_request_seconds_bucketwithlevalues of 0.1, 0.25, 0.5, 1 and+Inf. - The load balancer's p99 for checkout is 7.4 s. Customers say the Pay button spins for 5 to 10 seconds, and some give up.
- The incident lead says checkout is still inside its objective, 99% of requests under 2 s, so this stays a minor incident with no status page update.
What do you tell the incident lead, and what do you change so the dashboard shows how slow checkout really is?
It covers
- Prometheus
- Latency histograms
- Service objectives