Queue buildup hides server latency

On-call incidentHard40 min

Your task

Your service's dashboards report a healthy p99 of 25ms and the load test signs off at 30k RPS with p99 under 40ms, yet customers and the upstream gateway report frequent multi-hundred-ms stalls and timeouts during peak. When you overlay the gateway's client-side latency histogram on your server-side one, the server side looks great but the client side shows a fat tail and periodic gaps where almost no responses come back at all. The load generator is a closed-loop client that sends the next request only after the previous response. There was no code change, peak traffic just grew 20%. How do you triage this 'phantom' p99 regression?

It covers

  • Coordinated omission
  • Measurement
  • Load test