On-callHardoc-g037

Subject P99 regressionLevel Senior–Staff~40 minCommon in Reliability & on-call interviewsIndustries Technology, Software development

Question

After a service-mesh sidecar upgrade, p99 latency on an internal service-to-service call (gRPC over HTTP/2, mTLS) regressed from 8ms to 45ms, while p50 is unchanged. CPU on both client and server pods rose. Connection-level metrics show the count of new TCP connections + TLS handshakes per second jumped roughly 20x after the upgrade; the per-connection request count dropped to ~1. The mesh config that changed touched connection idle-timeout / keepalive settings. Triage and explain why p99 specifically suffered.

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.