Code RoomConnection churn from sidecar keepalive misconfiguration
HardPrep Room Coding #4544

Connection churn from sidecar keepalive misconfiguration

On-callReliability & on-callSenior–Staff~40 min

After a service-mesh sidecar upgrade, p99 latency on an internal service-to-service call (gRPC over HTTP/2, mTLS) regressed from 8ms to 45ms, while p50 is unchanged. CPU on both client and server pods rose. Connection-level metrics show the count of new TCP connections + TLS handshakes per second jumped roughly 20x after the upgrade; the per-connection request count dropped to ~1. The mesh config that changed touched connection idle-timeout / keepalive settings. Triage and explain why p99 specifically suffered.

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

0:00 of about 40 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.