Connection churn from sidecar keepalive misconfiguration
Your task
After a service-mesh sidecar upgrade, p99 latency on an internal service-to-service call (gRPC over HTTP/2, mTLS) regressed from 8ms to 45ms, while p50 is unchanged. CPU on both client and server pods rose. Connection-level metrics show the count of new TCP connections + TLS handshakes per second jumped roughly 20x after the upgrade; the per-connection request count dropped to ~1. The mesh config that changed touched connection idle-timeout / keepalive settings. Triage and explain why p99 specifically suffered.
It covers
- Tls handshake
- Keepalive
- Connection reuse