Connection limit metastable collapse
Your task
At 12:00 your gRPC service's goodput (successful RPS) collapses to near zero and stays there even after you scale it 3x. Dashboards: incoming connection count is pinned at the limit; clients have a 1s timeout and immediately reconnect+retry on failure; the server spends most of its time on TLS handshakes and request setup for connections whose deadline expires before the work finishes; CPU is 100% but useful completions ~0. A brief upstream slowdown at 11:58 kicked it off and has since cleared. How do you triage and mitigate?
It covers
- Connection saturation
- Goodput collapse
- Admission control