Connection limit metastable collapse

On-call incidentHard45 minGoogle style

Your task

At 12:00 your gRPC service's goodput (successful RPS) collapses to near zero and stays there even after you scale it 3x. Dashboards: incoming connection count is pinned at the limit; clients have a 1s timeout and immediately reconnect+retry on failure; the server spends most of its time on TLS handshakes and request setup for connections whose deadline expires before the work finishes; CPU is 100% but useful completions ~0. A brief upstream slowdown at 11:58 kicked it off and has since cleared. How do you triage and mitigate?

It covers

  • Connection saturation
  • Goodput collapse
  • Admission control