NAT gateway port exhaustion from cluster scale

On-call incidentHard40 minAmazon style

Your task

Pods on a Kubernetes cluster intermittently see ~1s connection delays and occasional failures when calling an external third-party API, starting after the cluster autoscaled from 30 to 120 nodes for a campaign. The failures correlate with a single NAT gateway used for egress. The third-party API is a single VIP. Cloud NAT metrics show 'allocated ports' near the per-instance maximum and a rising 'dropped/sNAT-port-exhaustion' counter. App-side fd counts and pool sizes look healthy on each pod. Triage and mitigate.

It covers

  • Snat
  • Kubernetes
  • Egress nat