NAT gateway port exhaustion from cluster scale
Your task
Pods on a Kubernetes cluster intermittently see ~1s connection delays and occasional failures when calling an external third-party API, starting after the cluster autoscaled from 30 to 120 nodes for a campaign. The failures correlate with a single NAT gateway used for egress. The third-party API is a single VIP. Cloud NAT metrics show 'allocated ports' near the per-instance maximum and a rising 'dropped/sNAT-port-exhaustion' counter. App-side fd counts and pool sizes look healthy on each pod. Triage and mitigate.
It covers
- Snat
- Kubernetes
- Egress nat