On-callHardoc-g191

Subject Ephemeral port exhaustionLevel Senior–Staff~40 minCommon in Reliability & on-call interviewsIndustries Software development

Question

Pods on a Kubernetes cluster intermittently see ~1s connection delays and occasional failures when calling an external third-party API, starting after the cluster autoscaled from 30 to 120 nodes for a campaign. The failures correlate with a single NAT gateway used for egress. The third-party API is a single VIP. Cloud NAT metrics show 'allocated ports' near the per-instance maximum and a rising 'dropped/sNAT-port-exhaustion' counter. App-side fd counts and pool sizes look healthy on each pod. Triage and mitigate.

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.