NAT gateway port allocation fails

On-call incidentHard30 minAmazon style

Your task

Your fleet makes lots of short-lived outbound HTTPS calls to several third-party APIs and egresses through a managed NAT gateway. You scaled the fleet from 60 to 150 instances for a campaign; outbound success rate dropped and you see intermittent connection timeouts and resets to external endpoints. The third-party APIs are healthy and not rate-limiting you. NAT gateway metrics show ErrorPortAllocation climbing and active connection count near a ceiling. How do you triage and what's the durable fix?

It covers

  • Nat gateway
  • Port exhaustion
  • Egress