Mesh sidecar cannot assign address

On-call incidentHard35 min

Your task

A service mesh sidecar fleet starts intermittently failing outbound calls at peak with 'cannot assign requested address', and the failures cluster on the busiest pods. ss -s on a hot pod shows tens of thousands of sockets to a handful of upstreams, mostly TIME_WAIT, and the node's conntrack table is also near its max. Traffic is up because a marketing event doubled load this afternoon, and a recent mesh config change reduced upstream connection idle timeout so connections close and reopen far more aggressively than before. CPU/mem/fds are fine. Triage and mitigate.

It covers

  • Dns
  • No connection reuse
  • Scale out