Some ticket buyers time out once the sale opens
Your task
It is 20:06. Tickets for a large concert went on sale at 20:00. Mobile apps reach your API through a network load balancer and an nginx ingress controller, which runs on 3 dedicated Kubernetes nodes. Last week those nodes moved from 16-vCPU to 4-vCPU machines to save cost. Tonight is their first big sale. This alert fires:
AppConnectTimeP95: connection setup p95 above 500 ms for 5 minutes (current 1.1 s)
What you see:
- In the app's telemetry, about 1 new connection in 8 takes over 1 second to set up or never connects. Requests that do connect get answers in about 140 ms.
- The load balancer sends each ingress node about 1,250 new connections a second, up from about 400 on a normal evening. Most app requests open a new connection.
- Ingress pods use 45% CPU, and the nodes have half their memory free. The pods behind the ingress show normal latency and no errors.
- Since 20:01,
dmesgon all 3 ingress nodes repeatsnf_conntrack: nf_conntrack: table full, dropping packet.
What do you do while the sale is still running, and what do you change before the next one?
It covers
- Conntrack
- Kubernetes nodes
- Connection rate