Some ticket buyers time out once the sale opens

On-call incidentEasy15 min

Your task

It is 20:06. Tickets for a large concert went on sale at 20:00. Mobile apps reach your API through a network load balancer and an nginx ingress controller, which runs on 3 dedicated Kubernetes nodes. Last week those nodes moved from 16-vCPU to 4-vCPU machines to save cost. Tonight is their first big sale. This alert fires:

AppConnectTimeP95: connection setup p95 above 500 ms for 5 minutes (current 1.1 s)

What you see:

  • In the app's telemetry, about 1 new connection in 8 takes over 1 second to set up or never connects. Requests that do connect get answers in about 140 ms.
  • The load balancer sends each ingress node about 1,250 new connections a second, up from about 400 on a normal evening. Most app requests open a new connection.
  • Ingress pods use 45% CPU, and the nodes have half their memory free. The pods behind the ingress show normal latency and no errors.
  • Since 20:01, dmesg on all 3 ingress nodes repeats nf_conntrack: nf_conntrack: table full, dropping packet.

What do you do while the sale is still running, and what do you change before the next one?

It covers

  • Conntrack
  • Kubernetes nodes
  • Connection rate