Chat server stalls at a thousand connections

On-call incidentEasy15 min

Your task

The support chat service holds one WebSocket per connected customer. It runs on two VMs behind a load balancer and normally peaks at about 1,800 connections per VM. This morning both VMs were replaced with new ones built from a new base image; the two old VMs are still running, out of the load balancer, kept for rollback.

At 10:05, as traffic grows, customers start seeing "Chat unavailable". Each new VM is stuck at about 1,010 open connections, with CPU at 20% and memory at 35%. The service log repeats accept4: too many open files. On a new VM, cat /proc/$(pgrep chatd)/limits shows Max open files 1024 1024 files; on an old VM the same line shows 65536. The service runs under systemd. How do you triage and mitigate?

It covers

  • File descriptors
  • Ulimit
  • Websockets