Memory leak in per-connection cache

On-call incidentMedium30 min

Your task

You deploy v205 of a Node.js websocket gateway at 09:00. Everything looks fine for ~40 minutes, then pods begin restarting one by one with OOMKilled, and as each restarts, clients reconnect en masse, briefly spiking load on the survivors which then OOM faster, a rolling brownout. The memory dashboard shows each pod's RSS climbing linearly from deploy time, never plateauing. v205 added a per-connection in-memory cache of recent messages. Request rate is flat versus yesterday. Triage and mitigate.

It covers

  • Memory leak
  • Oom
  • Gc