Memory alert pages for the document server overnight

On-call incidentEasy15 min

Your task

It is 01:25. The document service keeps customer files on docs-vm-2, a VM with 16 GiB of memory and no swap. A small API on the same VM serves file listings. At 23:00 the platform team added a memory alert to every VM in the company, after another team's host ran out of memory last week. It pages you:

HostMemoryHigh: memory in use above 90% for 15 minutes on docs-vm-2 (current 96%)

What you see:

  • Memory in use, as the alert measures it, sat near 60% until 01:00 and passed 90% at 01:03.
  • The nightly backup started at 01:00. It reads about 300 GB of files from the data disk, copies them to object storage and usually ends near 03:30.
  • free -m on docs-vm-2 shows 650 MiB free, 12,470 MiB in buff/cache and 12,560 MiB available.
  • The API's p95, the time 95% of requests finish within, is 110 ms, its usual night value. dmesg has no out-of-memory messages.
  • The same alert has paged 2 other on-call engineers tonight, for 6 VMs between them.

Is anything on docs-vm-2 at risk tonight, and what do you change before more of these pages go out?

It covers

  • Linux memory
  • Alert rules
  • Node exporter