Query engine p99 spikes nightly

On-call incidentHard35 min

Your task

A self-managed analytics box that colocates a query engine and several batch workers becomes molasses-slow every weeknight around 22:00: p99 on the query engine jumps from 80ms to 12s for ~20 minutes, then recovers on its own. During the window vmstat 1 shows si/so (swap in/out) in the thousands, CPU is ~60% iowait with low user CPU, and there are no OOM kills in dmesg. The box has 64GB RAM and 32GB swap, vm.overcommit_memory=1 (always overcommit), and a nightly batch job was scaled up last sprint to use more parallel workers. Triage and mitigate.

It covers

  • Overcommit
  • Page faults
  • Iowait