Swap thrashing node
Your task
A self-managed Elasticsearch data node becomes unresponsive: query latency goes from 50ms to 20s, the node drops in and out of the cluster, and CPU shows 70% iowait while user CPU is low. vmstat 1 shows si/so (swap in/out) in the hundreds of MB per second sustained, load average 40 on an 8-core box, and free memory near zero. A colleague added a heavy aggregation-heavy reporting workload this morning, and someone had earlier set swappiness to 60 and left a 16GB swap file enabled. Heap is sized at 26GB on a 32GB host. Describe triage and mitigation.
It covers
- Page faults
- Iowait
- Overcommit