Search stops updating when one node's disk fills
Your task
You're on call for product search, which runs on a three-node Elasticsearch cluster. Since 08:15, edits that sellers make save in the main database but never show up in search. The indexer's error log repeats:
cluster_block_exception: index [products] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block]
Searches still return results, just stale ones. GET _cat/allocation shows node es-2 at 96% disk and the other two at 61% and 63%. On es-2 the biggest directory is /var/log/elasticsearch at 140 GB; on the other nodes it is under 2 GB. es-2 was rebuilt last week from an older machine image. The indexer logs each failed write and moves on. How do you triage and restore indexing?
It covers
- Elasticsearch
- Disk watermark
- Index block