Search stops updating when one node's disk fills

On-call incidentEasy15 min

Your task

You're on call for product search, which runs on a three-node Elasticsearch cluster. Since 08:15, edits that sellers make save in the main database but never show up in search. The indexer's error log repeats:

cluster_block_exception: index [products] blocked by: [TOO_MANY_REQUESTS/12/disk usage
exceeded flood-stage watermark, index has read-only-allow-delete block]

Searches still return results, just stale ones. GET _cat/allocation shows node es-2 at 96% disk and the other two at 61% and 63%. On es-2 the biggest directory is /var/log/elasticsearch at 140 GB; on the other nodes it is under 2 GB. es-2 was rebuilt last week from an older machine image. The indexer logs each failed write and moves on. How do you triage and restore indexing?

It covers

  • Elasticsearch
  • Disk watermark
  • Index block