Elasticsearch split brain divergence

On-call incidentHard45 min

Your task

Search results are inconsistent: the same query returns different document sets on consecutive requests, and some recently-indexed docs appear then disappear. Started after a 14:05 network blip between two availability zones. Dashboards: the Elasticsearch cluster shows two nodes each claiming to be master (cluster state diverged), the AZ-A and AZ-B node groups can't see each other, indexing requests succeed in BOTH partitions, and shard allocation differs between the two views. The cluster has 4 master-eligible nodes (no dedicated tiebreaker). How do you triage this split-brain, stop divergence, and reconcile the data?

It covers

  • Split brain
  • Quorum
  • Network partition
  • Elasticsearch