Elasticsearch split brain divergence
Your task
Search results are inconsistent: the same query returns different document sets on consecutive requests, and some recently-indexed docs appear then disappear. Started after a 14:05 network blip between two availability zones. Dashboards: the Elasticsearch cluster shows two nodes each claiming to be master (cluster state diverged), the AZ-A and AZ-B node groups can't see each other, indexing requests succeed in BOTH partitions, and shard allocation differs between the two views. The cluster has 4 master-eligible nodes (no dedicated tiebreaker). How do you triage this split-brain, stop divergence, and reconcile the data?
It covers
- Split brain
- Quorum
- Network partition
- Elasticsearch