Elasticsearch split brain divergence
Search results are inconsistent: the same query returns different document sets on consecutive requests, and some recently-indexed docs appear then disappear. Started after a 14:05 network blip between two availability zones. Dashboards: the Elasticsearch cluster shows two nodes each claiming to be master (cluster state diverged), the AZ-A and AZ-B node groups can't see each other, indexing requests succeed in BOTH partitions, and shard allocation differs between the two views. The cluster has 4 master-eligible nodes (no dedicated tiebreaker). How do you triage this split-brain, stop divergence, and reconcile the data?
Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.