Divergent data after failover
Your task
A managed MySQL cluster (semi/async replication, automated failover) failed over at 02:50 when the primary's host became unreachable. A replica was promoted and writes resumed in ~30s. At 09:00, reconciliation finds a few hundred rows where the old primary and the new primary disagree: the old primary has writes from 02:48–02:50 that the new primary never received, and a couple of rows were written on both with different values. The old host has since rejoined as a replica. Dashboards: replication healthy now, no current lag. How do you triage, prevent the divergence from spreading, and reconcile the data safely?
It covers
- Failover divergence
- Forked writes
- Async replication
- Reconciliation