Split-brain cache after partition
Your task
Your distributed cache runs primary-replica with an automatic failover daemon (sentinel-style) that promotes a replica if it can't reach the primary. At 03:40 a brief partition isolates the failover daemon (and some clients) from the primary, but the primary itself is healthy and still serving the clients on its side. The daemon promotes the replica. Now you have TWO primaries accepting writes for ~3 minutes; clients on each side see different data, and after the partition heals you find conflicting/overwritten keys. How do you triage and mitigate this SPLIT-BRAIN and prevent recurrence?
It covers
- Split brain
- Cache
- Dual primary