Split-brain primary divergence

On-call incidentHard45 min

Your task

You run a 3-node self-managed PostgreSQL cluster with a custom Patroni-style failover. At 02:14 customers report some writes 'disappearing' on refresh. Dashboards: two nodes (A and B) both report themselves as primary; A is in us-east-1a, B in us-east-1c; the etcd DCS shows intermittent connectivity between AZs starting 02:09; both A and B are accepting writes and each has a different, diverging WAL position. A network team is mid-incident on an inter-AZ link. How do you triage and mitigate?

It covers

  • Quorum
  • Network partition
  • Leader election