Split-brain primary after failover

On-call incidentHard45 min

Your task

Postgres with Patroni + etcd. A network blip caused a failover: replica B was promoted to primary. After the blip cleared, the app is reporting intermittent 'duplicate key' errors and some users see writes that later vanish on refresh. Dashboards show TWO nodes advertising as primary for ~90 seconds, and your connection pooler (pgBouncer + a VIP) was briefly routing to both. Replication is now reporting a diverged timeline. Triage, contain the damage, and prevent split-brain next time.

It covers

  • Replica promotion
  • Split brain
  • Fencing