Split brain writes after failover

On-call incidentHard40 min

Your task

At 02:14 your Postgres primary (managed via Patroni + etcd) became unreachable and Patroni promoted a standby. Writes resumed within 40s. At 02:30 the old primary came back online and your monitoring now shows two nodes both accepting writes for ~6 minutes, and a few duplicate-key and constraint-violation errors are appearing in app logs. The network team reports a transient partition between availability zones during the incident. Walk through how you triage, contain, and recover, and how you prevent recurrence.

It covers

  • Failover
  • Split brain