Split-brain write divergence

On-call incidentHard40 min

Your task

A network partition briefly isolated your primary Postgres from the HA orchestrator. The orchestrator declared it dead and promoted a standby. Then the partition healed, and now you have a serious mess: some app instances (still able to reach the old node) kept writing to the OLD primary, while others wrote to the newly-promoted one. Both nodes accepted writes for ~90 seconds. Now the two diverge: conflicting rows, broken sequences, duplicate ids. Triage, contain the damage, and prevent recurrence.

It covers

  • Split brain
  • Fencing
  • Stonith