Orders lost after failover

On-call incidentHard40 minStripe style

Your task

A primary Postgres instance crashed (host failure) and your HA controller auto-promoted an async streaming standby. Service recovered in ~40s. But now support is getting reports that a handful of orders placed in the last minute before the crash 'disappeared', they were charged but the order doesn't exist in the DB. Reconciliation against the payment processor confirms ~30 orders are missing from the new primary. The promoted standby was healthy and ~2s behind at promotion time. Triage, mitigate the user impact, and prevent recurrence.

It covers

  • Async replication
  • Data loss
  • Rpo