Account balances zeroed

On-call incidentHard40 minStripe style

Your task

Support escalates that thousands of users' account balances went to 0 about 25 minutes ago. You confirm it in the database: a large swath of rows in the accounts table were overwritten with bad values. A migration job deployed 30 minutes ago ran an UPDATE without its intended WHERE clause and touched far more rows than expected; it has since finished. The data is replicated and backed up, you have continuous WAL archiving (PITR available), and writes are still flowing from live traffic. How do you triage, recover the correct data, and minimize further damage and downtime?

It covers

  • Durability
  • Point in time restore
  • Data loss