Batch update clobbers rows

On-call incidentHard45 min

Your task

At 09:15 a support escalation says ~40,000 user accounts suddenly have subscription_tier = 'free' who were paying customers yesterday. Billing reconciliation flagged it. There's no outage, the app is up and fast. Audit logs show a single batch UPDATE ran at 08:50 from a one-off data-migration script an engineer ran to 'fix a tier-naming inconsistency'; the WHERE clause was broader than intended and clobbered the subscription_tier column for far more rows than the few hundred targeted. The script did not back up the affected rows. Stripe still shows these customers as active subscribers. Your primary Postgres has continuous WAL archiving and a base backup from 02:00; the bad write was 6.5 hours after the last base backup. How do you recover the correct data with minimal collateral damage, and how do you prevent this class of incident?

It covers

  • Data corruption
  • Point in time restore
  • Bad write