Order database recovery for an online grocery

DesignMedium35 min

Your task

Your online grocery keeps its orders and customer accounts in one PostgreSQL database on its own servers. It holds 3 TB and takes about 2,000 writes a second at peak. The primary sends every change to two standbys within a second. A full backup runs each night, and the change log is archived every minute.

Twice this year a bad change hit the data. Once a script meant for one store updated the status of every order. Once a migration dropped a column the app still needed. Each time, restoring the nightly backup to a new server took about 5 hours. Replaying the day's change log took 2 more, and the shop was down the whole time.

The business wants the shop working again within 30 minutes of noticing a bad change, with nothing lost that was written before it. Bad changes are usually noticed within 1 hour.

What do you keep running or stored so recovery fits in 30 minutes, and what are the steps when it happens?

It covers

  • Database recovery
  • Replication
  • Backups