Order database recovery for an online grocery
Your task
Your online grocery keeps its orders and customer accounts in one PostgreSQL database on its own servers. It holds 3 TB and takes about 2,000 writes a second at peak. The primary sends every change to two standbys within a second. A full backup runs each night, and the change log is archived every minute.
Twice this year a bad change hit the data. Once a script meant for one store updated the status of every order. Once a migration dropped a column the app still needed. Each time, restoring the nightly backup to a new server took about 5 hours. Replaying the day's change log took 2 more, and the shop was down the whole time.
The business wants the shop working again within 30 minutes of noticing a bad change, with nothing lost that was written before it. Bad changes are usually noticed within 1 hour.
What do you keep running or stored so recovery fits in 30 minutes, and what are the steps when it happens?
It covers
- Database recovery
- Replication
- Backups