Orders stop saving in the middle of the night
Your task
It is 02:40. The shop's orders database is Amazon RDS for PostgreSQL with 500 GB of storage. Storage autoscaling is off. This alert fires:
OrderWriteErrors: 100% of POST /orders failed in the last 5 minutes (threshold 2%)
What you see:
- Since 02:31 every insert fails with
ERROR: could not extend file "base/16384/24811": No space left on device. - The RDS console shows the instance as
storage-full, with no free storage left. - Free storage has fallen in a straight line, about 1 GB an hour, for the past 9 days.
- The largest table is
partner_api_log, at 214 GB. The release 9 days ago created it for the new Acme integration, which writes every request and response there. The next largest isorders, at 120 GB.
How do you get orders saving again, and how do you get the space back for good?
It covers
- Rds storage
- Table growth
- Space reclaim