Orders stop saving in the middle of the night

On-call incidentEasy15 min

Your task

It is 02:40. The shop's orders database is Amazon RDS for PostgreSQL with 500 GB of storage. Storage autoscaling is off. This alert fires:

OrderWriteErrors: 100% of POST /orders failed in the last 5 minutes (threshold 2%)

What you see:

  • Since 02:31 every insert fails with ERROR: could not extend file "base/16384/24811": No space left on device.
  • The RDS console shows the instance as storage-full, with no free storage left.
  • Free storage has fallen in a straight line, about 1 GB an hour, for the past 9 days.
  • The largest table is partner_api_log, at 214 GB. The release 9 days ago created it for the new Acme integration, which writes every request and response there. The next largest is orders, at 120 GB.

How do you get orders saving again, and how do you get the space back for good?

It covers

  • Rds storage
  • Table growth
  • Space reclaim