Product images vanish after a storage cleanup rule

On-call incidentEasy20 min

Your task

Since about 09:00, product pages show broken images for older listings. The CDN reports 404s on /images/products/... rising from almost none to 18% of image requests. Images for products listed in the last 30 days load fine. The images live in an S3 bucket, shop-media, with versioning enabled.

Yesterday at 17:00 an engineer merged a Terraform change titled "expire temp uploads after 30 days", which added a lifecycle rule to the bucket. In the plan output, the rule's filter has prefix = "" where it should say prefix = "tmp/". Lifecycle actions run in the background, and the bucket's count of current objects has been falling since about 03:00. How do you triage, stop the damage and recover?

It covers

  • Object storage
  • Lifecycle rules
  • Data recovery