TTL backfill deletes active documents

On-call incidentHard35 minAmazon style

Your task

A DynamoDB table stores user documents with a TTL attribute expires_at so abandoned drafts auto-delete after 90 days. A backfill ran yesterday to add expires_at to ~12M legacy rows that lacked it. Overnight, support reports active users' live documents vanishing, not drafts, real saved work, and the count keeps growing hour over hour. Dashboards: no error spike, no manual deletes in CloudTrail, but the table's item count is dropping and the TTL-deletes metric is elevated. How do you triage, stop the loss immediately, and recover the deleted items?

It covers

  • Ttl misconfig
  • Dynamodb
  • Epoch units
  • Auto delete