Backfill race corrupts denormalized field

On-call incidentHard40 min

Your task

A backfill recomputes a denormalized lifetime_spend on each account by reading the account, computing the new value, and writing it back (read-modify-write, no locking). It ran during business hours at ~3k accounts/sec. Afterward, finance finds that for accounts that placed an order while the backfill was running, lifetime_spend is wrong, sometimes missing the new order, sometimes missing the backfill's correction. Dashboards: write success 100%, no errors, no lag. How do you triage, stop the divergence, and correct the affected accounts?

It covers

  • Lost update
  • Read modify write
  • Concurrency
  • Backfill vs live