Backfill race corrupts denormalized field
Your task
A backfill recomputes a denormalized lifetime_spend on each account by reading the account, computing the new value, and writing it back (read-modify-write, no locking). It ran during business hours at ~3k accounts/sec. Afterward, finance finds that for accounts that placed an order while the backfill was running, lifetime_spend is wrong, sometimes missing the new order, sometimes missing the backfill's correction. Dashboards: write success 100%, no errors, no lag. How do you triage, stop the divergence, and correct the affected accounts?
It covers
- Lost update
- Read modify write
- Concurrency
- Backfill vs live