Backfill incomplete at code cutover
Your task
An expand-contract migration on the subscriptions service is mid-flight. Step 1 (last week): add a nullable billing_cycle_anchor column, backfill it, and have the NEW code read it with a COALESCE fallback to the old created_at. Step 2 ran in today's 10:00 deploy: drop the COALESCE and read billing_cycle_anchor directly, assuming the backfill is complete. At 10:06, ~0.5% of invoice generations crash with NoneType has no attribute 'day', all for accounts created in a narrow window. Dashboards: error rate stepped up exactly at 10:06; DB is healthy; the backfill job's dashboard shows 100% complete and green. The crashing accounts were all created BETWEEN the start of the backfill and the COALESCE-removal deploy. Triage and mitigate.
It covers
- Forward compatibility
- Read path skew
- Expand contract