Backfill incomplete at code cutover
An expand-contract migration on the `subscriptions` service is mid-flight. Step 1 (last week): add a nullable `billing_cycle_anchor` column, backfill it, and have the NEW code read it with a COALESCE fallback to the old `created_at`. Step 2 ran in today's 10:00 deploy: drop the COALESCE and read `billing_cycle_anchor` directly, assuming the backfill is complete. At 10:06, ~0.5% of invoice generations crash with `NoneType has no attribute 'day'`, all for accounts created in a narrow window. Dashboards: error rate stepped up exactly at 10:06; DB is healthy; the backfill job's dashboard shows 100% complete and green. The crashing accounts were all created BETWEEN the start of the backfill and the COALESCE-removal deploy. Triage and mitigate.
Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.