Concurrent backfill with live ingest
Your task
A live streaming pipeline continuously ingests events into a partitioned fact table that powers production dashboards and downstream models. You've shipped a corrected transformation and need to backfill the last 180 days, but you CANNOT pause live ingest and the dashboards must stay queryable throughout. Re-running the new logic over old raw data alongside the live writes risks double-counting at the boundary, partial/torn partitions visible to readers, and contention with the live job. Design an idempotent backfill that runs concurrently with live ingest with zero downtime and no double-counting.
It covers
- Idempotent backfill
- Live ingest coexistence
- Watermark boundary