Concurrent backfill with live ingest

DesignHard45 minDatabricks style

Your task

A live streaming pipeline continuously ingests events into a partitioned fact table that powers production dashboards and downstream models. You've shipped a corrected transformation and need to backfill the last 180 days, but you CANNOT pause live ingest and the dashboards must stay queryable throughout. Re-running the new logic over old raw data alongside the live writes risks double-counting at the boundary, partial/torn partitions visible to readers, and contention with the live job. Design an idempotent backfill that runs concurrently with live ingest with zero downtime and no double-counting.

It covers

  • Idempotent backfill
  • Live ingest coexistence
  • Watermark boundary