Pipeline migration validation

DesignHard50 minDatabricks and Snowflake style

Your task

You're migrating a critical revenue-reporting pipeline from a legacy SQL/cron system to a new framework (e.g. Spark/dbt on a lakehouse). The new pipeline must produce numbers that exactly match the old one before you can cut over, and you must backfill 3 years of history into the new system. Finance signs off only after proving the two produce identical results. The legacy system is still the source of truth and still running daily. Design the migration: how you backfill history, how you validate equivalence, and how you cut over with zero reporting gap and a rollback path.

It covers

  • Dual running
  • Migration
  • Verification