Resumable schema migration

DesignHard45 minDatabricks style

Your task

You must migrate 5 years of a 20 TB event history from a legacy schema/storage (old Avro on HDFS, a deprecated partition scheme) to a new lakehouse table with a redesigned schema, WHILE new events keep landing every second and production queries must keep serving the whole time. A single giant batch job has failed twice (OOM, then a transient cluster outage at hour 9) and there's no clean restart point. Design a resumable, verifiable backfill + cutover that tolerates failures and proves the migrated data is correct before you switch reads over.

It covers

  • Chunked migration
  • Dual write
  • Validation