Resumable schema migration
Your task
You must migrate 5 years of a 20 TB event history from a legacy schema/storage (old Avro on HDFS, a deprecated partition scheme) to a new lakehouse table with a redesigned schema, WHILE new events keep landing every second and production queries must keep serving the whole time. A single giant batch job has failed twice (OOM, then a transient cluster outage at hour 9) and there's no clean restart point. Design a resumable, verifiable backfill + cutover that tolerates failures and proves the migrated data is correct before you switch reads over.
It covers
- Chunked migration
- Dual write
- Validation