Parquet schema evolution

DesignMedium40 minDatabricks and Netflix style

Your task

A data lake stores 5 years of event data as Parquet in a lakehouse table queried by Spark and Trino. Product needs ongoing schema changes: add columns, rename a column, change a column's type (int to bigint, string to struct), and reorder columns, without rewriting all 5 years of files and without breaking old queries or jobs that read older partitions. Files written under old schemas must still be readable alongside new ones. Design how the table handles schema evolution so reads stay correct across schema versions.

It covers

  • Data lake evolution
  • Column changes
  • Query compatibility