Data lake columnar storage

DesignHard45 minDatabricks and Snowflake style

Your task

Design the storage layer for a data lake holding petabytes of analytical data as columnar (Parquet) files on object storage, queried by Spark/Presto-style engines. Workloads are append-heavy ingestion plus large analytical scans that read a few columns across billions of rows, with occasional row-level updates/deletes (GDPR erasure). How do you organize files, make queries fast, and support mutation on an immutable object store?

It covers

  • Data lake
  • Columnar
  • Parquet