Tiered storage hot-cold migration

DesignHard50 minSnowflake and Databricks style

Your task

Design the tiered-storage layer for a database/analytics engine whose dataset (500TB and growing) far exceeds what fits on local SSD, but 95% of queries touch data from the last 30 days. You want recent/hot data on fast local NVMe and old/cold data on cheap object storage (S3-class), with migration happening automatically as data ages, and queries that occasionally reach into cold data must still work, just slower. Design how data is placed across tiers, how it migrates hot to cold (and back if it gets hot again), how a query transparently reads across tiers, and the trade-off you're managing.

It covers

  • Tiered storage
  • Hot cold migration
  • Object store offload