Blob store durability and tiering

DesignHard50 minAmazon and Google style

Your task

Design the storage engine for a blob/object store (S3-like) holding 50PB across trillions of objects ranging from 1KB to 5GB, with 11-nines durability, high read throughput, and cost as a primary constraint (mostly cold data, a hot working set of ~2%). You control the data plane down to the disks. Describe how objects are stored and addressed, the durability mechanism, the metadata/index for object lookup, hot/cold tiering, and the central trade-off versus simple replication.

It covers

  • Object storage
  • Tiering
  • Metadata