Large object S3-like store

DesignHard45 minAmazon style

Your task

Design the core write/read path of an S3-like object store focused on large objects: objects up to 5TB uploaded as multipart chunks over unreliable client connections, eleven-nines durability, strong read-after-write consistency for new objects, and high aggregate throughput. Ignore versioning/lifecycle, focus on how a multi-gigabyte object is durably written across the cluster and read back, and how the system stays consistent. How do you do it?

It covers

  • Multipart upload
  • Consistency
  • Erasure coding