Blob store with deduplication

DesignHard40 minAmazon style

Your task

Design a blob store with global deduplication for a backup/sync product (think Dropbox-style file storage). Constraints: petabytes stored, high duplication across users (same OS images, shared documents), block-level dedup with variable-size content-defined chunking, 10 nines durability, and you must never lose a block that is still referenced. Cover chunking, the dedup index, and reference lifecycle.

It covers

  • Deduplication
  • Content addressing
  • Reference counting