Blob store garbage collection

DesignHard45 minAmazon style

Your task

Design the garbage collection / orphan-reaping subsystem for a large blob store where the metadata layer (a database mapping logical keys to physical blob locations) and the data layer (the actual stored blobs across thousands of storage nodes) can drift out of sync. Orphans accumulate: a blob written but its metadata commit failed, a metadata entry deleted but the blob not yet freed, multipart uploads abandoned mid-flight. At 50PB you can't afford to leak storage, but you also must NEVER delete a blob that's actually still referenced. Design a safe, scalable reaper.

It covers

  • Garbage collection
  • Orphan reaping
  • Distributed consistency