Log broker storage layer

DesignHard50 min

Your task

Design the storage layer of a high-throughput append-only log broker. Requirements: sustain 2GB/s writes across partitions, retain data for 30 days but keep hot recent data fast, support replaying from any offset, replicate for durability, and support 'compacted' topics where only the latest value per key is retained. Disk is the bottleneck and you must keep per-partition cost low.

It covers

  • Broker storage
  • Log compaction
  • Tiered storage