Durable append-only log

DesignHard40 min

Your task

Design a durable append-only / write-ahead-log store that other services use as their commit log (think a Kafka/BookKeeper-style log). Constraints: writes must be durable before ack (no data loss on a single-node power failure), ordered within a partition, high append throughput (hundreds of MB/s per partition), readers tail the log with low latency, and you replicate for fault tolerance. Cover the append path, durability guarantee, and replication.

It covers

  • Write ahead log
  • Replication
  • Fsync