Strongly consistent metadata control plane

DesignHard50 minAmazon and Google style

Your task

Design the metadata/control plane for a distributed object store that must hold the authoritative mapping of buckets to shards to storage nodes (a few hundred GB of metadata, ~50k writes/sec of placement changes, millions of reads/sec). The mapping must be strongly consistent, two clients must never disagree on which node owns a shard, and must survive losing a minority of control-plane nodes with no data loss. How do you build it and how do you scale the read path without weakening consistency?

It covers

  • Raft
  • Log replication
  • Membership change