Adaptive partition splitting under skew

DesignHard45 minAmazon and Google style

Your task

Design the adaptive partition-splitting mechanism for an auto-sharded key-value store (think DynamoDB/Bigtable-class) that must keep partitions balanced as data and traffic grow, without an operator picking shard boundaries. A partition can get hot two ways: it grows too large (size), or one range of keys suddenly gets a traffic spike (a viral key range), even if it's small. Design how the system detects a partition needs splitting, where it places the split point, and how it splits live without dropping or misrouting requests, plus the failure case this design still struggles with.

It covers

  • Hot partition split
  • Adaptive splitting
  • Load based