Adaptive partition splitting under skew
Your task
Design the adaptive partition-splitting mechanism for an auto-sharded key-value store (think DynamoDB/Bigtable-class) that must keep partitions balanced as data and traffic grow, without an operator picking shard boundaries. A partition can get hot two ways: it grows too large (size), or one range of keys suddenly gets a traffic spike (a viral key range), even if it's small. Design how the system detects a partition needs splitting, where it places the split point, and how it splits live without dropping or misrouting requests, plus the failure case this design still struggles with.
It covers
- Hot partition split
- Adaptive splitting
- Load based