Rebalance transfer stalls

On-call incidentHard35 min

Your task

You added two nodes to a 10-node sharded storage cluster to relieve capacity pressure. 18 hours later on-call is paged: the rebalance/data-migration is stuck at ~62% complete and has stopped progressing; meanwhile the three originally-hottest nodes are at 91% disk and one is approaching full; client write latency on those shards is degrading. Dashboards: rebalance throughput dropped to ~0 bytes/s about 3 hours ago; one moving shard shows repeated transfer-retry/timeout errors; the new nodes are at only 18% disk (barely received data); a specific large shard (1.2 TB) keeps failing mid-transfer; network between racks is fine. The cluster won't shed load off the hot nodes until the move completes. How do you triage a stuck rebalance after a node add?

It covers

  • Rebalance
  • Data movement
  • Node add
  • Hot shard