# Seeing lots of rebalancing after fleet wide restarts

**URL:** <https://forums.foundationdb.org/t/seeing-lots-of-rebalancing-after-fleet-wide-restarts/2715>\
**Category:** Running FoundationDB\
**Tags:** performance\
**Created:** [May 26, 2021, 9:25pm UTC](https://forums.foundationdb.org/t/seeing-lots-of-rebalancing-after-fleet-wide-restarts/2715 "2021-05-26T21:25:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![rmanoj](https://avatars.discourse-cdn.com/v4/letter/r/c5a1d2/32.png) [@rmanoj](https://forums.foundationdb.org/u/rmanoj)\
**Post date:** [May 26, 2021, 9:25pm UTC](https://forums.foundationdb.org/t/seeing-lots-of-rebalancing-after-fleet-wide-restarts/2715/1 "2021-05-26T21:25:39Z")

</div>

We operate one of the largest FDB clusters. We happen to restart our cluster fleet wide for a kernal update. Since then, we have seen long stabilization periods. Specifically there was a long rebalancing occurring between the nodes. Has anyone who has run into this ? Any explanation of why this was happening and how to avoid this long stabilization ?. We would expect rebalancing to occur on adding or removing a node. This led to the storage queue bumping to above 1Gb. We have the following knob set changed from default. We changed custom shard size of `100MB` to the default of `500MB`  
# Knobs  
knob\_max\_shard\_bytes = 100000000

 ![Screen Shot 2021-05-26 at 2.24.25 PM](https://global.discourse-cdn.com/foundationdb/original/2X/d/da2584b5a2d5eb73a793a238bcef4e969e02f858.png)  
 ![Screen Shot 2021-05-26 at 2.21.37 PM](https://global.discourse-cdn.com/foundationdb/original/2X/0/0936ec0d9984c1d7f00a6685a11f45795ee6b2eb.png)  
 ![Screen Shot 2021-05-26 at 2.17.28 PM](https://global.discourse-cdn.com/foundationdb/original/2X/f/faae7fd5e699deab43a9d69f76967f86c452b883.png)  
 ![Screen Shot 2021-05-26 at 2.16.39 PM](https://global.discourse-cdn.com/foundationdb/original/2X/f/f65796c03afc35211a72e62a9350108dfcb4d93d.png)

---

<div class="post-metadata">

**Author:** ![mengxu](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/mengxu/32/893_2.png) [@mengxu](https://forums.foundationdb.org/u/mengxu)\
**Post date:** [May 31, 2021, 6:39pm UTC](https://forums.foundationdb.org/t/seeing-lots-of-rebalancing-after-fleet-wide-restarts/2715/2 "2021-05-31T18:39:24Z")

</div>

When shard size changes, data distribution will reshuffle data to create shards following the new size. For example, when you increase the size from 100MB o 500MB, existing shards will be merged to larger shards. Two to-be-merged shards can sit on two different hosts; Merging them requires relocating them.