Currently our FDB cluster has 70 storage nodes (K8s pods). We want to add 10 more to it. Our replication factor is 3, using FDB v6.2.27.
Is it better to add 10 pods at once or add in smaller batches (like 5+5, or 3+3+4)?
My main goal here is to minimize the temporary increase of disk usage during the rebalance. I don’t know how much (in terms of percentage) the rebalance would hike the disk usage. Is there a formula for a rough estimate?
In the past we had some unpleasant experience with the temp disk space spike, Last year with FDB v6.2.11 we had a trouble (Newly added storage nodes have disk usage at 98%). We upgraded to v6.2.27. It has worked better.
Last week our OS maintenance made several storage pods dropped out of the fdb cluster due to network security issue. Then I saw the disk usage in some pods go up to 90% during the rebalance. We know when disk usage reached 95%, the fdb cluster will be halted.
Therefore we’d like to be very cautious this time. Thanks.