I’m migrating some machines which participate to an FDB cluster.
I will phase out the old 5 machines which host the coordinators and I will be required to change the coordinator roles to process running on the new machines.
Is it better to do the coordinator change in a single step or gradually?
The gradual way would be to do 2 change steps by changing the coordinator list to 3 old + 2 new machines, and then on a second step switch out the 3 remaining old ones to 3 new ones having all 5 be new ones at that point.
Or in a single step by just telling the cluster the new 5 coordinators are on 5 new nodes in one operation?
EDIT: A further question, when then I will be excluding old nodes from the cluster, is there any downside to excluding half of the nodes in the cluster in one step? In my mind running a single exclude command will make it so the recalculation of the data topology is as optimal as possible.
Compared to running excludes node by node waiting for the cluster to fully heal.
The gradual way would be to do 2 change steps by changing the coordinator list to 3 old + 2 new machines, and then on a second step switch out the 3 remaining old ones to 3 new ones having all 5 be new ones at that point.
Or in a single step by just telling the cluster the new 5 coordinators are on 5 new nodes in one operation?
From the experience with the operator it’s better to change all coordinators in a single command. Changing coordinators will cause a recovery which could impact clients for a short amount of time.
EDIT: A further question, when then I will be excluding old nodes from the cluster, is there any downside to excluding half of the nodes in the cluster in one step? In my mind running a single exclude command will make it so the recalculation of the data topology is as optimal as possible.
Compared to running excludes node by node waiting for the cluster to fully heal.
I guess one of the downsides is that the read/write traffic will be elevated as more data must be replicated. On the other hand excluding all nodes at once has the benefit that you reduce unnecessary data movement, e.g. because some shards are replicated to the old not yet excluded nodes. And running a single exclude command will also reduce the observed recoveries.