after a lengthy development process I can confirm that it is possible to migrate a triple cluster to a three_data_hall cluster without going through any unavailability.
Let me enumerate the steps:
create 3 clones of the original k8s FoundationDB cluster object, thus still with triple replication, and set a different name/processGroupIDPrefix/dataHall/dataCenter for them; these cluster objects must also have skip: true upon creation via kubectl create so that operator will not attempt to configure them and they must have their seed connection string set to the original cluster
on these 3 new FoundationDB cluster objects set the configured state and connection string in their status subresource: kubectl patch fdb my_new_cluster... --type=merge --subresource status --patch "status: {configured: true, connectionString: \"...\" }"
set back skip: false
start a lengthy exclude procedure that will exclude all the processes of the original cluster; I exclude them in this order: log, storage, coordinator, stateless
delete the original cluster once all exclusions are complete
set redundancyMode to three_data_hall for the 3 new per-hall FoundationDB cluster objects, one after another
patch seed connection string of two of the 3 per-hall clusters to point to the one which will keep an empty seed connection string e.g. if you have A,B,C, set the seed connection string of B and C to point to A and make sure that A has no seed connection string; this step is not crucial but practically helpful sometimes
Shall I contribute this to the documentation? Not sure if it might be found useful by some other party.
Thought I could share our process: We’ve been migrating to three_data_hall in the following way with the kubernetes operator:
Add the second and third foundationdb k8s resource with seedConnectionString set to the connection string of the running foundationdb cluster. These resources will now join the foundationdb cluster. Set name /processGroupIDPrefix /dataHall according to the ‘data hall’/availability zone.
Scale the original foundationdb k8s resource such that the cluster is of desired size (we scale down to ~1/3). Optionally remove the seedConnectionString of the second and third foundationdb k8s resource, as they are no longer used.
Update processGroupIDPrefix , dataHall, and nodeSelector (if applicable), of the original foundationdb k8s resource.
Switch redundancy_mode to three_data_hall in all three foundationdb k8s resources.
We use the nodeSelector to make sure that pods are in the right availability zone.
@johscheuer I have posted here the steps I used, which include more checks/changes as it was necessary due to various failure modes discovered in the process of migration.
