Migrating a triple cluster to a three_data_hall cluster without unavailability

Using FoundationDB Kubernetes Operator
1

Hi there,

after a lengthy development process I can confirm that it is possible to migrate a triple cluster to a three_data_hall cluster without going through any unavailability.

Let me enumerate the steps:

  1. create 3 clones of the original k8s FoundationDB cluster object, thus still with triple replication, and set a different name/processGroupIDPrefix/dataHall/dataCenter for them; these cluster objects must also have skip: true upon creation via kubectl create so that operator will not attempt to configure them and they must have their seed connection string set to the original cluster
  2. on these 3 new FoundationDB cluster objects set the configured state and connection string in their status subresource: kubectl patch fdb my_new_cluster... --type=merge --subresource status --patch "status: {configured: true, connectionString: \"...\" }"
  3. set back skip: false
  4. start a lengthy exclude procedure that will exclude all the processes of the original cluster; I exclude them in this order: log, storage, coordinator, stateless
  5. delete the original cluster once all exclusions are complete
  6. set redundancyMode to three_data_hall for the 3 new per-hall FoundationDB cluster objects, one after another
  7. patch seed connection string of two of the 3 per-hall clusters to point to the one which will keep an empty seed connection string e.g. if you have A,B,C, set the seed connection string of B and C to point to A and make sure that A has no seed connection string; this step is not crucial but practically helpful sometimes

Shall I contribute this to the documentation? Not sure if it might be found useful by some other party.

1 Like