I want to run FoundationDB on multi Kubernetes clusters for fault tolerance and improve speed for application by deploy each kubernetes cluster on each aws/gcp region like
us-east-2,ap-south-1,eu-west-3
One requirement is that all Pods are able to communicate with each other. So you have to ensure that your network setup supports this.
I deploy and run FDB cluster by FDB operator on one Kubernetes cluser already.
To run on another K8s cluster, if I apply yaml file to new cluster, how can it synchronize data with current active cluster?
Or we have another way to run FDB cluster on multi K8s cluster and make sure data in those clusters are consistent?
This config example shows the basic ideas how to setup a FDB cluster across multiple Kubernetes clusters (each handled as a dedicated DC). In the future we have plans to make the operations of multi-dc (or multi-Kubernetes) FDB clusters easier with the operator. The import bit here is the following (we have an issue open in GitHub to document the HA setup better and probably it’s a good time to do that soon):
First you have to create a FDB cluster running in a single Kubernetes (you already have that) and you have to ensure that the configuration of that cluster uses a unique (for the FDB cluster) datacenter and a unique processGroupIDPrefix (you can use the same value there) https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/config/tests/multi_dc/final.yaml#L17 e.g. in your case you could call it us-east-2. Now you can update the config with the following changes:
Update the regions config of the database configuration in the current existing FoundationDBCluster (see: https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/config/tests/multi_dc/final.yaml#L25-L45). You probably can copy the config there and just replace the DC IDs with the real names. A higher priority means the DC is preferred over a lower priority.
In addition update the seedConnectionString to the current connection string of the cluster (https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/config/tests/multi_dc/final.yaml#L21). You can get that by either fetching the status of the FoundationDBCluster or by using fdbcli.
Once this is done you can create the FoundationDBCluster resources in the other two Kubernetes clusters using the same configuration except for the processGroupIDPrefix and the dataCenter those should be replaced by your new values (the ones that you defined in the database configuration).
Depending on the size of the cluster (or actually the stored values) it might take some minutes/hours to synchronise all data between the clusters.
One heads up if you make any changes to the database configuration ones you have an HA cluster running you have to make those changes in all configurations (in all FoundationDBClusters) otherwise you have conflicting settings which could result in the different operator instances trying to reconfigure the DB the whole time.
Another question if I have current FDB cluster use storage engine is
ssd-2, then I want to create new
K8s cluster and extend current FDB cluster on it with storage engine ismemory, can I do it?
That currently not possible since the database configuration is from an FDB perspective defined as a global configuration. You might be able to use the storage wiggle mechanism or something like this to emulate this, but I don’t think those were create for such a use case.