HI,
Is there any more details on setting up triple_data_hall with the FDB operator? In a bare metal env, I can setup each FDB process in a machine manually. But with the operator, how can I assign the pod location? Is there a way to assign it when we label the workers node to denote different hall?
As a test, I tried to enable three_data_hall using the databaseconfiguration section and set the mode to three_data_hall. But I run into questions that how many log/statelesss do I need to set in order to have a minimum functional three_data_hall? Also, does it know how to evenly distribute the pods or we need to some how use some kind of affinity rule for that?
Thanks,
three_data_hall is not directly supported in the operator. We have an open design discussion on how to support it, which we need to revisit: Add design for three_data_hall and three_datacenter by johscheuer · Pull Request #818 · FoundationDB/fdb-kubernetes-operator · GitHub
From that discussion, it is last active since July. And you said it is not directly supported, what’s missing? Is there a way to enable it?
Hello, since the design was initially from me. I try to review the design in the next weeks and bring it forward into an implementable status. There are a few things that we should change to support three data hall (see the review). One thing that we need to add is support for that replication mode: fdb-kubernetes-operator/foundationdbcluster_types.go at master · FoundationDB/fdb-kubernetes-operator · GitHub
One thing I still a bit confused on is how that locality/zone/machineid mapped to the compute nodes/pods in the kubernettes. As 3 data hall goes, we probably want to have 3 group of nodes so that if any one group is down, the cluster will still survive. It would have been nice if we can just list out the groups and let the operator spring up the necessary pods to form the cluster based on the group.
The node labels would be part of the requirement either your Kubernetes provider (many cloud provider do that already) have to label these nodes or you have to do that by yourself. The idea is then to have a flexible model to define the localities Add design for three_data_hall and three_datacenter by johscheuer · Pull Request #818 · FoundationDB/fdb-kubernetes-operator · GitHub e.g. for data_hall and zone those an be either some static configured values or they could be read from the node.
Should it also work with topology_spread_constraints?
You can already add topology spread constraint to your Pods if you want. The linked proposal already covers spread constraints so it’s intended once we implement it to add support for them.