My question is about deployment patterns using the FDB k8s Operator (https://github.com/FoundationDB/fdb-kubernetes-operator). When deploying the operator in global mode and then creating a cluster I noticed that the configmap containing the fdb cluster file is only created in the namespace in which the cluster was deployed. I was expecting that configmap would be generated in all namespaces. Is there a way to achieve this for the use case where you have microservices deployed in varying namespaces and want them to be able to access a single FDB deployment?
I had reviewed the user manual before asking the question; I don’t see any statements pertaining to my question above. Is there a particular piece of information you’re trying to point me to? Or are you just pointing to docs?
I was just pointing to docs on Global Mode. I think @alexmiller might be able to help with this question or point you in the right direction.
The operator only creates resources in the namespaces where you are running clusters. Are you trying to get a config map in a different namespace as a way of getting the cluster file to your clients? If so, that’s something that we unfortunately don’t have a clean, out-of-the box solution for. We could pursue a way of expressing that through the operator, but I’m not sure that doing all namespaces would be the best approach, since it creates reconciliation work that scales with the number of namespaces.
Another approach you could explore is to set up a simple web service that runs in the namespace with the FDB cluster and serves the cluster file based on the contents of the config map. Your other clients could call that namespace in startup or in an init container to pull the latest cluster file, and they would get updates automatically when the coordinator is changed as long as they maintain a connection to the cluster.
Another thing on our roadmap that could help with this is the RPC layer, which will allow clients to connect through a more standard discovery mechanism and protocol. Once that is available for FDB we will add direct support for it in the operator, and it should be easy from there to have the kind of cross-namespace connection you’re describing.
Hi John thanks for your response.
Are you trying to get a config map in a different namespace as a way of getting the cluster file to your clients?
That is exactly what we’re trying to achieve. I agree with you replicating the config map into all namespaces would be overkill, but being able to provide configuration on the namespaces it should be replicated to would be a very nice feature as I think this would be a common use case for people.
The web service approach may be a good short term solution for us until the RPC layer functionality is available or until the operator is modified to be able to publish the configmap to a list of namespaces.
We can try out the short term solution and see how it goes. Thanks again.