I have setup foundationdb using the operator on an openshift/k8s cluster. Everything is running fine if I try to connect to it on the k8s cluster. But what should I need to do if I want to run an java client on a host outside of the cluster to connect to it? I tried VPC-subnet to map the coordinators ip to the external ip but that does not work. Any suggestion on what can be done?
That was discussed in this issue: How can I visit the fdb server from outside of Kubernetes , eg, my desktop? · Issue #789 · FoundationDB/fdb-kubernetes-operator · GitHub. In order to be able to connect to the FDB cluster you have to be part of the Pod network and you must be able to directly communicate with those Pods e.g. by having a VPN or something like that into your cluster.
Following that issue 789 and then 391, I don’t see any more discussion in 391. So, is there a solution/workaround on this?
One more thing that we tried, we setup service (svc) and mapped all the storage pods to external ip and replaced the cluster file with those external ip when we use the fdbcli client to connect to the db.
Of course, we can’t get to it. When looking at the status, it returned this: “Unable to communicate with the cluster controller at 172.17.20.234:4501 to get status.” That ip is one of the stateless pod ip internally so the fdbcli did reach the coordinator some how but the coordinator tell it to talk to the controller on an internal ip which the fdbcli will not be able to do so. So, any advice on how to resolve this?
Currently there is no solution for that, you must be able to directly talk to the Pods. A workaround for that would be to allow services to be created as Type Loadbalancer, but that would be pretty “costly” since all Services in front of a Pod would get a Loadbalancer/cluster external IP (what that means depends on your Kubernetes setup).
You would need to change the
public_address in the configuration Configuration — FoundationDB 6.3 but it should be easier to add the support for Loadbalancer services than doing that manually (in addition to that the operator will try to change the config back once you do that manually).
I currently has services defined for all storage pods, so are you saying that I need to do the same thing for all the pods (maybe with the exception of log pods)? Assuming I do that for all storage+stateless pods, one thing I am still not too clear on is when using fdbcli -C clusterfile.txt and I replaced all the ip in that file with the external IP, how to make the coordinator to return the external IP instead of the internal IP when the fdbcli needs to talk to the controller? Do I also need to configure the public address as well as defining the services?