HI,
I have setup foundationdb using the operator on an openshift/k8s cluster. Everything is running fine if I try to connect to it on the k8s cluster. But what should I need to do if I want to run an java client on a host outside of the cluster to connect to it? I tried VPC-subnet to map the coordinators ip to the external ip but that does not work. Any suggestion on what can be done?
Thanks,
1 Like
That was discussed in this issue: How can I visit the fdb server from outside of Kubernetes , eg, my desktop? · Issue #789 · FoundationDB/fdb-kubernetes-operator · GitHub. In order to be able to connect to the FDB cluster you have to be part of the Pod network and you must be able to directly communicate with those Pods e.g. by having a VPN or something like that into your cluster.
Hi Johannes,
Following that issue 789 and then 391, I don’t see any more discussion in 391. So, is there a solution/workaround on this?
One more thing that we tried, we setup service (svc) and mapped all the storage pods to external ip and replaced the cluster file with those external ip when we use the fdbcli client to connect to the db.
Of course, we can’t get to it. When looking at the status, it returned this: “Unable to communicate with the cluster controller at 172.17.20.234:4501 to get status.” That ip is one of the stateless pod ip internally so the fdbcli did reach the coordinator some how but the coordinator tell it to talk to the controller on an internal ip which the fdbcli will not be able to do so. So, any advice on how to resolve this?
Currently there is no solution for that, you must be able to directly talk to the Pods. A workaround for that would be to allow services to be created as Type Loadbalancer, but that would be pretty “costly” since all Services in front of a Pod would get a Loadbalancer/cluster external IP (what that means depends on your Kubernetes setup).
You would need to change the public_address in the configuration Configuration — FoundationDB 6.3 but it should be easier to add the support for Loadbalancer services than doing that manually (in addition to that the operator will try to change the config back once you do that manually).
Hi,
I currently has services defined for all storage pods, so are you saying that I need to do the same thing for all the pods (maybe with the exception of log pods)? Assuming I do that for all storage+stateless pods, one thing I am still not too clear on is when using fdbcli -C clusterfile.txt and I replaced all the ip in that file with the external IP, how to make the coordinator to return the external IP instead of the internal IP when the fdbcli needs to talk to the controller? Do I also need to configure the public address as well as defining the services?
Thanks,
@johscheuer Hi expert, why if I change the public_address in configuration file manually, the operator will try to change the config back? How can I stop that? If I use service in front of a pod to provide external IP, that would lead to change in configuration file too, with both public_address and listen_address in there, and very similarly the operator would change the config back. Any advice to stop it? Thanks in advance.
@tangerine Cluster controller process is in one of the stateless pods, so if you change all the stateless pods IP to external IP, the coordinators would return some external IP I think.
This is the idea of the operator or a Kubernetes operator in general to reconcile to a desired state. If you want to modify something and don’t want the operator to change it back to the desired state you can use the skip setting https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/cluster_spec.md#foundationdbclusterspec keep in mind that when this setting is enable the operator will stop doing anything on the cluster so I would not recommend that besides for some manual tests or incident response.
To achieve your use case you want to configure the cluster spec section for the routingConfiguration (https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/cluster_spec.md#routingconfig) with publicIPSource == service. See: https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/manual/customization.md#service-ips