Using Service IPs over Pod IPs and running a DR to another cluster?

Using FoundationDB Kubernetes Operator
1

Hello,

I’m hoping for a little bit of advice, we’re running FDB in AWS EKS using the fdb-k8s-operator. We decided for stability to adopt using services for our public IP source .spec.routing.publicIpSource=service.

This has worked well internally within a k8s cluster and for the most part we’re very happy with the solution, as part of our infrastructure work we’ve looked at having a DR cluster using the dr_agent (we’re running it up on a pod using the existing images/scripts - just outside the operator). We’ve had this working well to another FDB cluster within the same k8s cluster.

However we’ve now come to attempt to do this cross region (the same problem between k8s clusters would exist) - because we’ve chosen to use service addresses as our publicIpSource they are unroutable outside of the k8s cluster.

Cluster files obviously reference the service addresses, and even if I replaced them, I suspect that as the DR agents needed to talk to the logs/storage we’d see connection issues/it wouldn’t work.

Has anyone attempted to do this and managed the best of both worlds, service IPs and cross k8s stack DR replication? This feels like something that dns might eventually solve (and we could just drop back to pod addresses).

Kind regards,

Peter.

2

I haven’t used such a setup but if you figure out the details I would appreciate if you could put together a doc or a blog post that can be referenced. I’m not to experienced with the possibilities for AWS EKS but in the end to allow cross-cluster communication you have to ensure that either the service IPs are available from both clusters or that the Pod IPs are available from both clusters (not sure how much configuration possibilities are exposed by AWS).

Depending on your use case using a service-mesh might be another option (but adding some additional complexity to the setup).

1 Like