I’m hoping for a little bit of advice, we’re running FDB in AWS EKS using the fdb-k8s-operator. We decided for stability to adopt using services for our public IP source .spec.routing.publicIpSource=service.
This has worked well internally within a k8s cluster and for the most part we’re very happy with the solution, as part of our infrastructure work we’ve looked at having a DR cluster using the dr_agent (we’re running it up on a pod using the existing images/scripts - just outside the operator). We’ve had this working well to another FDB cluster within the same k8s cluster.
However we’ve now come to attempt to do this cross region (the same problem between k8s clusters would exist) - because we’ve chosen to use service addresses as our publicIpSource they are unroutable outside of the k8s cluster.
Cluster files obviously reference the service addresses, and even if I replaced them, I suspect that as the DR agents needed to talk to the logs/storage we’d see connection issues/it wouldn’t work.
Has anyone attempted to do this and managed the best of both worlds, service IPs and cross k8s stack DR replication? This feels like something that dns might eventually solve (and we could just drop back to pod addresses).