# FDB EKS --DNS issue

**URL:** <https://forums.foundationdb.org/t/fdb-eks-dns-issue/4000>\
**Category:** Kubernetes Operator\
**Created:** [June 6, 2023, 9:01pm UTC](https://forums.foundationdb.org/t/fdb-eks-dns-issue/4000 "2023-06-06T21:01:06Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Minaxi](https://avatars.discourse-cdn.com/v4/letter/m/2bfe46/32.png) [@Minaxi](https://forums.foundationdb.org/u/Minaxi)\
**Post date:** [June 6, 2023, 9:01pm UTC](https://forums.foundationdb.org/t/fdb-eks-dns-issue/4000/1 "2023-06-06T21:01:06Z")

</div>

we are trying to deploy FDB in EKS and using the  
routing:  
#defineDNSLocalityFields: true  
useDNSInClusterFile: true

in our cluster.yaml

after deploying we are not able to connect to the db  
[root@test-cluster-log-3 /]# fdbcli  
Using cluster file `/var/dynamic-conf/fdb.cluster’.

The database is unavailable; type `status’ for more information.

Welcome to the fdbcli. For help, type `help’.  
fdb\> status details

Using cluster file `/var/dynamic-conf/fdb.cluster’.

The coordinator(s) have no record of this database. Either the coordinator  
addresses are incorrect, the coordination state on those machines is missing, or  
no database has been created.

test-cluster-log-1.test-cluster.platform-fdb.svc.cluster.local:4501 (reachable)  
test-cluster-log-2.test-cluster.platform-fdb.svc.cluster.local:4501 (reachable)  
test-cluster-log-3.test-cluster.platform-fdb.svc.cluster.local:4501 (reachable)

Unable to locate the data distributor worker.

Unable to locate the ratekeeper worker.

fdb\>

can anyone help with this

here is some more info

test\_cluster:kADaIC2PQGMtgkHs0QJUs8edyWuBxwGy@test-cluster-log-1.test-cluster.platform-fdb.svc.cluster.local:4501,test-cluster-log-2.test-cluster.platform-fdb.svc.cluster.local:4501,test-cluster-log-3.test-cluster.platform-fdb.svc.cluster.local:4501

logs:

{“level”:“error”,“ts”:1686084346.5272686,“logger”:“controller”,“msg”:“Error in reconciliation”,“namespace”:“platform-fdb”,“cluster”:“test-cluster”,“subReconciler”:“controllers.chooseRemovals”,“requeueAfter”:0,“error”:“FoundationDB error code 2104 (Connection string invalid)”,“stacktrace”:“[github.com/FoundationDB/fdb-kubernetes-operator/controllers.(\*FoundationDBClusterReconciler).Reconcile](http://github.com/FoundationDB/fdb-kubernetes-operator/controllers.(*FoundationDBClusterReconciler).Reconcile)\n\t/workspace/controllers/cluster\_controller.go:180\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234”}  
{“level”:“error”,“ts”:1686084346.5273292,“msg”:"Reconciler

---

<div class="post-metadata">

**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\
**Post date:** [June 7, 2023, 8:20pm UTC](https://forums.foundationdb.org/t/fdb-eks-dns-issue/4000/2 "2023-06-07T20:20:20Z")

</div>

Can you make sure you followed these steps: [fdb-kubernetes-operator/customization.md at main · FoundationDB/fdb-kubernetes-operator · GitHub](https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/manual/customization.md#using-dns)? Based on the error message it seems like the operator is using an older fdb library version than 7.1 as thee primary version. Can you also confirm that the cluster was created with 7.1?

---

<div class="post-metadata">

**Author:** ![Minaxi](https://avatars.discourse-cdn.com/v4/letter/m/2bfe46/32.png) [@Minaxi](https://forums.foundationdb.org/u/Minaxi)\
**Post date:** [June 13, 2023, 5:51pm UTC](https://forums.foundationdb.org/t/fdb-eks-dns-issue/4000/3 "2023-06-13T17:51:06Z")

</div>

ok. thanks that issue is resolved, we got the DNS working,

also want to check what is the lastest version we can install in EKS ? is it 7.1.26 ?  
or can we even have 7.2 in EKS…

we only see this image: foundationdb/foundationdb-kubernetes-sidecar:7.1.26-1  
here

> <https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/config/samples/deployment.yaml>

---

<div class="post-metadata">

**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\
**Post date:** [June 14, 2023, 11:52am UTC](https://forums.foundationdb.org/t/fdb-eks-dns-issue/4000/4 "2023-06-14T11:52:43Z")

</div>

> [@Minaxi](#):
>
> ok. thanks that issue is resolved, we got the DNS working,
> 
> also want to check what is the lastest version we can install in EKS ? is it 7.1.26 ?  
> or can we even have 7.2 in EKS…
> 
> we only see this image: foundationdb/foundationdb-kubernetes-sidecar:7.1.26-1  
> here

The operator supports the following FDB versions: [https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/compatibility.md#supported-versions](https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/compatibility.md#supported-versions). You can find all possible versions here: [Docker](https://hub.docker.com/r/foundationdb/foundationdb-kubernetes-sidecar/tags) and here [Docker](https://hub.docker.com/r/foundationdb/foundationdb/tags).

---

<div class="post-metadata">

**Author:** ![Minaxi](https://avatars.discourse-cdn.com/v4/letter/m/2bfe46/32.png) [@Minaxi](https://forums.foundationdb.org/u/Minaxi)\
**Post date:** [June 14, 2023, 9:20pm UTC](https://forums.foundationdb.org/t/fdb-eks-dns-issue/4000/5 "2023-06-14T21:20:19Z")

</div>

i changed the version from 7.1.26 to 7.2.0 in the deployment.yaml file and applied that with kubectl apply command , but the version is not getting updated  
any reason why

---

<div class="post-metadata">

**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\
**Post date:** [June 15, 2023, 4:51am UTC](https://forums.foundationdb.org/t/fdb-eks-dns-issue/4000/6 "2023-06-15T04:51:25Z")

</div>

That sounds like you hit: [Upgrades from 7.1 to 7.2 are not working · Issue #1468 · FoundationDB/fdb-kubernetes-operator · GitHub](https://github.com/FoundationDB/fdb-kubernetes-operator/issues/1468), I have to write the documentation for this. If you look at the operator logs, you should see an error message like this: “… clients do not support version …”. In order to let the operator proceed you have to set `ignoreLogGroupsForUpgrade`, a minimalistic example would be:

```auto
apiVersion: apps.foundationdb.org/v1beta2
kind: FoundationDBCluster
metadata:
  name: test-cluster
spec:
  version: 7.2.0
  # other fields
  automationOptions:
    ignoreLogGroupsForUpgrade:
    - fdb-kubernetes-operator
  # other fields

```

I try to get the documentation for this done by the end of this week.

Btw. regarding using 7.2.0: [Which version are recommended for production 7.1/7.2/7.3?](https://forums.foundationdb.org/t/which-version-are-recommended-for-production-7-1-7-2-7-3/3994)

---

<div class="post-metadata">

**Author:** ![Minaxi](https://avatars.discourse-cdn.com/v4/letter/m/2bfe46/32.png) [@Minaxi](https://forums.foundationdb.org/u/Minaxi)\
**Post date:** [June 15, 2023, 3:56pm UTC](https://forums.foundationdb.org/t/fdb-eks-dns-issue/4000/7 "2023-06-15T15:56:30Z")

</div>

i updated my cluster.yaml with those parameters u specified and again did a kubectl apply… its not changing the version and i see the same messages in logs  
this is my yaml file.  
apiVersion: [apps.foundationdb.org/v1beta2](http://apps.foundationdb.org/v1beta2)  
kind: FoundationDBCluster  
metadata:  
name: test-cluster  
spec:  
automationOptions:  
replacements:  
enabled: true  
faultDomain:  
key: [foundationdb.org/none](http://foundationdb.org/none)  
labels:  
filterOnOwnerReference: false  
matchLabels:  
[foundationdb.org/fdb-cluster-name:](http://foundationdb.org/fdb-cluster-name:) test-cluster  
processClassLabels:  
- [foundationdb.org/fdb-process-class](http://foundationdb.org/fdb-process-class)  
processGroupIDLabels:  
- [foundationdb.org/fdb-process-group-id](http://foundationdb.org/fdb-process-group-id)  
minimumUptimeSecondsForBounce: 60  
processCounts:  
cluster\_controller: 1  
stateless: -1  
processes:  
general:  
customParameters:  
- knob\_disable\_posix\_kernel\_aio=1  
podTemplate:  
spec:  
containers:  
- name: foundationdb  
resources:  
requests:  
cpu: 100m  
memory: 128Mi  
securityContext:  
runAsUser: 0  
- name: foundationdb-kubernetes-sidecar  
resources:  
limits:  
cpu: 100m  
memory: 128Mi  
requests:  
cpu: 100m  
memory: 128Mi  
securityContext:  
runAsUser: 0  
initContainers:  
- name: foundationdb-kubernetes-init  
resources:  
limits:  
cpu: 100m  
memory: 128Mi  
requests:  
cpu: 100m  
memory: 128Mi  
securityContext:  
runAsUser: 0  
volumeClaimTemplate:  
spec:  
resources:  
requests:  
storage: 16G  
routing:  
defineDNSLocalityFields: true  
sidecarContainer:  
enableLivenessProbe: true  
enableReadinessProbe: false  
useExplicitListenAddress: true  
version: 7.2.0  
automationOptions:  
ignoreLogGroupsForUpgrade:  
- fdb-kubernetes-operator

and here are logs  
{“level”:“error”,“ts”:1686843990.5132756,“logger”:“controller”,“msg”:“Error in reconciliation”,“namespace”:“default”,“cluster”:“test-cluster”,“subReconciler”:“controllers.checkClientCompatibility”,“requeueAfter”:0,“error”:“fork/exec /usr/bin/fdb/7.1/fdbcli: no such file or directory”,“stacktrace”:“[github.com/FoundationDB/fdb-kubernetes-operator/controllers.(\*FoundationDBClusterReconciler).Reconcile](http://github.com/FoundationDB/fdb-kubernetes-operator/controllers.(*FoundationDBClusterReconciler).Reconcile)\n\t/workspace/controllers/cluster\_controller.go:180\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234”}

---

<div class="post-metadata">

**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\
**Post date:** [June 16, 2023, 8:47am UTC](https://forums.foundationdb.org/t/fdb-eks-dns-issue/4000/8 "2023-06-16T08:47:22Z")

</div>

> [@Minaxi](#):
>
> {“level”:“error”,“ts”:1686843990.5132756,“logger”:“controller”,“msg”:“Error in reconciliation”,“namespace”:“default”,“cluster”:“test-cluster”,“subReconciler”:“controllers.checkClientCompatibility”,“requeueAfter”:0,“error”:“fork/exec /usr/bin/fdb/7.1/fdbcli: no such file or directory”,“stacktrace”:“[github.com/FoundationDB/fdb-kubernetes-operator/controllers.(\*FoundationDBClusterReconciler).Reconcile](http://github.com/FoundationDB/fdb-kubernetes-operator/controllers.(*FoundationDBClusterReconciler).Reconcile)\n\t/workspace/controllers/cluster\_controller.go:180\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234”}

`fork/exec /usr/bin/fdb/7.1/fdbcli` is telling you that the 7.1 binaries and libs are missing in the operator.
