FDB EKS --DNS issue

Using FoundationDB Kubernetes Operator
1

we are trying to deploy FDB in EKS and using the
routing:
#defineDNSLocalityFields: true
useDNSInClusterFile: true

in our cluster.yaml

after deploying we are not able to connect to the db
[root@test-cluster-log-3 /]# fdbcli
Using cluster file `/var/dynamic-conf/fdb.cluster’.

The database is unavailable; type `status’ for more information.

Welcome to the fdbcli. For help, type `help’.
fdb> status details

Using cluster file `/var/dynamic-conf/fdb.cluster’.

The coordinator(s) have no record of this database. Either the coordinator
addresses are incorrect, the coordination state on those machines is missing, or
no database has been created.

test-cluster-log-1.test-cluster.platform-fdb.svc.cluster.local:4501 (reachable)
test-cluster-log-2.test-cluster.platform-fdb.svc.cluster.local:4501 (reachable)
test-cluster-log-3.test-cluster.platform-fdb.svc.cluster.local:4501 (reachable)

Unable to locate the data distributor worker.

Unable to locate the ratekeeper worker.

fdb>

can anyone help with this

here is some more info

test_cluster:kADaIC2PQGMtgkHs0QJUs8edyWuBxwGy@test-cluster-log-1.test-cluster.platform-fdb.svc.cluster.local:4501,test-cluster-log-2.test-cluster.platform-fdb.svc.cluster.local:4501,test-cluster-log-3.test-cluster.platform-fdb.svc.cluster.local:4501

logs:

{“level”:“error”,“ts”:1686084346.5272686,“logger”:“controller”,“msg”:“Error in reconciliation”,“namespace”:“platform-fdb”,“cluster”:“test-cluster”,“subReconciler”:“controllers.chooseRemovals”,“requeueAfter”:0,“error”:“FoundationDB error code 2104 (Connection string invalid)”,“stacktrace”:“github.com/FoundationDB/fdb-kubernetes-operator/controllers.(*FoundationDBClusterReconciler).Reconcile\n\t/workspace/controllers/cluster_controller.go:180\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234”}
{“level”:“error”,“ts”:1686084346.5273292,“msg”:"Reconciler

2

Can you make sure you followed these steps: fdb-kubernetes-operator/customization.md at main · FoundationDB/fdb-kubernetes-operator · GitHub? Based on the error message it seems like the operator is using an older fdb library version than 7.1 as thee primary version. Can you also confirm that the cluster was created with 7.1?

3

ok. thanks that issue is resolved, we got the DNS working,

also want to check what is the lastest version we can install in EKS ? is it 7.1.26 ?
or can we even have 7.2 in EKS…

we only see this image: foundationdb/foundationdb-kubernetes-sidecar:7.1.26-1
here

4

The operator supports the following FDB versions: https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/compatibility.md#supported-versions. You can find all possible versions here: Docker and here Docker.

5

i changed the version from 7.1.26 to 7.2.0 in the deployment.yaml file and applied that with kubectl apply command , but the version is not getting updated
any reason why

6

That sounds like you hit: Upgrades from 7.1 to 7.2 are not working · Issue #1468 · FoundationDB/fdb-kubernetes-operator · GitHub, I have to write the documentation for this. If you look at the operator logs, you should see an error message like this: “… clients do not support version …”. In order to let the operator proceed you have to set ignoreLogGroupsForUpgrade, a minimalistic example would be:

apiVersion: apps.foundationdb.org/v1beta2
kind: FoundationDBCluster
metadata:
  name: test-cluster
spec:
  version: 7.2.0
  # other fields
  automationOptions:
    ignoreLogGroupsForUpgrade:
    - fdb-kubernetes-operator
  # other fields

I try to get the documentation for this done by the end of this week.

Btw. regarding using 7.2.0: Which version are recommended for production 7.1/7.2/7.3?

7

i updated my cluster.yaml with those parameters u specified and again did a kubectl apply… its not changing the version and i see the same messages in logs
this is my yaml file.
apiVersion: apps.foundationdb.org/v1beta2
kind: FoundationDBCluster
metadata:
name: test-cluster
spec:
automationOptions:
replacements:
enabled: true
faultDomain:
key: foundationdb.org/none
labels:
filterOnOwnerReference: false
matchLabels:
foundationdb.org/fdb-cluster-name: test-cluster
processClassLabels:
- foundationdb.org/fdb-process-class
processGroupIDLabels:
- foundationdb.org/fdb-process-group-id
minimumUptimeSecondsForBounce: 60
processCounts:
cluster_controller: 1
stateless: -1
processes:
general:
customParameters:
- knob_disable_posix_kernel_aio=1
podTemplate:
spec:
containers:
- name: foundationdb
resources:
requests:
cpu: 100m
memory: 128Mi
securityContext:
runAsUser: 0
- name: foundationdb-kubernetes-sidecar
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
securityContext:
runAsUser: 0
initContainers:
- name: foundationdb-kubernetes-init
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
securityContext:
runAsUser: 0
volumeClaimTemplate:
spec:
resources:
requests:
storage: 16G
routing:
defineDNSLocalityFields: true
sidecarContainer:
enableLivenessProbe: true
enableReadinessProbe: false
useExplicitListenAddress: true
version: 7.2.0
automationOptions:
ignoreLogGroupsForUpgrade:
- fdb-kubernetes-operator

and here are logs
{“level”:“error”,“ts”:1686843990.5132756,“logger”:“controller”,“msg”:“Error in reconciliation”,“namespace”:“default”,“cluster”:“test-cluster”,“subReconciler”:“controllers.checkClientCompatibility”,“requeueAfter”:0,“error”:“fork/exec /usr/bin/fdb/7.1/fdbcli: no such file or directory”,“stacktrace”:“github.com/FoundationDB/fdb-kubernetes-operator/controllers.(*FoundationDBClusterReconciler).Reconcile\n\t/workspace/controllers/cluster_controller.go:180\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234”}

8

fork/exec /usr/bin/fdb/7.1/fdbcli is telling you that the 7.1 binaries and libs are missing in the operator.