# FoundationDBBackup cannot be reconciled properly **URL:** \ **Category:** Kubernetes Operator\ **Created:** [August 15, 2025, 10:03am UTC](https://forums.foundationdb.org/t/foundationdbbackup-cannot-be-reconciled-properly/5023 "2025-08-15T10:03:24Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![zmylk](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/zmylk/32/2013_2.png) [@zmylk](https://forums.foundationdb.org/u/zmylk)\ **Post date:** [August 15, 2025, 10:03am UTC](https://forums.foundationdb.org/t/foundationdbbackup-cannot-be-reconciled-properly/5023/1 "2025-08-15T10:03:24Z")
I suspect this is similar to the question > > > \### What happened? > > We have configured an FDB cluster using the v1.7.0 operator,… this is operating correctly and can be interacted with. The cluster has reconcilled correctly. We have now moved to configure backups, however upon creating the fdbbackup object the operator never finishes reconcilling and the backup is never started. > > \`\`\`console > \# kubectl get fdbbackup -n fdb-primary-dev primary-dev-backup 08:58:03 > NAME GENERATION RECONCILED AGE > primary-dev-backup 1 23m > \`\`\` > > The operator does attempt to create a backup. > > \`\`\`console > {"level":"info","ts":1657699169.9587805,"logger":"fdbclient","msg":"Running command","namespace":"fdb-primary-dev","cluster":"primary-dev","path":"/usr/bin/fdb/7.1/fdbbackup","args":\["/usr/bin/fdb/7.1/fdbbackup","status","--json","-C","/tmp/1088749738","--log","--log","--logdir","/var/log/fdb"\]} > {"level":"info","ts":1657699170.037034,"logger":"fdbclient","msg":"Command completed","namespace":"fdb-primary-dev","cluster":"primary-dev","output":"{\\"SchemaVersion\\":\\"1...."} > {"level":"info","ts":1657699170.0374086,"logger":"fdbclient","msg":"Running command","namespace":"fdb-primary-dev","cluster":"primary-dev","path":"/usr/bin/fdb/7.1/fdbbackup","args":\["/usr/bin/fdb/7.1/fdbbackup","start","-d","blobstore://XXXXXXXXXXXXXXX@s3.eu-west-1.amazonaws.com/primary-dev-20220713?bucket=foundationdb-backups-dev-XXXXXXXXXX-XXXXXXXXXXX","-s","1200","-z","-C","/tmp/2189325739","--log","--log","--logdir","/var/log/fdb"\]} > {"level":"error","ts":1657699180.0770085,"logger":"fdbclient","msg":"Error from FDB command","namespace":"fdb-primary-dev","cluster":"primary-dev","code":-1,"stdout":"","stderr":"","error":"signal: killed","stacktrace":"github.com/FoundationDB/fdb-kubernetes-operator/fdbclient.(\*cliAdminClient).StartBackup\\n\\t/workspace/fdbclient/admin\_client.go:595\\ngithub.com/FoundationDB/fdb-kubernetes-operator/controllers.startBackup.reconcile\\n\\t/workspace/controllers/start\_backup.go:45\\ngithub.com/FoundationDB/fdb-kubernetes-operator/controllers.(\*FoundationDBBackupReconciler).Reconcile\\n\\t/workspace/controllers/backup\_controller.go:88\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler\\n\\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:298\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem\\n\\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:253\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2\\n\\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:214"} > {"level":"error","ts":1657699180.0771945,"logger":"controller","msg":"Error in reconciliation","namespace":"fdb-primary-dev","backup":"primary-dev-backup","subReconciler":"controllers.startBackup","requeueAfter":0,"error":"signal: killed","stacktrace":"github.com/FoundationDB/fdb-kubernetes-operator/controllers.(\*FoundationDBBackupReconciler).Reconcile\\n\\t/workspace/controllers/backup\_controller.go:93\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).reconcileHandler\\n\\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:298\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem\\n\\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:253\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2\\n\\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:214"} > {"level":"error","ts":1657699180.0772355,"logger":"controller-runtime.manager.controller.foundationdbbackup","msg":"Reconciler error","reconciler group":"apps.foundationdb.org","reconciler kind":"FoundationDBBackup","name":"primary-dev-backup","namespace":"fdb-primary-dev","error":"signal: killed","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).processNextWorkItem\\n\\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:253\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller).Start.func2.2\\n\\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:214"} > \`\`\` > > If I connect to operator and look for the appropriate logs of that execution, the first obvious error I find is: > > \`\`\`console > \ > \`\`\` > > The only reference to a DNS lookup failed I can find leads me to this forum post on the fdb forum. https://forums.foundationdb.org/t/foundationdb-backup-s3-https-connection-failed/2340 > > Which suggests it might be a TLS configuration failure - we provide the fdbbackup k8s object with a valid CA chain for aws s3 and the client certificate, that's also provided with its key - though no TLS is required on the cluster at this point. > > We can see in the fdb operator logs that the TLS configuration is not provided to fdbbackup. > > \`\`\`console > \ > \`\`\` > > This might be unrelated though. > > If I reconsturct the start command and run it on one of the two backup agent nodes (rather than on the operator), the backup is issued and starts successfully. > > \`\`\`console > fdbbackup start -d blobstore://XXXXXXXXXX@s3.eu-west-1.amazonaws.com/primary-dev-20220712?bucket=foundationdb-backups-dev-XXXXXXXXXXX-XXXXXXXXX -s 1200 -z --log --log --logdir . > \`\`\` > > The operator then reconcils and completes. > > \### What did you expect to happen? > > The operator to create the backup and reconcil. > > \### How can we reproduce it (as minimally and precisely as possible)? > > 1. Install v1.7.0 fdb-operator, in global mode. > 2. Create a cluster in its own namespace. > 3. Wait for reconcilliation. > 4. Create a backup in the same namespace as the cluster. > > \### Anything else we need to know? > > I've attached the following logs and k8s objects. > > \[fdb-operator-new-fdbbackup-logs.txt\](https://github.com/FoundationDB/fdb-kubernetes-operator/files/9100809/fdb-operator-new-fdbbackup-logs.txt) > > \[fdbackup-start-on-fdb-operator-failure.txt\](https://github.com/FoundationDB/fdb-kubernetes-operator/files/9100810/fdbackup-start-on-fdb-operator-failure.txt) > > \[fdbbackup-k8s-object.txt\](https://github.com/FoundationDB/fdb-kubernetes-operator/files/9100811/fdbbackup-k8s-object.txt) > > \[fdbbackup-start-on-backup-agent-success.txt\](https://github.com/FoundationDB/fdb-kubernetes-operator/files/9100812/fdbbackup-start-on-backup-agent-success.txt) > > \[fdbcluster-k8s-object.txt\](https://github.com/FoundationDB/fdb-kubernetes-operator/files/9100813/fdbcluster-k8s-object.txt) > > > \### FDB Kubernetes operator > > \ > > \`\`\`console > $ kubectl fdb version > \# paste output here > \`\`\` > > The above does not work for me, I don't know if I've missed a stage during installation for that to work. > > We install the operator using CDK8s charts, from tag v1.7.0. > > Operator is running the current image: foundationdb/fdb-kubernetes-operator:v1.7.0 > > \ > > > \### Kubernetes version > > \ > > \`\`\`console > $ kubectl version > Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:36:49Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"darwin/arm64"} > Kustomize Version: v4.5.4 > Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.12-eks-a64ea69", GitCommit:"d4336843ba36120e9ed1491fddff5f2fec33eb77", GitTreeState:"clean", BuildDate:"2022-05-12T18:29:27Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"} > \`\`\` > > \ > > > \### Cloud provider > > \ > AWS EKS > \ But after I added the certificate, it did not solve the problem Certificate Creation Reference > **[fdb-kubernetes-operator/config/test-certs at release\_v1.40.0 ·...](https://github.com/FoundationDB/fdb-kubernetes-operator/tree/release_v1.40.0/config/test-certs)** > > A kubernetes operator for FoundationDB. Contribute to FoundationDB/fdb-kubernetes-operator development by creating an account on GitHub. Possibly useful documents ```auto https://github.com/FoundationDB/fdb-kubernetes-operator/blob/release_v1.40.0/docs/manual/tls.md ``` operator exception information ```auto {"level":"info","ts":"2025-08-15T09:57:00Z","logger":"fdbclient","msg":"Running command","path":"/usr/bin/fdb/7.1/fdbbackup","args":["/usr/bin/fdb/7.1/fdbbackup","status","--json","-C","/tmp/99a55c37-abb6-496f-8d67-d4cdfe9531b8-cli/2559054587","--log","--logdir","/var/log/fdb"]} {"level":"info","ts":"2025-08-15T09:57:00Z","logger":"fdbclient","msg":"Command completed","output":"{\"SchemaVersion\":\"1...."} {"level":"info","ts":"2025-08-15T09:57:00Z","logger":"fdbclient","msg":"Running command","path":"/usr/bin/fdb/7.1/fdbbackup","args":["/usr/bin/fdb/7.1/fdbbackup","start","-d","blobstore://seaweedfs@seaweedfs:8333/test-cluster?bucket=fdb-backups®ion=us-west-1&secure_connection=0","-s","120","-z","-C","/tmp/99a55c37-abb6-496f-8d67-d4cdfe9531b8-cli/390835421","--log","--logdir","/var/log/fdb"]} {"level":"error","ts":"2025-08-15T09:57:10Z","logger":"fdbclient","msg":"Error from FDB command","code":-1,"stdout":"","stderr":"","error":"signal: killed","stacktrace":"github.com/FoundationDB/fdb-kubernetes-operator/v2/fdbclient.(*cliAdminClient).runCommand\n\t/workspace/fdbclient/admin_client.go:265\ngithub.com/FoundationDB/fdb-kubernetes-operator/v2/fdbclient.(*cliAdminClient).StartBackup\n\t/workspace/fdbclient/admin_client.go:835\ngithub.com/FoundationDB/fdb-kubernetes-operator/v2/controllers.startBackup.reconcile\n\t/workspace/controllers/start_backup.go:52\ngithub.com/FoundationDB/fdb-kubernetes-operator/v2/controllers.(*FoundationDBBackupReconciler).Reconcile\n\t/workspace/controllers/backup_controller.go:97\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202"} {"level":"error","ts":"2025-08-15T09:57:10Z","logger":"controller","msg":"Error in reconciliation","namespace":"keli","backup":"test-cluster","reconciler":"controllers.startBackup","requeueAfter":0,"error":"signal: killed","stacktrace":"github.com/FoundationDB/fdb-kubernetes-operator/v2/controllers.processRequeue\n\t/workspace/controllers/controllers.go:92\ngithub.com/FoundationDB/fdb-kubernetes-operator/v2/controllers.(*FoundationDBBackupReconciler).Reconcile\n\t/workspace/controllers/backup_controller.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202"} {"level":"error","ts":"2025-08-15T09:57:10Z","msg":"Reconciler error","controller":"foundationdbbackup","controllerGroup":"apps.foundationdb.org","controllerKind":"FoundationDBBackup","FoundationDBBackup":{"name":"test-cluster","namespace":"keli"},"namespace":"keli","name":"test-cluster","reconcileID":"4f6cd5ca-f30f-470f-98db-7745fb2660ef","error":"signal: killed","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202"} ``` ```auto root@v-sjc031-devops-001:/opt/FoundationDB/fdb-kubernetes-operator/config/samples# kubectl describe FoundationDBBackup test-cluster Name: test-cluster Namespace: keli Labels: Annotations: API Version: apps.foundationdb.org/v1beta2 Kind: FoundationDBBackup Metadata: Creation Timestamp: 2025-08-15T07:08:35Z Generation: 1 Resource Version: 1245789 UID: 975316fa-5ee2-4520-862b-ec259c715555 Spec: Allow Tag Override: false Blob Store Configuration: Account Name: seaweedfs@seaweedfs:8333 URL Parameters: region=us-west-1 secure_connection=0 Cluster Name: test-cluster Image Type: split Pod Template Spec: Spec: Containers: Env: Name: FDB_BLOB_CREDENTIALS Value: /var/backup-credentials/credentials Name: FDB_TLS_CERTIFICATE_FILE Value: /tmp/fdb-certs/tls.crt Name: FDB_TLS_CA_FILE Value: /tmp/fdb-certs/tls.crt Name: FDB_TLS_KEY_FILE Value: /tmp/fdb-certs/tls.key Name: foundationdb Resources: Limits: Cpu: 250m Memory: 128Mi Requests: Cpu: 250m Memory: 128Mi Security Context: Run As Group: 0 Run As User: 0 Volume Mounts: Mount Path: /tmp/fdb-certs Name: fdb-certs Mount Path: /var/backup-credentials Name: backup-credentials Init Containers: Name: foundationdb-kubernetes-init Resources: Limits: Cpu: 100m Memory: 128Mi Requests: Cpu: 100m Memory: 128Mi Security Context: Run As User: 0 Volumes: Name: backup-credentials Secret: Secret Name: backup-credentials Name: fdb-certs Secret: Secret Name: fdb-kubernetes-operator-secrets Snapshot Period Seconds: 120 Version: 7.1.26 Status: Agent Count: 2 Backup Details: Deployment Configured: true Generations: Needs Backup Start: 1 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ReconciliationTerminatedEarly 2m2s (x14 over 169m) foundationdbbackup-controller signal: killed ``` deployment.yaml ```auto --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: fdb-kubernetes-operator-controller-manager control-plane: controller-manager name: fdb-kubernetes-operator-controller-manager spec: replicas: 1 selector: matchLabels: app: fdb-kubernetes-operator-controller-manager template: metadata: labels: app: fdb-kubernetes-operator-controller-manager control-plane: controller-manager spec: containers: - command: - /manager env: - name: DISABLE_SIDECAR_TLS_CHECK value: "1" - name: FDB_TLS_CERTIFICATE_FILE value: /tmp/fdb-certs/tls.crt - name: FDB_TLS_CA_FILE value: /tmp/fdb-certs/tls.crt - name: FDB_TLS_KEY_FILE value: /tmp/fdb-certs/tls.key - name: WATCH_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: foundationdb/fdb-kubernetes-operator:v2.11.0 name: manager ports: - containerPort: 8080 name: metrics resources: limits: cpu: 500m memory: 2048Mi requests: cpu: 500m memory: 2048Mi securityContext: allowPrivilegeEscalation: false privileged: false readOnlyRootFilesystem: true volumeMounts: - mountPath: /tmp/fdb-certs name: fdb-certs readOnly: true - mountPath: /tmp name: tmp - mountPath: /var/log/fdb name: logs - mountPath: /usr/bin/fdb name: fdb-binaries initContainers: - args: - --copy-library - "7.1" - --copy-binary - fdbcli - --copy-binary - fdbbackup - --copy-binary - fdbrestore - --output-dir - /var/output-files - --mode - init image: foundationdb/fdb-kubernetes-monitor:7.1.67 name: foundationdb-kubernetes-init-7-1 volumeMounts: - mountPath: /var/output-files name: fdb-binaries - args: - --copy-library - "7.3" - --copy-binary - fdbcli - --copy-binary - fdbbackup - --copy-binary - fdbrestore - --output-dir - /var/output-files - --mode - init image: foundationdb/fdb-kubernetes-monitor:7.3.63 name: foundationdb-kubernetes-init-7-3 volumeMounts: - mountPath: /var/output-files name: fdb-binaries - args: - --copy-library - "7.4" - --copy-binary - fdbcli - --copy-binary - fdbbackup - --copy-binary - fdbrestore - --output-dir - /var/output-files - --mode - init image: foundationdb/fdb-kubernetes-monitor:7.4.1 name: foundationdb-kubernetes-init-7-4 volumeMounts: - mountPath: /var/output-files name: fdb-binaries securityContext: fsGroup: 4059 runAsGroup: 4059 runAsUser: 4059 serviceAccountName: fdb-kubernetes-operator-controller-manager terminationGracePeriodSeconds: 10 volumes: - name: fdb-certs secret: secretName: fdb-kubernetes-operator-secrets - emptyDir: {} name: tmp - emptyDir: {} name: logs - emptyDir: {} name: fdb-binaries ``` backup.yaml ```auto root@v-sjc031-devops-001:/opt/FoundationDB/fdb-kubernetes-operator/config/samples# cat backup.yaml apiVersion: apps.foundationdb.org/v1beta2 kind: FoundationDBBackup metadata: name: test-cluster spec: blobStoreConfiguration: accountName: seaweedfs@seaweedfs:8333 urlParameters: - region=us-west-1 - secure_connection=0 clusterName: test-cluster imageType: split podTemplateSpec: spec: containers: - env: - name: FDB_BLOB_CREDENTIALS value: /var/backup-credentials/credentials - name: FDB_TLS_CERTIFICATE_FILE value: /tmp/fdb-certs/tls.crt - name: FDB_TLS_CA_FILE value: /tmp/fdb-certs/tls.crt - name: FDB_TLS_KEY_FILE value: /tmp/fdb-certs/tls.key name: foundationdb resources: limits: cpu: 250m memory: 128Mi requests: cpu: 250m memory: 128Mi securityContext: runAsGroup: 0 runAsUser: 0 volumeMounts: - mountPath: /tmp/fdb-certs name: fdb-certs - mountPath: /var/backup-credentials name: backup-credentials initContainers: - name: foundationdb-kubernetes-init resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi securityContext: runAsUser: 0 volumes: - name: backup-credentials secret: secretName: backup-credentials - name: fdb-certs secret: secretName: fdb-kubernetes-operator-secrets snapshotPeriodSeconds: 120 version: 7.1.26 --- apiVersion: apps.foundationdb.org/v1beta2 kind: FoundationDBCluster metadata: name: test-cluster spec: automationOptions: replacements: enabled: true faultDomain: key: foundationdb.org/none imageType: split labels: filterOnOwnerReference: false matchLabels: foundationdb.org/fdb-cluster-name: test-cluster processClassLabels: - foundationdb.org/fdb-process-class processGroupIDLabels: - foundationdb.org/fdb-process-group-id minimumUptimeSecondsForBounce: 60 processCounts: cluster_controller: 1 stateless: -1 processes: general: customParameters: - knob_disable_posix_kernel_aio=1 podTemplate: spec: containers: - name: foundationdb resources: requests: cpu: 100m memory: 128Mi securityContext: runAsUser: 0 - name: foundationdb-kubernetes-sidecar resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi securityContext: runAsUser: 0 initContainers: - name: foundationdb-kubernetes-init resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi securityContext: runAsUser: 0 volumeClaimTemplate: spec: storageClassName: local-path resources: requests: storage: 16G routing: defineDNSLocalityFields: true sidecarContainer: enableLivenessProbe: true enableReadinessProbe: false useExplicitListenAddress: true version: 7.1.26 ``` ---
**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\ **Post date:** [September 1, 2025, 3:34pm UTC](https://forums.foundationdb.org/t/foundationdbbackup-cannot-be-reconciled-properly/5023/2 "2025-09-01T15:34:16Z")
Could you share the logs for the backup start command? Seems like the operator is not able to connect to the blobstore (in you example the seaweedfs instance). Does the operator and the seaweed setup run in the same namespace?