FoundationDBBackup cannot be reconciled properly

[zmylk (LiKe)]

I suspect this is similar to the question

FoundationDBBackup does not reconcile or start backup

But after I added the certificate, it did not solve the problem

Certificate Creation Reference

fdb-kubernetes-operator/config/test-certs at release_v1.40.0 ·...

Possibly useful documents

https://github.com/FoundationDB/fdb-kubernetes-operator/blob/release_v1.40.0/docs/manual/tls.md

operator exception information

{"level":"info","ts":"2025-08-15T09:57:00Z","logger":"fdbclient","msg":"Running command","path":"/usr/bin/fdb/7.1/fdbbackup","args":["/usr/bin/fdb/7.1/fdbbackup","status","--json","-C","/tmp/99a55c37-abb6-496f-8d67-d4cdfe9531b8-cli/2559054587","--log","--logdir","/var/log/fdb"]}
{"level":"info","ts":"2025-08-15T09:57:00Z","logger":"fdbclient","msg":"Command completed","output":"{\"SchemaVersion\":\"1...."}
{"level":"info","ts":"2025-08-15T09:57:00Z","logger":"fdbclient","msg":"Running command","path":"/usr/bin/fdb/7.1/fdbbackup","args":["/usr/bin/fdb/7.1/fdbbackup","start","-d","blobstore://seaweedfs@seaweedfs:8333/test-cluster?bucket=fdb-backups&region=us-west-1&secure_connection=0","-s","120","-z","-C","/tmp/99a55c37-abb6-496f-8d67-d4cdfe9531b8-cli/390835421","--log","--logdir","/var/log/fdb"]}
{"level":"error","ts":"2025-08-15T09:57:10Z","logger":"fdbclient","msg":"Error from FDB command","code":-1,"stdout":"","stderr":"","error":"signal: killed","stacktrace":"github.com/FoundationDB/fdb-kubernetes-operator/v2/fdbclient.(*cliAdminClient).runCommand\n\t/workspace/fdbclient/admin_client.go:265\ngithub.com/FoundationDB/fdb-kubernetes-operator/v2/fdbclient.(*cliAdminClient).StartBackup\n\t/workspace/fdbclient/admin_client.go:835\ngithub.com/FoundationDB/fdb-kubernetes-operator/v2/controllers.startBackup.reconcile\n\t/workspace/controllers/start_backup.go:52\ngithub.com/FoundationDB/fdb-kubernetes-operator/v2/controllers.(*FoundationDBBackupReconciler).Reconcile\n\t/workspace/controllers/backup_controller.go:97\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202"}
{"level":"error","ts":"2025-08-15T09:57:10Z","logger":"controller","msg":"Error in reconciliation","namespace":"keli","backup":"test-cluster","reconciler":"controllers.startBackup","requeueAfter":0,"error":"signal: killed","stacktrace":"github.com/FoundationDB/fdb-kubernetes-operator/v2/controllers.processRequeue\n\t/workspace/controllers/controllers.go:92\ngithub.com/FoundationDB/fdb-kubernetes-operator/v2/controllers.(*FoundationDBBackupReconciler).Reconcile\n\t/workspace/controllers/backup_controller.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202"}
{"level":"error","ts":"2025-08-15T09:57:10Z","msg":"Reconciler error","controller":"foundationdbbackup","controllerGroup":"apps.foundationdb.org","controllerKind":"FoundationDBBackup","FoundationDBBackup":{"name":"test-cluster","namespace":"keli"},"namespace":"keli","name":"test-cluster","reconcileID":"4f6cd5ca-f30f-470f-98db-7745fb2660ef","error":"signal: killed","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202"}

root@v-sjc031-devops-001:/opt/FoundationDB/fdb-kubernetes-operator/config/samples# kubectl describe FoundationDBBackup test-cluster
Name:         test-cluster
Namespace:    keli
Labels:       <none>
Annotations:  <none>
API Version:  apps.foundationdb.org/v1beta2
Kind:         FoundationDBBackup
Metadata:
  Creation Timestamp:  2025-08-15T07:08:35Z
  Generation:          1
  Resource Version:    1245789
  UID:                 975316fa-5ee2-4520-862b-ec259c715555
Spec:
  Allow Tag Override:  false
  Blob Store Configuration:
    Account Name:  seaweedfs@seaweedfs:8333
    URL Parameters:
      region=us-west-1
      secure_connection=0
  Cluster Name:  test-cluster
  Image Type:    split
  Pod Template Spec:
    Spec:
      Containers:
        Env:
          Name:   FDB_BLOB_CREDENTIALS
          Value:  /var/backup-credentials/credentials
          Name:   FDB_TLS_CERTIFICATE_FILE
          Value:  /tmp/fdb-certs/tls.crt
          Name:   FDB_TLS_CA_FILE
          Value:  /tmp/fdb-certs/tls.crt
          Name:   FDB_TLS_KEY_FILE
          Value:  /tmp/fdb-certs/tls.key
        Name:     foundationdb
        Resources:
          Limits:
            Cpu:     250m
            Memory:  128Mi
          Requests:
            Cpu:     250m
            Memory:  128Mi
        Security Context:
          Run As Group:  0
          Run As User:   0
        Volume Mounts:
          Mount Path:  /tmp/fdb-certs
          Name:        fdb-certs
          Mount Path:  /var/backup-credentials
          Name:        backup-credentials
      Init Containers:
        Name:  foundationdb-kubernetes-init
        Resources:
          Limits:
            Cpu:     100m
            Memory:  128Mi
          Requests:
            Cpu:     100m
            Memory:  128Mi
        Security Context:
          Run As User:  0
      Volumes:
        Name:  backup-credentials
        Secret:
          Secret Name:  backup-credentials
        Name:           fdb-certs
        Secret:
          Secret Name:      fdb-kubernetes-operator-secrets
  Snapshot Period Seconds:  120
  Version:                  7.1.26
Status:
  Agent Count:  2
  Backup Details:
  Deployment Configured:  true
  Generations:
    Needs Backup Start:  1
Events:
  Type    Reason                         Age                   From                           Message
  ----    ------                         ----                  ----                           -------
  Normal  ReconciliationTerminatedEarly  2m2s (x14 over 169m)  foundationdbbackup-controller  signal: killed

deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: fdb-kubernetes-operator-controller-manager
    control-plane: controller-manager
  name: fdb-kubernetes-operator-controller-manager
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fdb-kubernetes-operator-controller-manager
  template:
    metadata:
      labels:
        app: fdb-kubernetes-operator-controller-manager
        control-plane: controller-manager
    spec:
      containers:
      - command:
        - /manager
        env:
        - name: DISABLE_SIDECAR_TLS_CHECK
          value: "1"
        - name: FDB_TLS_CERTIFICATE_FILE
          value: /tmp/fdb-certs/tls.crt
        - name: FDB_TLS_CA_FILE
          value: /tmp/fdb-certs/tls.crt
        - name: FDB_TLS_KEY_FILE
          value: /tmp/fdb-certs/tls.key
        - name: WATCH_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        image: foundationdb/fdb-kubernetes-operator:v2.11.0
        name: manager
        ports:
        - containerPort: 8080
          name: metrics
        resources:
          limits:
            cpu: 500m
            memory: 2048Mi
          requests:
            cpu: 500m
            memory: 2048Mi
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          readOnlyRootFilesystem: true
        volumeMounts:
        - mountPath: /tmp/fdb-certs
          name: fdb-certs
          readOnly: true
        - mountPath: /tmp
          name: tmp
        - mountPath: /var/log/fdb
          name: logs
        - mountPath: /usr/bin/fdb
          name: fdb-binaries
      initContainers:
      - args:
        - --copy-library
        - "7.1"
        - --copy-binary
        - fdbcli
        - --copy-binary
        - fdbbackup
        - --copy-binary
        - fdbrestore
        - --output-dir
        - /var/output-files
        - --mode
        - init
        image: foundationdb/fdb-kubernetes-monitor:7.1.67
        name: foundationdb-kubernetes-init-7-1
        volumeMounts:
        - mountPath: /var/output-files
          name: fdb-binaries
      - args:
        - --copy-library
        - "7.3"
        - --copy-binary
        - fdbcli
        - --copy-binary
        - fdbbackup
        - --copy-binary
        - fdbrestore
        - --output-dir
        - /var/output-files
        - --mode
        - init
        image: foundationdb/fdb-kubernetes-monitor:7.3.63
        name: foundationdb-kubernetes-init-7-3
        volumeMounts:
        - mountPath: /var/output-files
          name: fdb-binaries
      - args:
        - --copy-library
        - "7.4"
        - --copy-binary
        - fdbcli
        - --copy-binary
        - fdbbackup
        - --copy-binary
        - fdbrestore
        - --output-dir
        - /var/output-files
        - --mode
        - init
        image: foundationdb/fdb-kubernetes-monitor:7.4.1
        name: foundationdb-kubernetes-init-7-4
        volumeMounts:
        - mountPath: /var/output-files
          name: fdb-binaries
      securityContext:
        fsGroup: 4059
        runAsGroup: 4059
        runAsUser: 4059
      serviceAccountName: fdb-kubernetes-operator-controller-manager
      terminationGracePeriodSeconds: 10
      volumes:
      - name: fdb-certs
        secret:
          secretName: fdb-kubernetes-operator-secrets
      - emptyDir: {}
        name: tmp
      - emptyDir: {}
        name: logs
      - emptyDir: {}
        name: fdb-binaries

backup.yaml

root@v-sjc031-devops-001:/opt/FoundationDB/fdb-kubernetes-operator/config/samples# cat backup.yaml
apiVersion: apps.foundationdb.org/v1beta2
kind: FoundationDBBackup
metadata:
  name: test-cluster
spec:
  blobStoreConfiguration:
    accountName: seaweedfs@seaweedfs:8333
    urlParameters:
    - region=us-west-1
    - secure_connection=0
  clusterName: test-cluster
  imageType: split
  podTemplateSpec:
    spec:
      containers:
      - env:
        - name: FDB_BLOB_CREDENTIALS
          value: /var/backup-credentials/credentials
        - name: FDB_TLS_CERTIFICATE_FILE
          value: /tmp/fdb-certs/tls.crt
        - name: FDB_TLS_CA_FILE
          value: /tmp/fdb-certs/tls.crt
        - name: FDB_TLS_KEY_FILE
          value: /tmp/fdb-certs/tls.key
        name: foundationdb
        resources:
          limits:
            cpu: 250m
            memory: 128Mi
          requests:
            cpu: 250m
            memory: 128Mi
        securityContext:
          runAsGroup: 0
          runAsUser: 0
        volumeMounts:
        - mountPath: /tmp/fdb-certs
          name: fdb-certs
        - mountPath: /var/backup-credentials
          name: backup-credentials
      initContainers:
      - name: foundationdb-kubernetes-init
        resources:
          limits:
            cpu: 100m
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 128Mi
        securityContext:
          runAsUser: 0
      volumes:
      - name: backup-credentials
        secret:
          secretName: backup-credentials
      - name: fdb-certs
        secret:
          secretName: fdb-kubernetes-operator-secrets
  snapshotPeriodSeconds: 120
  version: 7.1.26
---
apiVersion: apps.foundationdb.org/v1beta2
kind: FoundationDBCluster
metadata:
  name: test-cluster
spec:
  automationOptions:
    replacements:
      enabled: true
  faultDomain:
    key: foundationdb.org/none
  imageType: split
  labels:
    filterOnOwnerReference: false
    matchLabels:
      foundationdb.org/fdb-cluster-name: test-cluster
    processClassLabels:
    - foundationdb.org/fdb-process-class
    processGroupIDLabels:
    - foundationdb.org/fdb-process-group-id
  minimumUptimeSecondsForBounce: 60
  processCounts:
    cluster_controller: 1
    stateless: -1
  processes:
    general:
      customParameters:
      - knob_disable_posix_kernel_aio=1
      podTemplate:
        spec:
          containers:
          - name: foundationdb
            resources:
              requests:
                cpu: 100m
                memory: 128Mi
            securityContext:
              runAsUser: 0
          - name: foundationdb-kubernetes-sidecar
            resources:
              limits:
                cpu: 100m
                memory: 128Mi
              requests:
                cpu: 100m
                memory: 128Mi
            securityContext:
              runAsUser: 0
          initContainers:
          - name: foundationdb-kubernetes-init
            resources:
              limits:
                cpu: 100m
                memory: 128Mi
              requests:
                cpu: 100m
                memory: 128Mi
            securityContext:
              runAsUser: 0
      volumeClaimTemplate:
        spec:
          storageClassName: local-path
          resources:
            requests:
              storage: 16G
  routing:
    defineDNSLocalityFields: true
  sidecarContainer:
    enableLivenessProbe: true
    enableReadinessProbe: false
  useExplicitListenAddress: true
  version: 7.1.26

[johscheuer (Johannes Scheuermann)]

Could you share the logs for the backup start command? Seems like the operator is not able to connect to the blobstore (in you example the seaweedfs instance). Does the operator and the seaweed setup run in the same namespace?