`fdbbackup` with TLS not working

Using FoundationDB Kubernetes Operator
1

Hi all,
I setup FDB cluster by FDBCluster operator with TLS (self-signed certificate by cert-manager). Then I setup FDBBackup that use same TLS certificate. After backup deployment started, only backup_agent process running inside backup pod, fdbbackup process is not (1). Then I run fdbbackup start manual, if I use parameter sc=0 (disable TLS) the backup work, else sc=1 (enable TLS) it not (2).

I have 2 point unclear. First is as FDB backup document, after setup FDB backup by operator, it will start 2 process are backup_agent and fdbbackup start, in my case only backup_agent. Second is TLS is used and worked for communicate inside cluster only, not work for talk to S3.
Here is my config for FDBBackup operator

apiVersion: apps.foundationdb.org/v1beta2
kind: FoundationDBBackup
metadata:
  name: fdb-cluster
spec:
  blobStoreConfiguration:
    accountName: <access_key_id>@s3.us-east-2.amazonaws.com
    backupName: daily-backup
    bucket: ci-cd-code-deploy
  clusterName: fdb-cluster
  customParameters:
    - "knob_http_request_aws_v4_header=true"
    - "knob_http_verbose_level=4"
  podTemplateSpec:
    spec:
      containers:
        - env:
          - name: FDB_BLOB_CREDENTIALS
            value: /var/backup-credentials/credentials
          - name: FDB_TLS_CERTIFICATE_FILE
            value: /var/fdb-certs/tls.crt
          - name: FDB_TLS_CA_FILE
            value: /var/trust-ca/ca.crt
          - name: FDB_TLS_KEY_FILE
            value: /var/fdb-certs/tls.key
          name: foundationdb
          resources:
            limits:
              cpu: 250m
              memory: 128Mi
            requests:
              cpu: 250m
              memory: 128Mi
          securityContext:
            runAsGroup: 0
            runAsUser: 0
          volumeMounts:
            - mountPath: /var/fdb-certs
              name: fdb-certs
            - mountPath: /var/trust-ca
              name: trust-ca
            - mountPath: /var/backup-credentials
              name: backup-credentials
      initContainers:
        - name: foundationdb-kubernetes-init
          resources:
            limits:
              cpu: 100m
              memory: 128Mi
            requests:
              cpu: 100m
              memory: 128Mi
          securityContext:
            runAsUser: 0
      volumes:
        - name: backup-credentials
          secret:
            secretName: aws-credentials
        - name: fdb-certs
          secret:
            secretName: fdb-cluster-certificate
        - name: trust-ca
          secret:
            secretName: root-secret
  snapshotPeriodSeconds: 60
  version: 6.3.23
2

For your questions:

1.) The fdbbackup start command will be triggered by the operator itself and the backup_agent(s) will then to the actual backup work.
2.) TLS is used for communicating the the cluster and S3 and you have to ensure that the provided CA file has all required root CAs (the one for your cluster and the root CA for S3).

When you take a look at the trace files in the operator you’ll probably see some TLS verification issues they are located by default under /var/log/fdb.

3

Hi @johscheuer ,

  1. My understand is right ( fdbbackup start command will be triggered by the operator but it doesn’t work in my case)
  2. I read the log, it responses
<Event Severity="20" Time="1652766871.043128" DateTime="2022-05-17T05:54:31Z" Type="TLSPolicyFailure" ID="0000000000000000" SuppressedEventCount="0" Reason="preverification failed" VerifyError="unable to get local issuer certificate" ThreadID="13788610134641559662" Machine="10.42.1.46:143" LogGroup="default" />
<Event Severity="20" Time="1652766871.043128" DateTime="2022-05-17T05:54:31Z" Type="N2_ConnectHandshakeError" ID="6fa4c5f1a20071f6" SuppressedEventCount="0" ErrorCode="337047686" Message="certificate verify failed" WhichMeans="error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed" ThreadID="13788610134641559662" Machine="10.42.1.46:143" LogGroup="default" />

My question is can I use self signed certificate for this? And if yes, how can I create certificate that sufficient as your answer

TLS is used for communicating the the cluster and S3 and you have to ensure that the provided CA file has all required root CAs (the one for your cluster and the root CA for S3)
4

Is your cluster running with the same certificates? The following docs have more details about the TLS requirements: Transport Layer Security — FoundationDB 7.1. Yes you can do that, you just have to ensure that the CA file contains the root CA for your self-singed certificate and the root CA’s for S3.

5

Hi @johscheuer , I use same certificates (self signed) for cluster and use that CA for TLS communicate with S3 also. So now I need to create new certificate for S3? And root CA for S3 must be signed by third party like letencrypt or self signed is acceptable?

6

You don’t need to create a new certificate for S3 Amazon already have their own certificates, you just have to download the root CA for S3 from Amazon, see: Amazon Trust Services Repository

7

Hi @johscheuer , after download root CA from Amazon, I need to append Amazon root CA to my CA, like:

----------BEGIN---------
Here is my CA
----------END------------
----------BEGIN---------
Here is Amazon CA
----------END------------