FoundationDBBackup backup command error

Using FoundationDB Kubernetes Operator
1

The default fdbbackup command fails for the FoundationDBBackup resource.
Running the same command in a shell in a backup pod will also result in an error, unless the URL parameter is surrounded by double quotes. I couldn’t find a way to inject the double quotes using the FoundationDBBackup configuration.

apiVersion: apps.foundationdb.org/v1beta1
kind: FoundationDBBackup
metadata:
  name: storage-cluster
spec:
  version: 6.2.20
  clusterName: storage-cluster
  backupName: storage-cluster-auto
  bucket: fdb-backups&sc=0
  accountName: key@s3.us-gov-west-1.amazonaws.com:80
  snapshotPeriodSeconds: 60

Command Fails:

fdbbackup start -d blobstore://key@s3.us-gov-west-1.amazonaws.com:80/storage-cluster-auto?bucket=fdb-backups&sc=0

Command Succeeds: (added double quotes surrounding the URL param)

fdbbackup start -d "blobstore://key@s3.us-gov-west-1.amazonaws.com:80/storage-cluster-auto?bucket=fdb-backups&sc=0"

Error in the operator:

2020-09-17T19:39:10.283Z	INFO	controller	Running command	{"namespace": "default", "cluster": "storage-cluster", "path": "/usr/bin/fdb/6.2/fdbbackup", "args": ["/usr/bin/fdb/6.2/fdbbackup", "status", "--json", "-C", "/tmp/483506077", "--log", "--logdir", "/var/log/fdb"]}
2020-09-17T19:39:10.339Z	INFO	controller	Command completed	{"namespace": "default", "cluster": "storage-cluster", "output": "{\"SchemaVersion\":\"1...."}
2020-09-17T19:39:10.339Z	INFO	controller	Running command	{"namespace": "default", "cluster": "storage-cluster", "path": "/usr/bin/fdb/6.2/fdbbackup", "args": ["/usr/bin/fdb/6.2/fdbbackup", "start", "-d", "blobstore://key@s3.us-gov-west-1.amazonaws.com:80/storage-cluster-auto?bucket=fdb-backups&sc=0", "-s", "60", "-z", "-C", "/tmp/023463512", "--log", "--logdir", "/var/log/fdb"]}
2020-09-17T19:39:20.371Z	ERROR	controller	Error from FDB command	{"namespace": "default", "cluster": "storage-cluster", "code": -1, "stdout": "", "stderr": "", "error": "signal: killed"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
github.com/FoundationDB/fdb-kubernetes-operator/controllers.(*CliAdminClient).runCommand
	/workspace/controllers/admin_client.go:255
github.com/FoundationDB/fdb-kubernetes-operator/controllers.(*CliAdminClient).StartBackup
	/workspace/controllers/admin_client.go:509
github.com/FoundationDB/fdb-kubernetes-operator/controllers.StartBackup.Reconcile
	/workspace/controllers/start_backup.go:46
github.com/FoundationDB/fdb-kubernetes-operator/controllers.(*FoundationDBBackupReconciler).Reconcile
	/workspace/controllers/backup_controller.go:82
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.17.0/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.17.0/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/apimachinery@v0.17.0/pkg/util/wait/wait.go:88

Error from the fdbbackup logs:

<Event Severity="10" Time="1600371759.325817" Type="FBA_SubmitBackup" ID="0000000000000000" TagName="default" StopWhenDone="1" OutContainer="blobstore://key@s3.us-gov-west-1.amazonaws.com:80/storage-cluster-auto?bucket=fdb-backups" Machine="192.168.77.17:94" LogGroup="default" />
<Event Severity="20" Time="1600371769.349597" Type="BlobStoreEndpointRequestFailedRetryable" ID="0000000000000000" Error="timed_out" ErrorDescription="Operation timed out" ErrorCode="1004" SuppressedEventCount="0" ConnectionEstablished="0" RemoteHost="s3.us-gov-west-1.amazonaws.com" Verb="HEAD" Resource="/fdb-backups" ThisTry="1" RetryDelay="2" Machine="192.168.77.17:94" LogGroup="default" />
<Event Severity="40" Time="1600371789.329575" Type="StopAfterError" ID="0000000000000000" Error="backup_error" ErrorDescription="Backup error" ErrorCode="2300" Backtrace="addr2line -e fdbbackup.debug -p -C -f -i 0xaa654c 0xaa5e68 0xaa5f31 0x4d4ed3 0x4d50be 0x4f6339 0x4db605 0x4dbc3e 0x4f6339 0x4d03aa 0x4d0700 0x4f6339 0x699121 0x699430 0x4f6339 0x4ca230 0x4f7bb0 0xaeac50 0x83c8b2 0x4b1469 0x7f2f829afb97" Machine="192.168.77.17:94" LogGroup="default" />
2

With the way we’re executing the commands from the operator, I wouldn’t expect any kind of quotation to be necessary, because it’s not going through a shell.

3

Is there any info I could collect to help you debug this?
It seems that running from the shell works, but the operator command fails

4

Could you try running the command from a shell in the operator pod? There may be some differences in the environment between the operator and the backup agents.

5

I modified the minio instance in my dev environment to run on a non-TLS connection, and modified the cluster_with_backup example to use the bucket name fdb-backups&sc=0. It was able to start the backup and complete reconciliation. This suggests the problem is something other than the additional parameters.