FoundationDb fdbbackup to Azure Blob Store

Using FoundationDB
#1

Hi All,
I am using FoundationDb 7.1 (v7.1.9) and trying to setup for direct backup to azure blob store. I am using following command to run the backup
fdbbackup start -C /usr/local/etc/foundationdb/fdb.cluster --log -d ‘blobstore://fdbbackuptest:<secret_key>@fdbbackuptest.blob.core.windows.net/fdbtest?sc=0’ -t test3

and I get
ERROR: Could not create backup container: HTTP response code not received or indicated failure
ERROR: An error was encountered during submission
Fatal Error: Backup error

The log file shows

Any pointer on what could be wrong or what I am missing? Does FDB7.1 provide native support to Azure blobstore?

#2

I think currently only AWS S3 is well supported, there are some issues in the GitHub repository and one last piece to make fdbbackup work with other S3 compatible is this PR: Fix compatibility issue of s3 backup by wangzw · Pull Request #6348 · apple/foundationdb · GitHub (not merged). I’m not to familiar with Azure Blobstore but assuming it’s S3 compatible once the mentioned PR is merged into the 7.1 branch backups to Azure Blobstore should work.

edit you might be able to work around this limitation by using minio Using s3cmd and other Amazon S3-compatible apps with Azure Blob Storage - Microsoft Open Source Blog in this case you should use signature_v2 for minio and set --knob_http_request_aws_v4_header=false on the fddbbackup side to use the old signature version (see: foundationdb/backups.rst at main · apple/foundationdb · GitHub)

#3

Johannes,

From what I have seen, the open-source minio has stopped supporting gateway many years ago. The Azure “marketplace” product of minio was the only option.

In addition, they are also deprecating the paid version (except for continuing to support existing customers)

Thanks

Hari

#4

Minio Gateway is not working either. We use it to connect to the Azure and Google blob storage and they are both failing with the same error. I have increased the log verbosity and here is what we get:

[a35d88f5ac2cac5c41b3ddf617c7c74c] HTTP starting HEAD /foundationdb-backup ContentLen:0
Request Header: Accept: application/xml
Request Header: Authorization: AWS4-HMAC-SHA256 Credential=user/20221107/backup/minio-gateway/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=b7cf9069ddeab032f0d800d2eb1fc75cf94a69f368ed7fc17806b5b81573b1f7
Request Header: Content-Length: 0
Request Header: Host: minio-gateway.backup.svc
Request Header: x-amz-content-sha256: UNSIGNED-PAYLOAD
Request Header: x-amz-date: 20221107T192708Z
[a35d88f5ac2cac5c41b3ddf617c7c74c] HTTP code=400 early=0, time=0.002093s HEAD /foundationdb-backup contentLen=0 [423 out, response content len 0]
[a35d88f5ac2cac5c41b3ddf617c7c74c] HTTP RESPONSE:  HEAD /foundationdb-backup
Response Code: 400
Response ContentLen: 0
Reponse Header: Accept-Ranges: bytes
Reponse Header: Content-Length: 0
Reponse Header: Content-Security-Policy: block-all-mixed-content
Reponse Header: Date: Mon, 07 Nov 2022 19:27:08 GMT
Reponse Header: Server: MinIO
Reponse Header: Vary: Origin
Reponse Header: X-Amz-Request-Id: 172564095FEDD0A7
Reponse Header: X-Xss-Protection: 1; mode=block
-- RESPONSE CONTENT--

For some reason there is an UNSIGNED-PAYLOAD.

#5

I confirm that after enabling the knob to use v2 header. The backup submission succeeds using MinIO gateway.

The backup on tag `debug-manual-backup-01' was successfully submitted.
#6

Actually it is not working. Looks like we just moved the problem elsewhere:

$ kubectl exec -it foundationdb-cluster-backup-agent-783f487c58-4mq5a -c foundationdb --context azure-dev -ntimeseries -- fdbbackup status -t debug-manual-backup-01 -C /var/dynamic-conf/cluster-file
The previous backup on tag `debug-manual-backup-01' at blobstore://user:password@minio-gateway.backup.svc:9000/automatic_backups/debug-manual-backup-01?bucket=foundationdb-backup&sc=0 has been aborted.
BackupUID: d3a931ea363396b65ff2d11d07812341
BackupURL: blobstore://user:password@minio-gateway.backup.svc:9000/automatic_backups/debug-manual-backup-01?bucket=foundationdb-backup&sc=0
Recent Errors (since initialization)
5 second(s) ago : 'HTTP response code not received or indicated failure' on 'file_backup_write_range_5.2'
#7

We ended up using s3proxy. You may want to try that proxy instead.

If the gateway product is opensource/self hosted, try running it in the same node as backup nodes so you can bypass the authentication need between fdbbackup ← > proxy.

#8

Thanks for the recommendation. Migrating to s3proxy is probably the best way to go indeed.
However I have managed to make minIO work with aws v2 headers.
The problem was that the knob on the fdbbackup command does not propagate to the backup agents. By setting the knob in the agents manifest we successfully resumed the backups.