S3 fdbrestore: Missing properties object

Hi Folks,

I’m trying to restore a backup from S3 to an empty FDB cluster. It is failing with the below error:

ERROR: HTTP response code not received or indicated failure
Fatal Error: HTTP response code not received or indicated failure

I checked the Objects list via S3 Java APIs and there is no key present with prefix “data/testing/properties” in the bucket. Is there something that I am missing out?
P.S. Can see that keys “data/testing/kvranges”, “data/testing/logs” and “data/testing/snapshots” exist. The backups ran fine and exited.

BG:

Cluster setup: 1 node, 2 processes, ssd, single redundancy mode.1 backup_agent is running on the same host that the cluster is set up at. TLS not enabled.

Backup command:
fdbbackup start -w -t "checkdummy"-C fdb.cluster --log --logdir backuplog --knob_http_verbose_level=3 -d blobstore://accessId:secretKey@url.gs.com:100/testing?bucket=backup-02-03-2021\&sc=0

Restore command:
fdbrestore start -t "checkdummy"--dest_cluster_file fdb.cluster --log --logdir backuplog --knob_http_verbose_level=2 -r blobstore://accessId:secretKey@url.gs.com:100/testing?bucket=backup-02-03-2021\&sc=0

1 Like

@alloc @SteavedHams Any ideas?

What is it at url.gs.com:100 ? Regular S3? Any proxy/gateway?

Regular s3. This is a sample dummy url for reference!

@mengxu @panda @alexmiller @SteavedHams bumping this up just in case

@VibhutiD @rahul-nitkkr

Very sorry I did not see this thread until now!

The issue here is the http_bad_response error, NOT that the property files are missing. It is okay for the property files to be missing, they are optional. They are just helpful metadata files that are updated during backup management operations (not backup itself, but things like backup describe or expire or restore) which store pre-computed information about the backup’s log content.

Bad Response means that the HTTP client in FDB could not parse the response, so the HTTP response code is unknown.

You can use --knob_http_verbose_level=3 to see the full requests and responses, can you paste one of those?

Ahh okay! Also, describe and restore throw the same errors.

Command fired rn:
fdbbackup describe -C backup.cluster --log --logdir backuplogs3 --knob_http_verbose_level=3 -d blobstore://id:key@host:9020/testing?bucket=backup-16-03-2021\&sc=0

Request:


Response:

Hmm it appears that the debugging info does not dump response bytes for un-parseable responses. Since you are using sc=0 you could capture the stream some other way such as WireShark or tcpdump.

Hi @VibhutiD
(my apologies for posting to a dead thread)

If you have managed to get a backup and restore from AWS S3 working, could you please share your configuration and foundation version.

I have been trying to get a simple backup and restore to AWS working on Foundationdb version 6.3.9 & 6.2.30 – with no success. Any help you can provide would be highly appreciated.

This is the thread you could post your configuration.