S3 fdbrestore: Missing properties object

Using FoundationDB
1

Hi Folks,

I’m trying to restore a backup from S3 to an empty FDB cluster. It is failing with the below error:

image
image1192×164 7.17 KB

ERROR: HTTP response code not received or indicated failure
Fatal Error: HTTP response code not received or indicated failure

image
image1875×81 15.1 KB

I checked the Objects list via S3 Java APIs and there is no key present with prefix “data/testing/properties” in the bucket. Is there something that I am missing out?
P.S. Can see that keys “data/testing/kvranges”, “data/testing/logs” and “data/testing/snapshots” exist. The backups ran fine and exited.

BG:

Cluster setup: 1 node, 2 processes, ssd, single redundancy mode.1 backup_agent is running on the same host that the cluster is set up at. TLS not enabled.

Backup command:
fdbbackup start -w -t "checkdummy"-C fdb.cluster --log --logdir backuplog --knob_http_verbose_level=3 -d blobstore://accessId:secretKey@url.gs.com:100/testing?bucket=backup-02-03-2021\&sc=0

Restore command:
fdbrestore start -t "checkdummy"--dest_cluster_file fdb.cluster --log --logdir backuplog --knob_http_verbose_level=2 -r blobstore://accessId:secretKey@url.gs.com:100/testing?bucket=backup-02-03-2021\&sc=0

1 Like
2

@alloc @SteavedHams Any ideas?

3

What is it at url.gs.com:100 ? Regular S3? Any proxy/gateway?

4

Regular s3. This is a sample dummy url for reference!

5

@mengxu @panda @alexmiller @SteavedHams bumping this up just in case

6

@VibhutiD @rahul-nitkkr

Very sorry I did not see this thread until now!

The issue here is the http_bad_response error, NOT that the property files are missing. It is okay for the property files to be missing, they are optional. They are just helpful metadata files that are updated during backup management operations (not backup itself, but things like backup describe or expire or restore) which store pre-computed information about the backup’s log content.

Bad Response means that the HTTP client in FDB could not parse the response, so the HTTP response code is unknown.

You can use --knob_http_verbose_level=3 to see the full requests and responses, can you paste one of those?

7

Ahh okay! Also, describe and restore throw the same errors.

Command fired rn:
fdbbackup describe -C backup.cluster --log --logdir backuplogs3 --knob_http_verbose_level=3 -d blobstore://id:key@host:9020/testing?bucket=backup-16-03-2021\&sc=0

Request:

image
image1480×1013 50.2 KB

Response:
image
image1479×977 55.3 KB

image
image1565×960 57.2 KB

8

Hmm it appears that the debugging info does not dump response bytes for un-parseable responses. Since you are using sc=0 you could capture the stream some other way such as WireShark or tcpdump.

9

Hi @VibhutiD
(my apologies for posting to a dead thread)

If you have managed to get a backup and restore from AWS S3 working, could you please share your configuration and foundation version.

I have been trying to get a simple backup and restore to AWS working on Foundationdb version 6.3.9 & 6.2.30 – with no success. Any help you can provide would be highly appreciated.

This is the thread you could post your configuration.

10

any response or luck in solving the issue ? I am facing the similar issue where my faddbackup works but when i try to list or describe it does give HTTP error and i am not able to understand how I can list the available backups and use the one i need to restore.

i do see backup taken on s3 successfully using aws s3 ls so writing to s3 is not an issue but i cant list and still not clear how to use this URL while restoring specific backup.