FoundationDB fdbbackup

Using FoundationDB
1

Hello,

I am new to foundationdb. I am trying to run the fdbbackup command line tool to back up to aws S3.

I am using the following command:

fdbbackup start -d blobstore://<api_key>:@s3.amazonaws.com:443/mytestbackup?bucket=<aws_s3_bucket_name>.

I get the following error:
ERROR: Could not create backup container: Operation timed out

ERROR: An error was encountered during submission.

I have a few of questions:

  1. Is the blobstore URL I have provided have the correct URL format?

  2. I am not sure what the error is and where can I lookup any specific logs pertaining to the error. I lookuped up under /etc/foundationdb/logs - the path specified in the foundationdb.conf but could not figure out.

  3. We are using IAM roll based buckets in S3 so as long as the fdb instance has the right bucket policy we don’t need to specify the <api_key>: in the command above. Is there a way around that?

Thanks!

2

First, try adding --log, like fdbbackup start --log -d blobstore://<api_key>:@s3.amazonaws.com:443/mytestbackup?bucket=<aws_s3_bucket_name>. You can read through the .xml file to find out exactly what’s going wrong.

In our case, there was a TLS issue and so we prefixed fdbbackup and backup_agent with FDB_TLS_VERIFY_PEERS=Check.Valid=0 FDB_TLS_CERTIFICATE_FILE="/etc/foundationdb/fdb.pem" FDB_TLS_KEY_FILE="/etc/foundationdb/private.key" FDB_TLS_CA_FILE="/etc/foundationdb/cert.crt". More information on that here.

1 Like
3

@surprisetalk Thanks for the information. Unfortunately --log doesn’t publish any more details for debugging the backup failure. Do I need to enable something else for it to publish details of the backup?

I double checked my credentials by using the same to access the backup S3 bucket throught a boto3 client running on the machine and it is able to access the bucket.

4

@surprisetalk @SteavedHams I was looking for the log file in the wrong place ie. in the folder for the foundationdb logs. The log file was generated in the folder where I was running the fdbbackup command.

The below is the command I used after setting the FDB_TLS environment variables you mentioned.
fdbbackup start --log -d blobstore://$myKey:$mySecret@s3.amazonaws.com/test?bucket=fdbtest

Here are some of the logs I found.

Event Severity=“10” Time=“1570565864.614500” Type=“BlobStoreEndpointNewConnection” ID=“0000000000000000” SuppressedEventCount=“0” RemoteEndpoint=“52.216.146.5:443:tls” ExpiresIn=“120” Machine=“xx.xx.44.179:20242” LogGroup=“default”

Event Severity=“10” Time=“1570565864.634025” Type=“TLSConnectionHandshakeSuccessful” ID=“074da8bfda998311” SuppressedEventCount=“0” Peer=“52.216.146.5:443:tls” Machine=“xx.xx.44.179:20242” LogGroup=“default”

Event Severity=“40” Time=“1570565864.637650” Type=“StopAfterError” ID=“0000000000000000” Error=“backup_error” ErrorDescription=“Backup error” ErrorCode=“2300” Backtrace=“addr2line -e fdbbackup.debug -p -C -f -i 0x9f0136 0x9ee88f 0x44a067 0x44a367 0x476969 0x45255c 0x452d87 0x476969 0x440b5f 0x4412e4 0x476969 0x5f8f2a 0x5f9284 0x476969 0x43c4c2 0x476969 0x4e1a4f 0x476969 0x5118ec 0x52cce9 0x5117bc 0x52d809 0x51db56 0x51e0d0 0x51e52f 0x5143b8 0x6bc470 0x6be6aa 0x478fd0 0x6bb414 0x6bb883 0x6bbbb7 0x6b9f28 0x6ba655 0x478fd0 0x43e2ab 0x6b9870 0x6ba7f3 0x478f40 0xa19e8a 0x74046e 0x423c15 0x7fbeefa07505” Machine=“xx.xx.44.179:20242” LogGroup=“default”

Event Severity=“20” Time=“1570565864.637650” Type=“BlobStoreEndpointRequestFailed” ID=“0000000000000000” SuppressedEventCount=“0” ResponseCode=“403” ConnectionEstablished=“1” RemoteEndpoint=“52.216.146.5:443:tls” Verb=“HEAD” Resource=“/fdbtest” ThisTry=“1” Machine=“xx.xx.44.179:20242” LogGroup=“default”

Event Severity=“10” Time=“1570565864.637650” Type=“MachineLoadDetail” ID=“0000000000000000” User=“191970” Nice=“4904” System=“186544” Idle=“36388023” IOWait=“5685” IRQ=“0” SoftIRQ=“3260” Steal=“707” Guest=“0” Machine=“xx.xx.44.179:20242” LogGroup=“default”

I do see that the HEAD request for the fdbtest s3 bucket failed with 403 but using the same credentials on the same instance I can get the bucket via boto3.

Thanks for your help in advance!

5

Try
fdbbackup start --log -d blobstore://$myKey:$mySecret@s3.amazonaws.com/test?bucket=fdbtest --knob_http_verbose_level=3

The HTTP response content from S3 for the failed request might give further details, and this knob will dump HTTP requests and responses to standard output.

6

Thanks @SteavedHams, tried that but didn’t get much information other than its a 403.

fdbbackup start --log -d blobstore://$myKey:$mySecret@s3.amazonaws.com/cptest?bucket=fdbtest --knob_http_verbose_level=3

[16b4b3e34cd7d328f9f6b0436de1c07c] HTTP starting HEAD /fdbtest ContentLen:0

Request Header: Accept: application/xml

Request Header: Authorization: AWS access_key:2wHDZCZ5tqiH6hpxju3GNkkrd18=

Request Header: Content-Length: 0

Request Header: Date: 20191008T235823Z

Request Header: Host: s3.amazonaws.com

[16b4b3e34cd7d328f9f6b0436de1c07c] HTTP code=403 early=0, time=0.029062s HEAD /fdbtest contentLen=0 [188 out, response content len -1]

[16b4b3e34cd7d328f9f6b0436de1c07c] HTTP RESPONSE: HEAD /fdbtest

Response Code: 403

Response ContentLen: -1

Reponse Header: Content-Type: application/xml

Reponse Header: Date: Tue, 08 Oct 2019 23:58:22 GMT

Reponse Header: Server: AmazonS3

Reponse Header: Transfer-Encoding: chunked

Reponse Header: x-amz-bucket-region: us-east-1

Reponse Header: x-amz-id-2: OSs5ovyScyjLiNIAuq6Vxzjn5ue0L7tuhxV3r2vOtSH9s6bzVeGfbRIFc7tpFJmefZygZJqrXS4=

Reponse Header: x-amz-request-id: 91F636A9187543D5

– RESPONSE CONTENT–

ERROR: Could not create backup container: HTTP response code not received or indicated failure

ERROR: An error was encountered during submission

Fatal Error: Backup error

7

Can you use the x-amz-request-id to get more information from S3?

I can tell you that it’s not an auth failure, that would return a different error. It’s saying that you are forbidden to access the URL you are trying to access, which in this case is a bucket. Could this access key be lacking permissions for listing buckets, or this specific bucket?