Fdbbackup not work with ipv6 resolver, work with ipv4

Using FoundationDB
1

Hi all,
I got issue but I don’t know how to fix it.
I create fdbbackup task and got error. It comes from ipv4 and ipv6.
When I use network with resolve domain like: s3.us-east-2.amazonaws.com to two type of ipv4 and ipv6, backup container cannot connect to blob endpoint.

root@a1d0acf38b0d:/var/fdb# host s3.us-east-2.amazonaws.com
s3.us-east-2.amazonaws.com has address 52.219.98.217
s3.us-east-2.amazonaws.com has IPv6 address 64:ff9b::34db:62d9
root@a1d0acf38b0d:/var/fdb# fdbbackup start -d "blobstore://<access_key>:<secret_key>@s3.us-east-2.amazonaws.com/fdb-service?bucket=aa-tmp-1&sc=0" --knob_http_verbose_level=4 --knob_http_request_aws_v4_header=true

# Debug log to get IP type
DEBUG: Host: s3.us-east-2.amazonaws.com, Service: 
DEBUG: Connect and resolve IP
DEBUG: IPAddress: 64:ff9b::34db:6e39

ERROR: Could not create backup container: Operation timed out
ERROR: An error was encountered during submission
Fatal Error: Backup error

Then I use another network with ipv4 only, it work

root@a1d0acf38b0d:/var/fdb# host s3.us-east-2.amazonaws.com
s3.us-east-2.amazonaws.com has address 52.219.107.73
root@a1d0acf38b0d:/var/fdb# ./fdbbackup start -d "blobstore://<access_key>:<secret_key>@s3.us-east-2.amazonaws.com/fdb-service?bucket=aa-tmp-1&sc=0" --knob_http_verbose_level=4 --knob_http_request_aws_v4_header=true

DEBUG: Host: s3.us-east-2.amazonaws.com, Service: 
DEBUG: Connect and resolve IP
DEBUG: IPAddress: 52.219.107.73

[2d75f8d81bb7c2cf053fae81f7dee075] HTTP starting HEAD /aa-tmp-1 ContentLen:0
Request Header: Accept: application/xml
Request Header: Authorization: AWS4-HMAC-SHA256 Credential=<access_key>/20221215/us-east-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a7dcfb4d459614618ecf41c5e5462e802d3f4b5615ab66cdcd4892c5ecf7a24c
Request Header: Content-Length: 0
Request Header: Host: s3.us-east-2.amazonaws.com
Request Header: x-amz-content-sha256: UNSIGNED-PAYLOAD
Request Header: x-amz-date: 20221215T145201Z
[2d75f8d81bb7c2cf053fae81f7dee075] HTTP code=200 early=0, time=0.295300s HEAD /aa-tmp-1 contentLen=0 [403 out, response content len -1]
[2d75f8d81bb7c2cf053fae81f7dee075] HTTP RESPONSE:  HEAD /aa-tmp-1
Response Code: 200
Response ContentLen: -1
Reponse Header: Content-Type: application/xml
Reponse Header: Date: Thu, 15 Dec 2022 14:52:02 GMT
Reponse Header: Server: AmazonS3
Reponse Header: x-amz-access-point-alias: false
Reponse Header: x-amz-bucket-region: us-east-2
Reponse Header: x-amz-id-2: ZbZSnIoFIdx74I9IL2A159e26wnlSVD3oD8tE6vtx2b2d1QgX+G8FWEOXV8VvkokCeCUAZ/7iSA=
Reponse Header: x-amz-request-id: Y270XM0WTB69J8PT
-- RESPONSE CONTENT--

--------

[2d75f8d81bb7c2cf053fae81f7dee075] HTTP starting HEAD /aa-tmp-1/backups/fdb-service ContentLen:0
Request Header: Accept: application/xml
Request Header: Authorization: AWS4-HMAC-SHA256 Credential=<access_key>/20221215/us-east-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f312e1938eb3d412dcdbf1e9fff75b76640fe860946c989980909e433c844713
Request Header: Content-Length: 0
Request Header: Host: s3.us-east-2.amazonaws.com
Request Header: x-amz-content-sha256: UNSIGNED-PAYLOAD
Request Header: x-amz-date: 20221215T145201Z
[2d75f8d81bb7c2cf053fae81f7dee075] HTTP code=200 early=0, time=0.286125s HEAD /aa-tmp-1/backups/fdb-service contentLen=0 [423 out, response content len 0]
[2d75f8d81bb7c2cf053fae81f7dee075] HTTP RESPONSE:  HEAD /aa-tmp-1/backups/fdb-service
Response Code: 200
Response ContentLen: 0
Reponse Header: Accept-Ranges: bytes
Reponse Header: Content-Length: 0
Reponse Header: Content-Type: binary/octet-stream
Reponse Header: Date: Thu, 15 Dec 2022 14:52:03 GMT
Reponse Header: ETag: "d41d8cd98f00b204e9800998ecf8427e"
Reponse Header: Last-Modified: Thu, 15 Dec 2022 14:18:46 GMT
Reponse Header: Server: AmazonS3
Reponse Header: x-amz-id-2: KIh66yL7cpa2whYFocvZ3icWQvvykCDtSIYFqrOsAUh9h+5ElzuBDOUzGSdH/+Z2uw49Gwj9N9k=
Reponse Header: x-amz-request-id: ATRKRA9TAN620SJ5
-- RESPONSE CONTENT--

--------

The backup on tag `default' was successfully submitted but no backup agents are responding.
2

This looks like a bug to me. We can file a github issue to track this.

3

Hi @jzhou
After several tests, I realize that it is not bug of FDB.
I deploy FDB on docker and as default, docker run on IP v4 only, not support IP v6.
So I add ipv6 and ip6tables support for docker, it works well.

For any deploy FDB on docker, you can use these configs to make it work for Ip v6
/etc/docker/daemon.json

{
  "ipv6": true,
  "fixed-cidr-v6": "2001:db8:1::/64",
  "ip6tables": true,
  "experimental": true
}
1 Like