Backup to GCS failing TLS preverification

Using FoundationDB
1

We are trying to run backups directly to Google Cloud Storage (GCS) from our FDB cluster. We are running on 7.1.39, which should have all the necessary compatibility fixes, but it is still failing TLS verification, complaining that the cert for storage.googleapis.com is self-signed.

We have tried running with Check.Valid=0, and then the backup runs successfully, so we are certain TLS is the only issue. The cluster we are backing up from is not running with TLS internally. We have tried downloading root certs from Google, concatenating it in a file and pointing to it with FDB_TLS_CA_FILE, but that has not helped. We have also tried creating a certificate for FDB, and populating FDB_TLS_KEY_FILE,FDB_TLS_CA_FILE,FDB_TLS_CERTIFICATE_FILE, but that did not help (even when combining FDB CA with Google CA in CA file).

Examining the cert from storage.googleapis.com with openssl, I am fairly certain that it is not self-signed, and that FDB is mistaken for some reason.

Any help would be appreciated.

<Event Severity="20" Time="1695126371.137127" DateTime="2023-09-19T12:26:11Z" Type="TLSPolicyFailure" ID="0000000000000000" SuppressedEventCount="0" Reason="preverification failed" VerifyError="self signed certificate" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126371.137127" DateTime="2023-09-19T12:26:11Z" Type="N2_ConnectHandshakeError" ID="4ccee7736a127f57" SuppressedEventCount="0" ErrorCode="337047686" Message="certificate verify failed (SSL routines, tls_process_server_certificate)" WhichMeans="error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126371.137127" DateTime="2023-09-19T12:26:11Z" Type="S3BlobStoreEndpointRequestFailedRetryable" ID="0000000000000000" Error="connection_failed" ErrorDescription="Network connection failed" ErrorCode="1026" SuppressedEventCount="0" ConnectionEstablished="0" RemoteHost="storage.googleapis.com" Verb="HEAD" Resource="/cognitedata-greenfield-foundationdb-backup" ThisTry="1" RetryDelay="2" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126373.147324" DateTime="2023-09-19T12:26:13Z" Type="TLSPolicyFailure" ID="0000000000000000" SuppressedEventCount="0" Reason="preverification failed" VerifyError="self signed certificate" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126373.147324" DateTime="2023-09-19T12:26:13Z" Type="N2_ConnectHandshakeError" ID="a2ede36bc2b4133f" SuppressedEventCount="0" ErrorCode="337047686" Message="certificate verify failed (SSL routines, tls_process_server_certificate)" WhichMeans="error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126377.157543" DateTime="2023-09-19T12:26:17Z" Type="TLSPolicyFailure" ID="0000000000000000" SuppressedEventCount="0" Reason="preverification failed" VerifyError="self signed certificate" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126377.157543" DateTime="2023-09-19T12:26:17Z" Type="N2_ConnectHandshakeError" ID="92f8fc9ca4e98b67" SuppressedEventCount="0" ErrorCode="337047686" Message="certificate verify failed (SSL routines, tls_process_server_certificate)" WhichMeans="error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126385.175072" DateTime="2023-09-19T12:26:25Z" Type="TLSPolicyFailure" ID="0000000000000000" SuppressedEventCount="0" Reason="preverification failed" VerifyError="self signed certificate" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126385.175072" DateTime="2023-09-19T12:26:25Z" Type="N2_ConnectHandshakeError" ID="1e37e11865b22d65" SuppressedEventCount="0" ErrorCode="337047686" Message="certificate verify failed (SSL routines, tls_process_server_certificate)" WhichMeans="error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="40" ErrorKind="Unset" Time="1695126401.103105" DateTime="2023-09-19T12:26:41Z" Type="StopAfterError" ID="0000000000000000" Error="backup_error" ErrorDescription="Backup error" ErrorCode="2300" ThreadID="17885686041784278561" Backtrace="addr2line -e fdbbackup.debug -p -C -f -i 0x113f7c8 0x113dffb 0x113e201 0x619b21 0x619d59 0x656a70 0x635d25 0x6364cc 0x656a70 0x64380f 0x6441ab 0x656a70 0x7fc3b1 0x8696cc 0x656a70 0x60ea6f 0x10c69a8 0xafe832 0x5f2f4a 0x7ff0fb88b555" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
2

We have tried downloading root certs from Google, concatenating it in a file and pointing to it with FDB_TLS_CA_FILE , but that has not helped.

Have you tried to connect with openssl and the new CA file to GCS? FDB requires the full CA cert chain when connecting to a TLS service. If you are able to connect to the storage API with curl and the default certificate chains on your host/container, you could try to point FDB_TLS_CA_FILE to the chain (given that your cluster is running without TLS).

3

Steps taken to verify:

echo | openssl s_client -showcerts -servername "storage.googleapis.com" -connect "storage.googleapis.com:443" 2>/dev/null

Store the direct certificate in cert.crt and the rest of the chain in ca-chain.crt.
Run verification with openssl

openssl verify -CAfile ./ca-chain.crt ./cert.pem

This returned an error:

C = US, O = Google Trust Services LLC, CN = GTS Root R1
error 2 at 2 depth lookup: unable to get issuer certificate
error ./cert.pem: verification failed

After some digging, I found that the ca file was missing a root cert from GlobalSign. After adding it, I got

./cert.pem: OK

Rerunning fdbbackup with --tls_ca_file ca-chain.crt, I got the same verification error.

4

So, in short, FDB fails verification even when the full chain can be verified by openssl.

5

So, in short, FDB fails verification even when the full chain can be verified by openssl.

Are you able to share your full chain for GCS? It might be that openssl itself is more tolerant than FDB. Have you tried to let FDB use the default host CA chain?

6

cert.crt:

-----BEGIN CERTIFICATE-----
MIIEljCCA36gAwIBAgIQKe5Vu9vzeskSF/rMvrgUIDANBgkqhkiG9w0BAQsFADBG
MQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExM
QzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMzA4MTQwODI1MDlaFw0yMzExMDYw
ODI1MDhaMCExHzAdBgNVBAMTFnN0b3JhZ2UuZ29vZ2xlYXBpcy5jb20wWTATBgcq
hkjOPQIBBggqhkjOPQMBBwNCAARRhrnR3yF4RIO44LQ55HTVd9qlNvfHSYx6JoEs
XjLPw6eiD7i/fKMcNDhF51Qi47206lvN2w3xGG4iANI15AWJo4ICbjCCAmowDgYD
VR0PAQH/BAQDAgeAMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAw
HQYDVR0OBBYEFPQeKzSKEfsAnfwJP6cKlSZ5Od8gMB8GA1UdIwQYMBaAFIp0f6+F
ze6VzT2c0OJGFPNxNR0nMGoGCCsGAQUFBwEBBF4wXDAnBggrBgEFBQcwAYYbaHR0
cDovL29jc3AucGtpLmdvb2cvZ3RzMWMzMDEGCCsGAQUFBzAChiVodHRwOi8vcGtp
Lmdvb2cvcmVwby9jZXJ0cy9ndHMxYzMuZGVyMCEGA1UdEQQaMBiCFnN0b3JhZ2Uu
Z29vZ2xlYXBpcy5jb20wIQYDVR0gBBowGDAIBgZngQwBAgEwDAYKKwYBBAHWeQIF
AzA8BgNVHR8ENTAzMDGgL6AthitodHRwOi8vY3Jscy5wa2kuZ29vZy9ndHMxYzMv
UXFGeGJpOU00OGMuY3JsMIIBAwYKKwYBBAHWeQIEAgSB9ASB8QDvAHUArfe++nz/
EMiLnT2cHj4YarRnKV3PsQwkyoWGNOvcgooAAAGJ810SpQAABAMARjBEAiAYYZcH
BglKWaRZZwHmSyivFL7R9mfQ3aTTQW3m82NKMQIgcjvSOUpxSu8mV2yQXWO6qpQQ
XnTCsnfbglNhUSfYo3MAdgB6MoxU2LcttiDqOOBSHumEFnAyE4VNO9IrwTpXo1Lr
UgAAAYnzXRKpAAAEAwBHMEUCIHhWO7OWbwV12jAhZ0tU8reCe/TRr/N5rCUfbnvj
RmgDAiEA+87j3l2GNS1KM/KntN1Fvga7yBmrRIaZgNOZ2YrTvV8wDQYJKoZIhvcN
AQELBQADggEBAJXYtoC6tsqm2dBSzT6s58nyi+pbAe9i229mVrRoyLWSNuhzff4C
dncA0AqivaeiDvjGWP1U0UtT9VAbf4XzFsv6QjKL6kI1rKHXCNjGYzELeeAXl5DL
wnGQTdhoMuHotNF5p+hZpzk9tAAvyZk+WeKdrjDQoBptGznZ5dE6NdgEL82u+TnD
7UI1cgJB6c7xPHnJ5s7QKAUyA9lf4ZFiCllVSbVgMU92WQKbExy8FEzHPlzYxWBB
FM4dMXxNbGA1ltOzR4W4FIASZ0m8Oce38HogWXOc6cbX61CubqiKczQZ/6WtIUWF
kes3Q2vdrL64S6lsrNQ5nfqaDV8Dc2CK2rE=
-----END CERTIFICATE-----

ca-chain.crt

-----BEGIN CERTIFICATE-----
MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQsw
CQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEU
MBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAw
MDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZp
Y2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzCCASIwDQYJKoZIhvcNAQEBBQAD
ggEPADCCAQoCggEBAPWI3+dijB43+DdCkH9sh9D7ZYIl/ejLa6T/belaI+KZ9hzp
kgOZE3wJCor6QtZeViSqejOEH9Hpabu5dOxXTGZok3c3VVP+ORBNtzS7XyV3NzsX
lOo85Z3VvMO0Q+sup0fvsEQRY9i0QYXdQTBIkxu/t/bgRQIh4JZCF8/ZK2VWNAcm
BA2o/X3KLu/qSHw3TT8An4Pf73WELnlXXPxXbhqW//yMmqaZviXZf5YsBvcRKgKA
gOtjGDxQSYflispfGStZloEAoPtR28p3CwvJlk/vcEnHXG0g/Zm0tOLKLnf9LdwL
tmsTDIwZKxeWmLnwi/agJ7u2441Rj72ux5uxiZ0CAwEAAaOCAYAwggF8MA4GA1Ud
DwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0T
AQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUinR/r4XN7pXNPZzQ4kYU83E1HScwHwYD
VR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYG
CCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcw
AoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQt
MCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsMFcG
A1UdIARQME4wOAYKKwYBBAHWeQIFAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3Br
aS5nb29nL3JlcG9zaXRvcnkvMAgGBmeBDAECATAIBgZngQwBAgIwDQYJKoZIhvcN
AQELBQADggIBAIl9rCBcDDy+mqhXlRu0rvqrpXJxtDaV/d9AEQNMwkYUuxQkq/BQ
cSLbrcRuf8/xam/IgxvYzolfh2yHuKkMo5uhYpSTld9brmYZCwKWnvy15xBpPnrL
RklfRuFBsdeYTWU0AIAaP0+fbH9JAIFTQaSSIYKCGvGjRFsqUBITTcFTNvNCCK9U
+o53UxtkOCcXCb1YyRt8OS1b887U7ZfbFAO/CVMkH8IMBHmYJvJh8VNS/UKMG2Yr
PxWhu//2m+OBmgEGcYk1KCTd4b3rGS3hSMs9WYNRtHTGnXzGsYZbr8w0xNPM1IER
lQCh9BIiAfq0g3GvjLeMcySsN1PCAJA/Ef5c7TaUEDu9Ka7ixzpiO2xj2YC/WXGs
Yye5TBeg2vZzFb8q3o/zpWwygTMD0IZRcZk0upONXbVRWPeyk+gB9lm+cZv9TSjO
z23HFtz30dZGm6fKa+l3D/2gthsjgx0QGtkJAITgRNOidSOzNIb2ILCkXhAd4FJG
AJ2xDx8hcFH1mt0G/FX0Kw4zd8NLQsLxdxP8c4CU6x+7Nz/OAipmsHMdMqUybDKw
juDEI/9bfU1lcKwrmz3O2+BtjjKAvpafkmO8l7tdufThcV4q5O8DIrGKZTqPwJNl
1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBX
MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UE
CxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYx
OTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoT
GUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIx
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63
ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwS
iV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351k
KSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZ
DrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zk
j5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5
cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esW
CruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499
iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35Ei
Eua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbap
sZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b
9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAf
BgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIw
JQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUH
MAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6Al
oCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAy
MAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIF
AwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9
NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9
WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw
9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy
+qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvi
d0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDdTCCAl2gAwIBAgILAgAAAAAA1ni3lAUwDQYJKoZIhvcNAQEEBQAwVzELMAkG
A1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExEDAOBgNVBAsTB1Jv
b3QgQ0ExGzAZBgNVBAMTEkdsb2JhbFNpZ24gUm9vdCBDQTAeFw05ODA5MDExMjAw
MDBaFw0xNDAxMjgxMjAwMDBaMFcxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9i
YWxTaWduIG52LXNhMRAwDgYDVQQLEwdSb290IENBMRswGQYDVQQDExJHbG9iYWxT
aWduIFJvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDaDuaZ
jc6j40+Kfvvxi4Mla+pIH/EqsLmVEQS98GPR4mdmzxzdzxtIK+6NiY6arymAZavp
xy0Sy6scTHAHoT0KMM0VjU/43dSMUBUc71DuxC73/OlS8pF94G3VNTCOXkNz8kHp
1Wrjsok6Vjk4bwY8iGlbKk3Fp1S4bInMm/k8yuX9ifUSPJJ4ltbcdG6TRGHRjcdG
snUOhugZitVtbNV4FpWi6cgKOOvyJBNPc1STE4U6G7weNLWLBYy5d4ux2x8gkasJ
U26Qzns3dLlwR5EiUWMWea6xrkEmCMgZK9FGqkjWZCrXgzT/LCrBbBlDSgeF59N8
9iFo7+ryUp9/k5DPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIABjAdBgNVHQ4EFgQU
YHtmGkUNl8qJUC99BM00qP/8/UswDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0B
AQQFAAOCAQEArqqf/LfSyx9fOSkoGJ40yWxPbxrwZKJwSk8ThptgKJ7ogUmYfQq7
5bCdPTbbjwVR/wkxKh/diXeeDy5slQTthsu0AD+EAk2AaioteAuubyuig0SDH81Q
gkwkr733pbTIWg/050deSY43lv6aiAU62cDbKYfmGZZHpzqmjIs8d/5GY6dT2iHR
rH5Jokvmw2dZL7OKDrssvamqQnw1wdh/1acxOk5jQzmvCLBhNIzTmKlDNPYPhyk7
ncJWWJh3w/cbrPad+D6qp1RF8PX51TFl/mtYnHGzHtdS6jIX/EBgHcl5JLL2bP2o
Zg6C3ZjL2sJETy6ge/L3ayx2EYRGinij4w==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDdTCCAl2gAwIBAgILBAAAAAABFUtaw5QwDQYJKoZIhvcNAQEFBQAwVzELMAkG
A1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExEDAOBgNVBAsTB1Jv
b3QgQ0ExGzAZBgNVBAMTEkdsb2JhbFNpZ24gUm9vdCBDQTAeFw05ODA5MDExMjAw
MDBaFw0yODAxMjgxMjAwMDBaMFcxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9i
YWxTaWduIG52LXNhMRAwDgYDVQQLEwdSb290IENBMRswGQYDVQQDExJHbG9iYWxT
aWduIFJvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDaDuaZ
jc6j40+Kfvvxi4Mla+pIH/EqsLmVEQS98GPR4mdmzxzdzxtIK+6NiY6arymAZavp
xy0Sy6scTHAHoT0KMM0VjU/43dSMUBUc71DuxC73/OlS8pF94G3VNTCOXkNz8kHp
1Wrjsok6Vjk4bwY8iGlbKk3Fp1S4bInMm/k8yuX9ifUSPJJ4ltbcdG6TRGHRjcdG
snUOhugZitVtbNV4FpWi6cgKOOvyJBNPc1STE4U6G7weNLWLBYy5d4ux2x8gkasJ
U26Qzns3dLlwR5EiUWMWea6xrkEmCMgZK9FGqkjWZCrXgzT/LCrBbBlDSgeF59N8
9iFo7+ryUp9/k5DPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8E
BTADAQH/MB0GA1UdDgQWBBRge2YaRQ2XyolQL30EzTSo//z9SzANBgkqhkiG9w0B
AQUFAAOCAQEA1nPnfE920I2/7LqivjTFKDK1fPxsnCwrvQmeU79rXqoRSLblCKOz
yj1hTdNGCbM+w6DjY1Ub8rrvrTnhQ7k4o+YviiY776BQVvnGCv04zcQLcFGUl5gE
38NflNUVyRRBnMRddWQVDf9VMOyGj/8N7yy5Y0b2qvzfvGn9LhJIZJrglfCm7ymP
AbEVtQwdpf5pLGkkeB6zpxxxyU7KyJesF12KwvhHhm4qxFYxldBniYUr+WymXUad
DKqC5JlR3XC321Y9YeRq4VzW9v493kHMB65jUr9TU/Qr6cf9tveCX4XSQRjbgbME
HMUfpIBvFSDJ3gyICh3WZlXi/EjJKSZp4A==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDdTCCAl2gAwIBAgILBAAAAAABFUtaw5QwDQYJKoZIhvcNAQEFBQAwVzELMAkG
A1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExEDAOBgNVBAsTB1Jv
b3QgQ0ExGzAZBgNVBAMTEkdsb2JhbFNpZ24gUm9vdCBDQTAeFw05ODA5MDExMjAw
MDBaFw0yODAxMjgxMjAwMDBaMFcxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9i
YWxTaWduIG52LXNhMRAwDgYDVQQLEwdSb290IENBMRswGQYDVQQDExJHbG9iYWxT
aWduIFJvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDaDuaZ
jc6j40+Kfvvxi4Mla+pIH/EqsLmVEQS98GPR4mdmzxzdzxtIK+6NiY6arymAZavp
xy0Sy6scTHAHoT0KMM0VjU/43dSMUBUc71DuxC73/OlS8pF94G3VNTCOXkNz8kHp
1Wrjsok6Vjk4bwY8iGlbKk3Fp1S4bInMm/k8yuX9ifUSPJJ4ltbcdG6TRGHRjcdG
snUOhugZitVtbNV4FpWi6cgKOOvyJBNPc1STE4U6G7weNLWLBYy5d4ux2x8gkasJ
U26Qzns3dLlwR5EiUWMWea6xrkEmCMgZK9FGqkjWZCrXgzT/LCrBbBlDSgeF59N8
9iFo7+ryUp9/k5DPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8E
BTADAQH/MB0GA1UdDgQWBBRge2YaRQ2XyolQL30EzTSo//z9SzANBgkqhkiG9w0B
AQUFAAOCAQEA1nPnfE920I2/7LqivjTFKDK1fPxsnCwrvQmeU79rXqoRSLblCKOz
yj1hTdNGCbM+w6DjY1Ub8rrvrTnhQ7k4o+YviiY776BQVvnGCv04zcQLcFGUl5gE
38NflNUVyRRBnMRddWQVDf9VMOyGj/8N7yy5Y0b2qvzfvGn9LhJIZJrglfCm7ymP
AbEVtQwdpf5pLGkkeB6zpxxxYu7KyJesF12KwvhHhm4qxFYxldBniYUr+WymXUad
DKqC5JlR3XC321Y9YeRq4VzW9v493kHMB65jUr9TU/Qr6cf9tveCX4XSQRjbgbME
HMUfpIBvFSDJ3gyICh3WZlXi/EjJKSZp4A==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIC6DCCAlGgAwIBAgIJAPBr4r6shQxeMA0GCSqGSIb3DQEBBQUAMFcxCzAJBgNV
BAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMRAwDgYDVQQLEwdSb290
IENBMRswGQYDVQQDExJHbG9iYWxTaWduIFJvb3QgQ0EwHhcNMTMwNjA0MjA1NDE4
WhcNMjMwNjAyMjA1NDE4WjBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFs
U2lnbiBudi1zYTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2ln
biBSb290IENBMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDg55m6XhSNLxB0
4lp3GDpLjMAeF6tPao2Mj87SUci8Qwx9toWszM28ibLNshvGqKvGao4o5JnUgsDZ
RQDlzHzMST1B6Veg4xbcr9jh2sBYDopydG+NO22PhR6yM76wewWxtCRSfhdeSzEX
3pF5ippi75006Y1Df1X4fojI97wdIwIDAQABo4G7MIG4MB0GA1UdDgQWBBRNm3Lf
BzkCLwams3kvTbTbmwo6ZjCBiAYDVR0jBIGAMH6AFE2bct8HOQIvBqazeS9NtNub
CjpmoVukWTBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1z
YTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENB
ggkA8GvivqyFDF4wDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQUFAAOBgQChN7d4
DJ952inrDBDMqThF1GtFKdePCPoWePN5P+J6e+WWjXcqaczA3YT3zyY1ARWFGTry
h7KYNgEvKdG6ZHp+LFpJUeDWEa31Yw7+lkNTWrFHL10N8PQ0HT2MK4ictIeyNPsN
jE1I//XMQbjEEIWNOFRlawM8m2I0l5E+1C3DMg==
-----END CERTIFICATE-----
7

@johscheuer If you have time to try running fdbbackup with these certs, it would be much appreciated.

8

I’m also facing this issue.

A minimum reproducible for the issue is as follows for FoundationDB 7.3.37. All of these commands are run on EC2 instance running Amazon Linux 2023 in eu-west-2.

I’ve found that just a single, self-signed certificate is sufficient to connect to S3 - ‘Amazon Root CA 1’ expiring January 17 2038 (found at https://www.amazontrust.com/repository/AmazonRootCA1.pem)

FDB_TLS_CA_FILE=AmazonRootCA1.pem fdbbackup describe \
  -d blobstore://<AWS_ACCESS_KEY>@s3.eu-central-1.amazonaws.com/<BACKUP_NAME>?bucket=<BUCKET_NAME>

[...]
Restorable: true
Partitioned logs: false
[...]

This succeeds, despite the fact that the chain when connecting to ‘s3.eu-central-1.amazonaws.com’ looks like so:

[root@ ~]# openssl s_client -connect s3.eu-central-1.amazonaws.com:443
[...]
Certificate chain
 0 s:CN = *.s3.eu-central-1.amazonaws.com
   i:C = US, O = Amazon, CN = Amazon RSA 2048 M01
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Nov 13 00:00:00 2024 GMT; NotAfter: Nov 11 23:59:59 2025 GMT
 1 s:C = US, O = Amazon, CN = Amazon RSA 2048 M01
   i:C = US, O = Amazon, CN = Amazon Root CA 1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 23 22:21:28 2022 GMT; NotAfter: Aug 23 22:21:28 2030 GMT
 2 s:C = US, O = Amazon, CN = Amazon Root CA 1
   i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 25 12:00:00 2015 GMT; NotAfter: Dec 31 01:00:00 2037 GMT
 3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
   i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  2 00:00:00 2009 GMT; NotAfter: Jun 28 17:39:16 2034 GMT

So it seems that FDB does NOT require the full certificate chain when connecting.

For reference, I’ve tested the negative case and it fails as expected. If I supply ‘GTS Root R1’ expiring June 22 2036 (found at https://i.pki.goog/r1.pem), it fails as expected as this CA is not anywhere in the chain for s3.eu-central-1.amazonaws.com. See the command and the errors.

FDB_TLS_CA_FILE=gts-root-r1.pem fdbbackup describe \
  -d blobstore://<AWS_ACCESS_KEY>@s3.eu-central-1.amazonaws.com/<BACKUP_NAME>?bucket=<BUCKET_NAME>

<Event Severity="20" Time="1736950738.455323" DateTime="2025-01-15T14:18:58Z" Type="TLSPolicyFailure" ID="0000000000000000" SuppressedEventCount="0" Reason="preverification failed" VerifyError="unable to get local issuer certificate" ThreadID="10946202708076983502" LogGroup="" />
<Event Severity="20" Time="1736950738.455323" DateTime="2025-01-15T14:18:58Z" Type="N2_ConnectHandshakeError" ID="887d16759e6b9c32" ErrorCode="337047686" Message="certificate verify failed (SSL routines, tls_process_server_certificate)" WhichMeans="error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed" ThreadID="10946202708076983502" LogGroup="" />

I’ve also attempted to use the default certificate chain as suggested by @johscheuer. This works for S3, but not for Google Cloud Storage.

[root@ ~]# openssl version -d
OPENSSLDIR: "/etc/pki/tls"

[root@ ~]# wc -l /etc/pki/tls/certs/ca-bundle.crt
3915 /etc/pki/tls/certs/ca-bundle.crt

[root@ ~]# FDB_TLS_CA_FILE=/etc/pki/tls/certs/ca-bundle.crt fdbbackup describe \
  -d blobstore://<ACCESS_KEY>@s3.eu-central-1.amazonaws.com/<BACKUP_NAME>?bucket=<BUCKET_NAME>
[...]
Restorable: true
Partitioned logs: false
[...]

[root@ ~]# FDB_TLS_CA_FILE=/etc/pki/tls/certs/ca-bundle.crt fdbbackup describe -d "blobstore://<ACCESS_KEY>@storage.googleapis.com/<BACKUP_NAME>?bucket=<BUCKET_NAME>&region=<BUCKET_REGION_NAME>" --knob_resolve_prefer_ipv4_addr=1 --knob_http_verbose_level=4 --knob_http_request_aws_v4_header=true
ERROR: Network connection failed
Fatal Error: Network connection failed

The failure logs are identical to @larshagen’s.

<Event Severity="20" Time="1736951324.626108" DateTime="2025-01-15T14:28:44Z" Type="TLSPolicyFailure" ID="0000000000000000" SuppressedEventCount="0" Reason="preverification failed" VerifyError="self signed certificate" ThreadID="977878255095166814" LogGroup="" />
<Event Severity="20" Time="1736951324.626108" DateTime="2025-01-15T14:28:44Z" Type="N2_ConnectHandshakeError" ID="7e545e1c25d9a3be" ErrorCode="337047686" Message="certificate verify failed (SSL routines, tls_process_server_certificate)" WhichMeans="error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed" ThreadID="977878255095166814" LogGroup="" />

Adding ‘&sc=0’ to the URL like so:

blobstore://<ACCESS_KEY>@storage.googleapis.com/<BACKUP_NAME>?bucket=<BUCKET_NAME>&region=<BUCKET_REGION_NAME>&sc=0

Succeeds. However, I don’t want to send my keys over the internet unencrypted.

I’ve tried using openssl to connect to ‘storage.googleapis.com’ directly, and this works just fine.

[root@ ~]# openssl s_client -no-CApath -CAfile /etc/pki/tls/certs/ca-bundle.crt -connect storage.googleapis.com:443 -verify_return_error
CONNECTED(00000003)
[...]
Verify return code: 0 (ok)

For fun, we can also try some other domains to see who we CAN connect to! As we’re still at the TLS stage, it can only help to illuminate us as to who we may and may not connect to using the system root certificate store using FoundationDB.

‘disney.com’:

[root@ ~]# FDB_TLS_CA_FILE=/etc/pki/tls/certs/ca-bundle.crt fdbbackup describe -d "blobstore://<ACCESS_KEY>@disney.com/<BACKUP_NAME>?bucket=<BUCKET_NAME>&region=<BUCKET_REGION_NAME>" --knob_resolve_prefer_ipv4_addr=1 --knob_http_verbose_level=4 --knob_http_request_aws_v4_header=true

<Event Severity="10" Time="1736951756.342401" DateTime="2025-01-15T14:35:56Z" Type="S3BlobStoreEndpointNewConnectionSuccess" ID="0000000000000000" SuppressedEventCount="0" RemoteEndpoint="130.211.198.204:443:tls(fromHostname)" ExpiresIn="120" Proxy="" ThreadID="15557844396177594054" LogGroup="" />

Same as above, but ‘forums.foundationdb.org’ instead.

<Event Severity="10" Time="1736951899.166071" DateTime="2025-01-15T14:38:19Z" Type="S3BlobStoreEndpointNewConnectionSuccess" ID="0000000000000000" SuppressedEventCount="0" RemoteEndpoint="184.105.99.43:443:tls(fromHostname)" ExpiresIn="120" Proxy="" ThreadID="11171483904373147865" LogGroup="" />

Unfortunately, neither Disney nor this forum are where I want to back up to.

Testing additional Google and Google-adjacent domains such as ‘google.com’ and ‘android.com’ in the same way as above yield the same errors as for ‘storage.googleapis.com’. I suppose Google has their TLS configured across all their domains in such a way that is incompatible with how FoundationDB is performing handshakes.

Using ssllabs (SSL Server Test (Powered by Qualys SSL Labs)) I’ve attempted to narrow down some differentiating factor between Google’s TLS and everyone else. I’ve considered:

  • TLS versions
  • The sending of multiple certificate (chains)
  • The presence of non-RSA2048-keyed certificates

but I have not found one yet.

I’ve also tested this against the latest versions of FoundationDB 7.1 & 7.3 on my laptop running macOS 15.2 and see the same behaviour.

If you have any ideas, please let me know!