Backup to GCS failing TLS preverification

larshagen · September 20, 2023, 7:36am

We are trying to run backups directly to Google Cloud Storage (GCS) from our FDB cluster. We are running on 7.1.39, which should have all the necessary compatibility fixes, but it is still failing TLS verification, complaining that the cert for storage.googleapis.com is self-signed.

We have tried running with Check.Valid=0, and then the backup runs successfully, so we are certain TLS is the only issue. The cluster we are backing up from is not running with TLS internally. We have tried downloading root certs from Google, concatenating it in a file and pointing to it with FDB_TLS_CA_FILE, but that has not helped. We have also tried creating a certificate for FDB, and populating FDB_TLS_KEY_FILE,FDB_TLS_CA_FILE,FDB_TLS_CERTIFICATE_FILE, but that did not help (even when combining FDB CA with Google CA in CA file).

Examining the cert from storage.googleapis.com with openssl, I am fairly certain that it is not self-signed, and that FDB is mistaken for some reason.

Any help would be appreciated.

<Event Severity="20" Time="1695126371.137127" DateTime="2023-09-19T12:26:11Z" Type="TLSPolicyFailure" ID="0000000000000000" SuppressedEventCount="0" Reason="preverification failed" VerifyError="self signed certificate" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126371.137127" DateTime="2023-09-19T12:26:11Z" Type="N2_ConnectHandshakeError" ID="4ccee7736a127f57" SuppressedEventCount="0" ErrorCode="337047686" Message="certificate verify failed (SSL routines, tls_process_server_certificate)" WhichMeans="error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126371.137127" DateTime="2023-09-19T12:26:11Z" Type="S3BlobStoreEndpointRequestFailedRetryable" ID="0000000000000000" Error="connection_failed" ErrorDescription="Network connection failed" ErrorCode="1026" SuppressedEventCount="0" ConnectionEstablished="0" RemoteHost="storage.googleapis.com" Verb="HEAD" Resource="/cognitedata-greenfield-foundationdb-backup" ThisTry="1" RetryDelay="2" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126373.147324" DateTime="2023-09-19T12:26:13Z" Type="TLSPolicyFailure" ID="0000000000000000" SuppressedEventCount="0" Reason="preverification failed" VerifyError="self signed certificate" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126373.147324" DateTime="2023-09-19T12:26:13Z" Type="N2_ConnectHandshakeError" ID="a2ede36bc2b4133f" SuppressedEventCount="0" ErrorCode="337047686" Message="certificate verify failed (SSL routines, tls_process_server_certificate)" WhichMeans="error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126377.157543" DateTime="2023-09-19T12:26:17Z" Type="TLSPolicyFailure" ID="0000000000000000" SuppressedEventCount="0" Reason="preverification failed" VerifyError="self signed certificate" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126377.157543" DateTime="2023-09-19T12:26:17Z" Type="N2_ConnectHandshakeError" ID="92f8fc9ca4e98b67" SuppressedEventCount="0" ErrorCode="337047686" Message="certificate verify failed (SSL routines, tls_process_server_certificate)" WhichMeans="error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126385.175072" DateTime="2023-09-19T12:26:25Z" Type="TLSPolicyFailure" ID="0000000000000000" SuppressedEventCount="0" Reason="preverification failed" VerifyError="self signed certificate" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="20" Time="1695126385.175072" DateTime="2023-09-19T12:26:25Z" Type="N2_ConnectHandshakeError" ID="1e37e11865b22d65" SuppressedEventCount="0" ErrorCode="337047686" Message="certificate verify failed (SSL routines, tls_process_server_certificate)" WhichMeans="error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed" ThreadID="17885686041784278561" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />
<Event Severity="40" ErrorKind="Unset" Time="1695126401.103105" DateTime="2023-09-19T12:26:41Z" Type="StopAfterError" ID="0000000000000000" Error="backup_error" ErrorDescription="Backup error" ErrorCode="2300" ThreadID="17885686041784278561" Backtrace="addr2line -e fdbbackup.debug -p -C -f -i 0x113f7c8 0x113dffb 0x113e201 0x619b21 0x619d59 0x656a70 0x635d25 0x6364cc 0x656a70 0x64380f 0x6441ab 0x656a70 0x7fc3b1 0x8696cc 0x656a70 0x60ea6f 0x10c69a8 0xafe832 0x5f2f4a 0x7ff0fb88b555" Machine="10.52.7.175:213" LogGroup="default" ClientDescription="primary-7.1.39-17885686041784278561" />

johscheuer · September 20, 2023, 8:58am

We have tried downloading root certs from Google, concatenating it in a file and pointing to it with FDB_TLS_CA_FILE , but that has not helped.

Have you tried to connect with openssl and the new CA file to GCS? FDB requires the full CA cert chain when connecting to a TLS service. If you are able to connect to the storage API with curl and the default certificate chains on your host/container, you could try to point FDB_TLS_CA_FILE to the chain (given that your cluster is running without TLS).

larshagen · September 20, 2023, 11:10am

Steps taken to verify:

echo | openssl s_client -showcerts -servername "storage.googleapis.com" -connect "storage.googleapis.com:443" 2>/dev/null

Store the direct certificate in cert.crt and the rest of the chain in ca-chain.crt.
Run verification with openssl

openssl verify -CAfile ./ca-chain.crt ./cert.pem

This returned an error:

C = US, O = Google Trust Services LLC, CN = GTS Root R1
error 2 at 2 depth lookup: unable to get issuer certificate
error ./cert.pem: verification failed

After some digging, I found that the ca file was missing a root cert from GlobalSign. After adding it, I got

./cert.pem: OK

Rerunning fdbbackup with --tls_ca_file ca-chain.crt, I got the same verification error.

larshagen · September 22, 2023, 7:37am

So, in short, FDB fails verification even when the full chain can be verified by openssl.

johscheuer · September 25, 2023, 8:55am

So, in short, FDB fails verification even when the full chain can be verified by openssl.

Are you able to share your full chain for GCS? It might be that openssl itself is more tolerant than FDB. Have you tried to let FDB use the default host CA chain?

larshagen · September 25, 2023, 8:58am

cert.crt:

ca-chain.crt

larshagen · September 30, 2023, 8:21am

@johscheuer If you have time to try running fdbbackup with these certs, it would be much appreciated.