TLS implemenation for foundationdb

Using FoundationDB
1

We have implemented TLS in our fdb cluster using self signed certificates. Even the clients can able to connect with the cluster using tls certs and can able to write the data. By using below command, we can able to connect to fdbcli using the certs.

fdbcli --tls_ca_file /etc/foundationdb/pki/ca.crt --tls_certificate_file /etc/foundationdb/pki/client-server.crt --tls_key_file /etc/foundationdb/pki/client-server.key --tls_verify_peers Check.Valid=0

fdb server version:
FoundationDB 7.1 (v7.1.5)

By using below modules in ansible, we have generated self signed certificates.

community.crypto.openssl_privatekey:
community.crypto.x509_certificate:
community.crypto.openssl_privatekey:
community.crypto.openssl_csr_pipe:

Issue:
In the foundationdb logs, we noticed that below traces are keep on generating with TLS failures.

Event Severity=“10” Time=“1660736446.267173” DateTime=“2022-08-17T11:40:46Z” Type=“IncomingConnectionError” ID=“26ae06d3d759554e” Error=“connection_failed” ErrorDescription=“Network connection failed” ErrorCode=“1026” SuppressedEventCount=“4” FromAddress=“xxxxxxxx:57414:tls” ThreadID=“7499685985715584756” Machine=“10.23.216.10:4500” LogGroup=“default” /

Event Severity=“20” Time=“1660736447.498181” DateTime=“2022-08-17T11:40:47Z” Type=“N2_AcceptHandshakeError” ID=“0000000000000000” ErrorCode=“336151576” ErrorMsg=“tlsv1 alert unknown ca (SSL routines, ssl3_read_bytes)” BackgroundThread=“1” ThreadID=“1547189418734296536” Machine=“xxxxxxx:4500” LogGroup=“default” /

what could be the issue? Please let us know the fix for this.

2

It’s trying to tell you that the accepting process doesn’t recognize the certificate that the client is presenting. You’ll need to match that up on the client and server side to understand why that’s the case.

If you add --debug-tls to your fdbcli invocation, it’ll print out the certificates passed in with the rest of the arguments, which can make the process a bit simpler.

3

Thanks for your prompt reply. we faced couple of issues with our certs then finally fixed it.

Issue 1:
While doing the cert validation with below command, we got the error as verification failed. So we recreated the certs and copied in the required path.
openssl verify -CAfile /etc/foundationdb/pki/ca.crt /etc/foundationdb/pki/client-server.crt

Issue 2:
Even after changing the certs in foundationdb VM, still we noticed few TLS failure lines in the fdb logs.

  1. In the fdb configuration file, we have multiple sections for the fdb daemons like [fdbserver], [backup-agent] etc…
    2)In the configuration file, fdbserver section as below TLS parameters but backup-agent section doesn’t have that parameters.
    [fdbserver]
    tls_certificate_file = /etc/foundationdb/pki/client-server.crt
    tls_key_file = /etc/foundationdb/pki/client-server.key
    tls_ca_file = /etc/foundationdb/pki/ca.crt
    tls_verify_peers = Check.Valid=0

  2. Since the TLS parameters are missing in the backup-agent section, the backup daemon is throwing the failure logs in the trace files.

  3. After adding the above TLS parameters, i don’t see any failures