After digging into the answers on my first question I’m investing the time to enable TLS in my cluster. For now, I’m working in a test environment, which fact might be relevant later in this question. I’ve got a self-signed cert created per the project documentation and have got the ‘crt’ installed with the other system certificates. That is, the relevant section in my
foundationdb.conf looks like so:
tls_certificate_file = /foundationdb/tls/fdb.pem tls_key_file = /foundationdb/tls/private.key tls_ca_file = /etc/ssl/certs/ca-certificates.crt
The relevant environment variables are set similarly. So far, so good I think. Now, however, when I run a backup I get the following error message in the trace logs:
<Event Severity="40" Time="1563409564.569001" Type="FDBLibTLSConnectError" ID="94cd3c72f39be8ce" LibTLSErrorMessage="ssl verify memory setup failure" Backtrace="addr2line -e fdbback up.debug -p -C -f -i 0x9f0136 0x9ee88f 0xa385ef 0xa361d4 0x94287e 0x942b57 0x94359d 0xa2246c 0xa2303a 0x478f40 0xa2181f 0xa21b16 0x8a7327 0xa1d3c1 0xa1eb14 0xa18b1a 0xa19acb 0x74046 e 0x423c15 0x7fb4b710fb97" Machine="172.17.0.6:1024" LogGroup="default" /> <Event Severity="20" Time="1563409564.569001" Type="BlobStoreEndpointRequestFailedRetryable" ID="0000000000000000" Error="tls_error" ErrorDescription="TLS error" ErrorCode="2107" Su ppressedEventCount="0" ConnectionEstablished="0" RemoteHost="s3.us-west-1.amazonaws.com" Verb="HEAD" Resource="/fdb-backups" ThisTry="1" RetryDelay="2" Machine="172.17.0.6:1024" Log Group="default" />
For reference, here’s how I’m running backups (with some redactions):
> fdbbackup start --log -d "blobstore://REDACT:REDACT@s3.us-west-1.amazonaws.com:443/test?bucket=fdb-backups"
Okay, now about my test environment. It is a touch memory constrained: processes in the test cluster don’t have a full 4Gi of memory available to them but giving the test cluster more memory – but not quite 4Gi – didn’t resolve the backup error. None of the worker processes in the cluster demonstrated memory pressure when I ran the backup command, either. I tend to disbelieve that a lack of memory in my test cluster is the culprit here.
I can’t find much online about the error message bubbling up from LibreSSL. This is maybe the most comprehensive discussion. Has anyone else encountered “ssl verify memory setup failure” before or have a clearer idea of what it might mean than I do at present?