Backup failure: "ssl verify memory setup failure"

Using FoundationDB
1

After digging into the answers on my first question I’m investing the time to enable TLS in my cluster. For now, I’m working in a test environment, which fact might be relevant later in this question. I’ve got a self-signed cert created per the project documentation and have got the ‘crt’ installed with the other system certificates. That is, the relevant section in my foundationdb.conf looks like so:

tls_certificate_file = /foundationdb/tls/fdb.pem
tls_key_file = /foundationdb/tls/private.key
tls_ca_file = /etc/ssl/certs/ca-certificates.crt

The relevant environment variables are set similarly. So far, so good I think. Now, however, when I run a backup I get the following error message in the trace logs:

<Event Severity="40" Time="1563409564.569001" Type="FDBLibTLSConnectError" ID="94cd3c72f39be8ce" LibTLSErrorMessage="ssl verify memory setup failure" Backtrace="addr2line -e fdbback
up.debug -p -C -f -i 0x9f0136 0x9ee88f 0xa385ef 0xa361d4 0x94287e 0x942b57 0x94359d 0xa2246c 0xa2303a 0x478f40 0xa2181f 0xa21b16 0x8a7327 0xa1d3c1 0xa1eb14 0xa18b1a 0xa19acb 0x74046
e 0x423c15 0x7fb4b710fb97" Machine="172.17.0.6:1024" LogGroup="default" />
<Event Severity="20" Time="1563409564.569001" Type="BlobStoreEndpointRequestFailedRetryable" ID="0000000000000000" Error="tls_error" ErrorDescription="TLS error" ErrorCode="2107" Su
ppressedEventCount="0" ConnectionEstablished="0" RemoteHost="s3.us-west-1.amazonaws.com" Verb="HEAD" Resource="/fdb-backups" ThisTry="1" RetryDelay="2" Machine="172.17.0.6:1024" Log
Group="default" />

For reference, here’s how I’m running backups (with some redactions):

> fdbbackup start --log -d "blobstore://REDACT:REDACT@s3.us-west-1.amazonaws.com:443/test?bucket=fdb-backups"

Okay, now about my test environment. It is a touch memory constrained: processes in the test cluster don’t have a full 4Gi of memory available to them but giving the test cluster more memory – but not quite 4Gi – didn’t resolve the backup error. None of the worker processes in the cluster demonstrated memory pressure when I ran the backup command, either. I tend to disbelieve that a lack of memory in my test cluster is the culprit here.

I can’t find much online about the error message bubbling up from LibreSSL. This is maybe the most comprehensive discussion. Has anyone else encountered “ssl verify memory setup failure” before or have a clearer idea of what it might mean than I do at present?

2

For what it’s worth, my production cluster now has the relevant TLS changes and the error when performing backups persists.

3

I have never encountered a ssl verify memory setup failure. I’ve been… relatively displeased with the quality of error messages that LibreSSL provides. For the few problems that I’ve run into, I’ve ended up doing custom builds of FDB and gdb/printf debugged LibreSSL to figure out what went wrong.

The searches I did commented that a large certificate could cause this, and someone else claimed to solve it when setting their CA certificate to be their self-signed certificate. Both sound reasonably unlikely to be your problem though.

1 Like
4

Ah, yeah, I should have mentioned the size of the certificate. It’s not huge, just 2048 bytes. I did run with setting the CA certificate to my self-signed cert but that causes backups to fail with local issuer errors, as expected. My self-signed cert needs to be in the root certificate pool for the cluster to form up and I need root certs to get far enough in the handshake with s3 to fail in the way described above.

Running a debuggable build is not a bad idea, since I can replicate this issue in a test environment.