Enabling TLS on first install for single node setup

I’m unable to decipher what goes in foundationdb.conf to have a TLS by
default cluster. I’m hoping to provide the FreeBSD port by default with TLS
enabled, and enough instructions for an admin to deploy a single node
self-signed cert setup, to get started fast.

more notes

Here’s my setup (FreeBSD, recent master, single node), compiled against
FreeBSD 13.0-CURRENT’s openssl in base, as a dynamic library (cmake
tweaked to allow that).

basic config pre-TLS

# echo foundationdb:foundationdb@127.0.0.1:4500 \
    | tee /usr/local/etc/foundationdb/fdb.cluster

Add users to the foundationdb group so that they can read the cluster
file, and access the DB, via pw groupmod -m foundationdb $USER.

set permissions

# touch /var/run/fdbmonitor.pid
# echo foundationdb_enable=YES > /etc/rc.conf.d/foundationdb
# zfs create zroot/var/db/foundationdb
# chown -R foundationdb:foundationdb \
    /usr/local/etc/foundationdb \
    /var/log/foundationdb \
    /var/db/foundationdb \
    /var/run/fdbmonitor.pid
# chmod -R u=rw,g=r,o-rwx,ug+X \
    /usr/local/etc/foundationdb \
    /var/log/foundationdb \
    /var/db/foundationdb \
    /var/run/fdbmonitor.pid

initialise the DB

As root, service foundationdb start and then run the rest as a normal
user:

$ fdbcli --exec "configure new single memory"
$ fdbcli
fdbcli
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.

The database is available.

Welcome to the fdbcli. For help, type `help'.
fdb> status

Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.

Configuration:
  Redundancy mode        - single
  Storage engine         - memory-2
  Coordinators           - 1

Cluster:
  FoundationDB processes - 1
  Zones                  - 1
  Machines               - 1
  Memory availability    - 25.5 GB per process on machine with least available
  Fault Tolerance        - 0 machines
  Server time            - 01/21/20 00:07:14

Data:
  Replication health     - Healthy
  Moving data            - 0.000 GB
  Sum of key-value sizes - 0 MB
  Disk space used        - 105 MB

Operating space:
  Storage server         - 1.0 GB free on most full server
  Log server             - 226.0 GB free on most full server

Workload:
  Read rate              - 7 Hz
  Write rate             - 0 Hz
  Transactions started   - 3 Hz
  Transactions committed - 0 Hz
  Conflict rate          - 0 Hz

Backup and DR:
  Running backups        - 0
  Running DRs            - 0

Client time: 01/21/20 00:07:14

fdb> 

use the DB

$ fdbserver --version
FoundationDB 6.3 (v6.3.0)
source version
protocol fdb00b063010001

$ fdbcli --exec "writemode on; set poop emoji"
>>> writemode on
>>> set poop emoji
Committed (1366086432)

$ fdbcli --exec "get poop emoji"
`poop' is `emoji'

$ fdbcli --exec "writemode on; set poop emojis"
>>> writemode on
>>> set poop emojis
Committed (1382502961)

$ fdbcli --exec "get poop emoji"
`poop' is `emojis'

processes

# pgrep -ilU foundationdb
77073 backup_agent
94492 fdbmonitor
72585 fdbserver

# pgrep -ilfU foundationdb
77073 /usr/local/bin/backup_agent \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --logdir=/var/log/foundationdb
94492 fdbmonitor \
    --conffile /usr/local/etc/foundationdb/foundationdb.conf \
    --lockfile /var/run/fdbmonitor.pid
72585 /usr/local/bin/fdbserver \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --datadir=/var/db/foundationdb/data/4500 \
    --listen_address=public \
    --logdir=/var/log/foundationdb \
    --public_address=auto:4500

files

# tree /var/db/foundationdb/ /usr/local/etc/foundationdb/ /var/log/foundationdb/
/var/db/foundationdb/
└── data
    └── 4500
        ├── coordination-0.fdq
        ├── coordination-1.fdq
        ├── log2-V_3_LS_2-6f5d01fbbca6ef21906798ec22911edf.sqlite
        ├── log2-V_3_LS_2-6f5d01fbbca6ef21906798ec22911edf.sqlite-wal
        ├── logqueue-V_3_LS_2-6f5d01fbbca6ef21906798ec22911edf-0.fdq
        ├── logqueue-V_3_LS_2-6f5d01fbbca6ef21906798ec22911edf-1.fdq
        ├── processId
        ├── storage-d528650a51442d78738b28cc972a9e61-0.fdq
        └── storage-d528650a51442d78738b28cc972a9e61-1.fdq
/usr/local/etc/foundationdb/
├── fdb.cluster
├── foundationdb.conf
└── foundationdb.conf.sample
/var/log/foundationdb/
├── trace.127.0.0.1.4500.1579564792.dAiyK8.0.1.xml
└── trace.127.0.0.1.4500.1579564944.GhdRBR.0.1.xml

switching to TLS

This doesn’t work yet I’m just documenting what I did

# cat /usr/local/etc/foundationdb/foundationdb.conf
## foundationdb.conf
##
## Configuration file for FoundationDB server processes
## Full documentation is available at
## https://apple.github.io/foundationdb/configuration.html#the-configuration-file

[fdbmonitor]
user = foundationdb
group = foundationdb

[general]
trace_format = json
restart_delay = 60
## by default, restart_backoff = restart_delay_reset_interval = restart_delay
# initial_restart_delay = 0
# restart_backoff = 60
# restart_delay_reset_interval = 60
cluster_file = /usr/local/etc/foundationdb/fdb.cluster
# delete_envvars =
# kill_on_configuration_change = true

## Default parameters for individual fdbserver processes
# https://apple.github.io/foundationdb/configuration.html#foundationdb-conf-fdbserver
[fdbserver]
command = /usr/local/bin/fdbserver
public_address = auto:$ID
listen_address = public
datadir = /var/db/foundationdb/data/$ID
logdir = /var/log/foundationdb
# logsize = 10MiB
# maxlogssize = 100MiB
# machine_id =
# datacenter_id =
# class =
# memory = 8GiB
# storage_memory = 1GiB
# cache_memory = 2GiB
# metrics_cluster =
# metrics_prefix =

# TLS settings
# https://apple.github.io/foundationdb/tls.html
# https://forums.foundationdb.org/t/solved-correct-setup-of-tls-for-foundationdb/1037/4
tls_certificate_file=/usr/local/etc/foundationdb/fdb.pem
tls_verify_peers=Check.Valid=0


## An individual fdbserver process with id 4500
## Parameters set here override defaults from the [fdbserver] section
[fdbserver.4500]

[backup_agent]
command = /usr/local/bin/backup_agent
logdir = /var/log/foundationdb

[backup_agent.1]

Then stop & restart DB with new TLS configs:

#### stop all the things
# service foundationdb stop
# cd /usr/local/etc/foundationdb
# openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout private.key -out cert.crt
# cat cert.crt private.key > fdb.pem
#### switch to TLS cluster
# cat /usr/local/etc/foundationdb/fdb.cluster
foundationdb:foundationdb@127.0.0.1:4600:tls
# ls -AFGhl /usr/local/etc/foundationdb/
total 23
-rw-r-----  1 dch  foundationdb   1.3K May  3 20:22 cert.crt
-rw-r-----  1 dch  foundationdb    45B May  8 16:36 fdb.cluster
-rw-r-----  1 dch  foundationdb   3.0K May  3 20:22 fdb.pem
-rw-r-----  1 dch  foundationdb   1.6K May  9 09:11 foundationdb.conf
-rw-r-----  1 dch  foundationdb   1.2K May  3 11:07 foundationdb.conf.sample
-rw-r-----  1 dch  foundationdb   1.7K May  3 20:21 private.key
# service foundationdb start# pgrep -ilfU foundationdb
82184 /usr/local/bin/fdbserver --cluster_file --datadir --listen_address --logdir --public_address --tls_certificate_file --tls_verify_peers --trace_format
62203 /usr/local/bin/backup_agent --cluster_file=/usr/local/etc/foundationdb/fdb.cluster --logdir=/var/log/foundationdb
# sockstat -46l | grep fdb
foundationdb fdbserver82184 17 tcp4 127.0.0.1:4500        *:*
# ldd /usr/local/bin/fdbserver
/usr/local/bin/fdbserver:
	libeio.so.1 => /usr/local/lib/libeio.so.1 (0x802a13000)
	librt.so.1 => /usr/lib/librt.so.1 (0x802a1f000)
	libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x802a28000)
	libdevstat.so.7 => /lib/libdevstat.so.7 (0x802a2e000)
	libssl.so.111 => /usr/lib/libssl.so.111 (0x802a37000)   <--- OpenSSL1.1.1 in base
	libcrypto.so.111 => /lib/libcrypto.so.111 (0x802acf000)
	libc++.so.1 => /usr/lib/libc++.so.1 (0x802d9e000)
	libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x802e6b000)
	libm.so.5 => /lib/libm.so.5 (0x802e8e000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x802ec0000)
	libthr.so.3 => /lib/libthr.so.3 (0x802eda000)
	libc.so.7 => /lib/libc.so.7 (0x802f06000)
	libelf.so.2 => /lib/libelf.so.2 (0x803312000)
	libkvm.so.7 => /lib/libkvm.so.7 (0x80332e000)

At this point the fdbserver process is not listening on TLS port and I’m not sure what I should do differently. TLS file permissions seem OK, & I’ve tried a bunch of fdbserver direct commandline invocations without luck.

Is the config correct in fdb.cluster and foundationdb.conf?

I’ll attach the json logs in the gist.