Enabling TLS on first install for single node setup

dch · May 9, 2020, 9:32am

I’m unable to decipher what goes in foundationdb.conf to have a TLS by
default cluster. I’m hoping to provide the FreeBSD port by default with TLS
enabled, and enough instructions for an admin to deploy a single node
self-signed cert setup, to get started fast.

TLDR: fdbserver doesn’t listen on 4600 on startup
logs & configs: fdb debug failed TLS startup https://forums.foundationdb.org/t/enabling-tls-on-first-install-for-single-node-setup/2111 · GitHub

more notes

Here’s my setup (FreeBSD, recent master, single node), compiled against
FreeBSD 13.0-CURRENT’s openssl in base, as a dynamic library (cmake
tweaked to allow that).

basic config pre-TLS

# echo foundationdb:foundationdb@127.0.0.1:4500 \
    | tee /usr/local/etc/foundationdb/fdb.cluster

Add users to the foundationdb group so that they can read the cluster
file, and access the DB, via pw groupmod -m foundationdb $USER.

set permissions

# touch /var/run/fdbmonitor.pid
# echo foundationdb_enable=YES > /etc/rc.conf.d/foundationdb
# zfs create zroot/var/db/foundationdb
# chown -R foundationdb:foundationdb \
    /usr/local/etc/foundationdb \
    /var/log/foundationdb \
    /var/db/foundationdb \
    /var/run/fdbmonitor.pid
# chmod -R u=rw,g=r,o-rwx,ug+X \
    /usr/local/etc/foundationdb \
    /var/log/foundationdb \
    /var/db/foundationdb \
    /var/run/fdbmonitor.pid

initialise the DB

As root, service foundationdb start and then run the rest as a normal
user:

$ fdbcli --exec "configure new single memory"
$ fdbcli
fdbcli
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.

The database is available.

Welcome to the fdbcli. For help, type `help'.
fdb> status

Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.

Configuration:
  Redundancy mode        - single
  Storage engine         - memory-2
  Coordinators           - 1

Cluster:
  FoundationDB processes - 1
  Zones                  - 1
  Machines               - 1
  Memory availability    - 25.5 GB per process on machine with least available
  Fault Tolerance        - 0 machines
  Server time            - 01/21/20 00:07:14

Data:
  Replication health     - Healthy
  Moving data            - 0.000 GB
  Sum of key-value sizes - 0 MB
  Disk space used        - 105 MB

Operating space:
  Storage server         - 1.0 GB free on most full server
  Log server             - 226.0 GB free on most full server

Workload:
  Read rate              - 7 Hz
  Write rate             - 0 Hz
  Transactions started   - 3 Hz
  Transactions committed - 0 Hz
  Conflict rate          - 0 Hz

Backup and DR:
  Running backups        - 0
  Running DRs            - 0

Client time: 01/21/20 00:07:14

fdb>

use the DB

$ fdbserver --version
FoundationDB 6.3 (v6.3.0)
source version
protocol fdb00b063010001

$ fdbcli --exec "writemode on; set poop emoji"
>>> writemode on
>>> set poop emoji
Committed (1366086432)

$ fdbcli --exec "get poop emoji"
`poop' is `emoji'

$ fdbcli --exec "writemode on; set poop emojis"
>>> writemode on
>>> set poop emojis
Committed (1382502961)

$ fdbcli --exec "get poop emoji"
`poop' is `emojis'

processes

# pgrep -ilU foundationdb
77073 backup_agent
94492 fdbmonitor
72585 fdbserver

# pgrep -ilfU foundationdb
77073 /usr/local/bin/backup_agent \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --logdir=/var/log/foundationdb
94492 fdbmonitor \
    --conffile /usr/local/etc/foundationdb/foundationdb.conf \
    --lockfile /var/run/fdbmonitor.pid
72585 /usr/local/bin/fdbserver \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --datadir=/var/db/foundationdb/data/4500 \
    --listen_address=public \
    --logdir=/var/log/foundationdb \
    --public_address=auto:4500

files

# tree /var/db/foundationdb/ /usr/local/etc/foundationdb/ /var/log/foundationdb/
/var/db/foundationdb/
└── data
    └── 4500
        ├── coordination-0.fdq
        ├── coordination-1.fdq
        ├── log2-V_3_LS_2-6f5d01fbbca6ef21906798ec22911edf.sqlite
        ├── log2-V_3_LS_2-6f5d01fbbca6ef21906798ec22911edf.sqlite-wal
        ├── logqueue-V_3_LS_2-6f5d01fbbca6ef21906798ec22911edf-0.fdq
        ├── logqueue-V_3_LS_2-6f5d01fbbca6ef21906798ec22911edf-1.fdq
        ├── processId
        ├── storage-d528650a51442d78738b28cc972a9e61-0.fdq
        └── storage-d528650a51442d78738b28cc972a9e61-1.fdq
/usr/local/etc/foundationdb/
├── fdb.cluster
├── foundationdb.conf
└── foundationdb.conf.sample
/var/log/foundationdb/
├── trace.127.0.0.1.4500.1579564792.dAiyK8.0.1.xml
└── trace.127.0.0.1.4500.1579564944.GhdRBR.0.1.xml

switching to TLS

This doesn’t work yet I’m just documenting what I did

# cat /usr/local/etc/foundationdb/foundationdb.conf
## foundationdb.conf
##
## Configuration file for FoundationDB server processes
## Full documentation is available at
## https://apple.github.io/foundationdb/configuration.html#the-configuration-file

[fdbmonitor]
user = foundationdb
group = foundationdb

[general]
trace_format = json
restart_delay = 60
## by default, restart_backoff = restart_delay_reset_interval = restart_delay
# initial_restart_delay = 0
# restart_backoff = 60
# restart_delay_reset_interval = 60
cluster_file = /usr/local/etc/foundationdb/fdb.cluster
# delete_envvars =
# kill_on_configuration_change = true

## Default parameters for individual fdbserver processes
# https://apple.github.io/foundationdb/configuration.html#foundationdb-conf-fdbserver
[fdbserver]
command = /usr/local/bin/fdbserver
public_address = auto:$ID
listen_address = public
datadir = /var/db/foundationdb/data/$ID
logdir = /var/log/foundationdb
# logsize = 10MiB
# maxlogssize = 100MiB
# machine_id =
# datacenter_id =
# class =
# memory = 8GiB
# storage_memory = 1GiB
# cache_memory = 2GiB
# metrics_cluster =
# metrics_prefix =

# TLS settings
# https://apple.github.io/foundationdb/tls.html
# https://forums.foundationdb.org/t/solved-correct-setup-of-tls-for-foundationdb/1037/4
tls_certificate_file=/usr/local/etc/foundationdb/fdb.pem
tls_verify_peers=Check.Valid=0


## An individual fdbserver process with id 4500
## Parameters set here override defaults from the [fdbserver] section
[fdbserver.4500]

[backup_agent]
command = /usr/local/bin/backup_agent
logdir = /var/log/foundationdb

[backup_agent.1]

Then stop & restart DB with new TLS configs:

#### stop all the things
# service foundationdb stop
# cd /usr/local/etc/foundationdb
# openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout private.key -out cert.crt
# cat cert.crt private.key > fdb.pem
#### switch to TLS cluster
# cat /usr/local/etc/foundationdb/fdb.cluster
foundationdb:foundationdb@127.0.0.1:4600:tls
# ls -AFGhl /usr/local/etc/foundationdb/
total 23
-rw-r-----  1 dch  foundationdb   1.3K May  3 20:22 cert.crt
-rw-r-----  1 dch  foundationdb    45B May  8 16:36 fdb.cluster
-rw-r-----  1 dch  foundationdb   3.0K May  3 20:22 fdb.pem
-rw-r-----  1 dch  foundationdb   1.6K May  9 09:11 foundationdb.conf
-rw-r-----  1 dch  foundationdb   1.2K May  3 11:07 foundationdb.conf.sample
-rw-r-----  1 dch  foundationdb   1.7K May  3 20:21 private.key
# service foundationdb start# pgrep -ilfU foundationdb
82184 /usr/local/bin/fdbserver --cluster_file --datadir --listen_address --logdir --public_address --tls_certificate_file --tls_verify_peers --trace_format
62203 /usr/local/bin/backup_agent --cluster_file=/usr/local/etc/foundationdb/fdb.cluster --logdir=/var/log/foundationdb
# sockstat -46l | grep fdb
foundationdb fdbserver82184 17 tcp4 127.0.0.1:4500        *:*
# ldd /usr/local/bin/fdbserver
/usr/local/bin/fdbserver:
	libeio.so.1 => /usr/local/lib/libeio.so.1 (0x802a13000)
	librt.so.1 => /usr/lib/librt.so.1 (0x802a1f000)
	libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x802a28000)
	libdevstat.so.7 => /lib/libdevstat.so.7 (0x802a2e000)
	libssl.so.111 => /usr/lib/libssl.so.111 (0x802a37000)   <--- OpenSSL1.1.1 in base
	libcrypto.so.111 => /lib/libcrypto.so.111 (0x802acf000)
	libc++.so.1 => /usr/lib/libc++.so.1 (0x802d9e000)
	libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x802e6b000)
	libm.so.5 => /lib/libm.so.5 (0x802e8e000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x802ec0000)
	libthr.so.3 => /lib/libthr.so.3 (0x802eda000)
	libc.so.7 => /lib/libc.so.7 (0x802f06000)
	libelf.so.2 => /lib/libelf.so.2 (0x803312000)
	libkvm.so.7 => /lib/libkvm.so.7 (0x80332e000)

At this point the fdbserver process is not listening on TLS port and I’m not sure what I should do differently. TLS file permissions seem OK, & I’ve tried a bunch of fdbserver direct commandline invocations without luck.

Is the config correct in fdb.cluster and foundationdb.conf?

I’ll attach the json logs in the gist.

Dretch · December 12, 2020, 4:01pm

I had a similar issue, but with FoundationDB 6.2.28-1 on Ubuntu 18.

There was a helpful log message in the systemd log: service foundationdb status -n50

Dec 12 15:45:30 gareth fdbmonitor[16107]: LogGroup="default" Process="fdbserver.4500": Launching /usr/sbin/fdbserver (16233) for fdbserver.4500
Dec 12 15:45:30 gareth fdbmonitor[16107]: LogGroup="default" Process="fdbserver.4500": ERROR: TLS state of public address auto:4500 does not match in coordinator list.
Dec 12 15:45:30 gareth fdbmonitor[16107]: LogGroup="default" Process="fdbserver.4500": Process 16233 exited 1, restarting in 66 seconds

Based on this message, I changed a line in foundationdb.conf from public_address = auto:$ID to public_address = auto:$ID:tls … and then FoundationDB was able to startup.

dch · June 30, 2021, 10:51am

getting started with FoundationDB on FreeBSD

With some help from #CouchDB folk, I have this working. Note the :tls at the end, and adapted ports, and also the requirement for fdbcli to repeat the parameters already set in the config file. I suspect there is something wrong with parameter generation for the fdbserver process.

The databases/foundationdb port version was 6.3.13. There is also
a 7.0.0-master port tracking upstream work reasonably closely.

basic config

The port installs a basic /usr/local/etc/foundationdb/foundationdb.conf
but you need to set up the cluster file yourself. A starting point is:

# echo foundationdb:foundationdb@127.0.0.1:4500 \
    | tee /usr/local/etc/foundationdb/fdb.cluster

Alternatively, use a native TLS setup directly:

# echo foundationdb:foundationdb@127.0.0.1:4600:tls \
    | tee /usr/local/etc/foundationdb/fdb.cluster

Add users to the foundationdb group so that they can read the cluster
file, and access the DB, via pw groupmod -m foundationdb $USER.

set permissions

# echo foundationdb_enable=YES > /etc/rc.conf.d/foundationdb
# zfs create zroot/var/db/foundationdb
# chown -R foundationdb:foundationdb \
    /var/db/foundationdb /usr/local/etc/foundationdb
# chmod -R ug=rw,o-rwx,ug+X \
    /var/db/foundationdb /usr/local/etc/foundationdb

amend the config for TLS

If you didn’t use TLS above, the default port config is fine.

# /usr/local/etc/foundationdb/foundationdb.conf
## foundationdb.conf
##
## Configuration file for FoundationDB server processes
## Full documentation is available at
## https://apple.github.io/foundationdb/configuration.html#the-configuration-file

[fdbmonitor]
user = foundationdb
group = foundationdb

[general]
trace_format = json
restart_delay = 60
## by default, restart_backoff = restart_delay_reset_interval = restart_delay
# initial_restart_delay = 0
# restart_backoff = 60
# restart_delay_reset_interval = 60
cluster_file = /usr/local/etc/foundationdb/fdb.cluster
# delete_envvars =
# kill_on_configuration_change = true

## Default parameters for individual fdbserver processes
[fdbserver]
command = /usr/local/bin/fdbserver
public_address = auto:$ID:tls
listen_address = public
datadir = /var/db/foundationdb/data/$ID
logdir = /var/log/foundationdb
# logsize = 10MiB
# maxlogssize = 100MiB
# machine_id =
# datacenter_id =
# class =
# memory = 8GiB
# storage_memory = 1GiB
# cache_memory = 2GiB
# metrics_cluster =
# metrics_prefix =

# TLS settings
tls_ca_file=/usr/local/etc/foundationdb/cert.crt
tls_key_file=/usr/local/etc/foundationdb/fdb.pem
tls_certificate_file=/usr/local/etc/foundationdb/fdb.pem
# required if we are using self-signed certificates
tls_verify_peers=Check.Valid=0

# NB parameters set here override defaults from the [fdbserver] section
# each [fdbserver.XYZ] entry here will create an additional fdbserver
# process listening on that port, and inheriting existing TLS options.
[fdbserver.4600]

[backup_agent]
command = /usr/local/bin/backup_agent
logdir = /var/log/foundationdb

[backup_agent.1]

create TLS certificates as required

$ cd /usr/local/etc/foundationdb
$ openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
  -keyout private.key -out cert.crt
$ cat cert.crt private.key > fdb.pem

connect to the DB

As root, service foundationdb start and then run the rest as a normal
user. Again, note that the user requires read access to all of
/usr/local/etc/foundationdb which is unusual on FreeBSD, especially
for TLS certificates, and also write access to the fdb.cluster file.

If you are not running TLS, then skip the additional command-line
parameters.

$ fdbcli --tls_verify_peers Check.Valid=0 \
    --tls_key_file /usr/local/etc/foundationdb/private.key 

Using cluster file `fdb.cluster'.

The database is unavailable; type `status' for more information.

initialise the DB

Now we have an fdb shell, it’s time to initialise an in-memory DB for
testing, and write some data. If you run status a few times quickly
after the configure new single memory command, you’ll also see FDB
initialising and starting up - it’s not instant.

Welcome to the fdbcli. For help, type `help'.

fdb> configure new single memory
Database created

fdb> status

Using cluster file `fdb.cluster'.

Configuration:
  Redundancy mode        - single
  Storage engine         - memory-2
  Coordinators           - 1
  Usable Regions         - 1

Cluster:
  FoundationDB processes - 1
  Zones                  - 1
  Machines               - 1
  Memory availability    - 8.0 GB per process on machine with least available
  Retransmissions rate   - 0 Hz
  Fault Tolerance        - 0 machines
  Server time            - 06/30/21 10:30:57

Data:
  Replication health     - Healthy
  Moving data            - 0.000 GB
  Sum of key-value sizes - 0 MB
  Disk space used        - 105 MB

Operating space:
  Storage server         - 1.0 GB free on most full server
  Log server             - 4157.8 GB free on most full server

Workload:
  Read rate              - 6 Hz
  Write rate             - 0 Hz
  Transactions started   - 2 Hz
  Transactions committed - 0 Hz
  Conflict rate          - 0 Hz

Backup and DR:
  Running backups        - 0
  Running DRs            - 0

Client time: 06/30/21 10:30:57

fdb>

use the DB

fdb> writemode on
fdb> set poop emoji
Committed (219266389)
fdb> get poop
`poop' is `emoji'
fdb> set poop 💩  
Committed (252691029)
fdb> get poop
`poop' is `\xf0\x9f\x92\xa9'
fdb>

processes when using TLS

Note that in the fdbserver entry, parameters seem to be missing from
the supplied options. This is … unexpected.

# pgrep -ilU foundationdb
18843 backup_agent
6891 fdbmonitor
44182 fdbserver

# pgrep -ilf fdb
6981 /usr/local/bin/fdbmonitor \
    --conffile /usr/local/etc/foundationdb/foundationdb.conf \
    --lockfile /var/run/fdbmonitor.pid \
    --daemonize
44182 /usr/local/bin/fdbserver \
    --cluster_file \
    --datadir \
    --listen_address \
    --logdir \
    --public_address \
    --tls_ca_file \
    --tls_certificate_file \
    --tls_key_file \
    --tls_verify_peers \
    --trace_format
18843 /usr/local/bin/backup_agent \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --logdir=/var/log/foundationdb \
    --trace_format=json

processes without TLS

# pgrep -ilfU foundationdb
77073 /usr/local/bin/backup_agent \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --logdir=/var/log/foundationdb
94492 fdbmonitor \
    --conffile /usr/local/etc/foundationdb/foundationdb.conf \
    --lockfile /var/run/fdbmonitor.pid
72585 /usr/local/bin/fdbserver \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --datadir=/var/db/foundationdb/data/4500 \
    --listen_address=public \
    --logdir=/var/log/foundationdb \
    --public_address=auto:4500

files

#  tree -AF /usr/local/etc/foundationdb/ /var/log/foundationdb/ /var/run/fdbmonitor.pid /var/db/foundationdb/
/usr/local/etc/foundationdb/
├── cert.crt
├── fdb.cluster
├── fdb.pem
├── foundationdb.conf
├── foundationdb.conf.sample
└── private.key
/var/log/foundationdb/
├── trace.127.0.0.1.18843.1625048558.1QF9z6.0.1.json
└── trace.127.0.0.1.4600.1625048558.jSIe2n.0.1.json
/var/run/fdbmonitor.pid [error opening dir]
/var/db/foundationdb/
└── data/
    └── 4600/
        ├── coordination-0.fdq
        ├── coordination-1.fdq
        ├── log2-V_4_LS_2-2ab07e66d565347b440298a035eaa3fa.sqlite
        ├── log2-V_4_LS_2-2ab07e66d565347b440298a035eaa3fa.sqlite-wal
        ├── logqueue-V_4_LS_2-2ab07e66d565347b440298a035eaa3fa-0.fdq
        ├── logqueue-V_4_LS_2-2ab07e66d565347b440298a035eaa3fa-1.fdq
        ├── processId
        ├── storage-50936d3cd197ddd6b529413c8420d20c-0.fdq
        └── storage-50936d3cd197ddd6b529413c8420d20c-1.fdq

Enabling TLS on first install for single node setup

more notes

basic config pre-TLS

set permissions

initialise the DB

use the DB

processes

files

switching to TLS

getting started with FoundationDB on FreeBSD

basic config

set permissions

amend the config for TLS

create TLS certificates as required

connect to the DB

initialise the DB

use the DB

processes when using TLS

processes without TLS

files

notes and references