# Enabling TLS on first install for single node setup

**URL:** <https://forums.foundationdb.org/t/enabling-tls-on-first-install-for-single-node-setup/2111>\
**Category:** Running FoundationDB\
**Created:** [May 9, 2020, 9:32am UTC](https://forums.foundationdb.org/t/enabling-tls-on-first-install-for-single-node-setup/2111 "2020-05-09T09:32:49Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dch](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/dch/32/473_2.png) [@dch](https://forums.foundationdb.org/u/dch)\
**Post date:** [May 9, 2020, 9:32am UTC](https://forums.foundationdb.org/t/enabling-tls-on-first-install-for-single-node-setup/2111/1 "2020-05-09T09:32:50Z")

</div>

I’m unable to decipher what goes in `foundationdb.conf` to have a TLS by  
default cluster. I’m hoping to provide the FreeBSD port by default with TLS  
enabled, and enough instructions for an admin to deploy a single node  
self-signed cert setup, to get started fast.

- TLDR: fdbserver doesn’t listen on 4600 on startup
- logs & configs: [fdb debug failed TLS startup https://forums.foundationdb.org/t/enabling-tls-on-first-install-for-single-node-setup/2111 · GitHub](https://gist.github.com/dch/3fe08fdb5f10d1c653c80b6eae2f4889)

## more notes

Here’s my setup (FreeBSD, recent master, single node), compiled against  
FreeBSD 13.0-CURRENT’s openssl in base, as a dynamic library (cmake  
tweaked to allow that).

## basic config pre-TLS

```auto
# echo foundationdb:foundationdb@127.0.0.1:4500 \
    | tee /usr/local/etc/foundationdb/fdb.cluster

```

Add users to the foundationdb group so that they can read the cluster  
file, and access the DB, via `pw groupmod -m foundationdb $USER`.

## set permissions

```auto
# touch /var/run/fdbmonitor.pid
# echo foundationdb_enable=YES > /etc/rc.conf.d/foundationdb
# zfs create zroot/var/db/foundationdb
# chown -R foundationdb:foundationdb \
    /usr/local/etc/foundationdb \
    /var/log/foundationdb \
    /var/db/foundationdb \
    /var/run/fdbmonitor.pid
# chmod -R u=rw,g=r,o-rwx,ug+X \
    /usr/local/etc/foundationdb \
    /var/log/foundationdb \
    /var/db/foundationdb \
    /var/run/fdbmonitor.pid

```

## initialise the DB

As root, `service foundationdb start` and then run the rest as a normal  
user:

```auto
$ fdbcli --exec "configure new single memory"
$ fdbcli
fdbcli
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.

The database is available.

Welcome to the fdbcli. For help, type `help'.
fdb> status

Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.

Configuration:
  Redundancy mode - single
  Storage engine - memory-2
  Coordinators - 1

Cluster:
  FoundationDB processes - 1
  Zones - 1
  Machines - 1
  Memory availability - 25.5 GB per process on machine with least available
  Fault Tolerance - 0 machines
  Server time - 01/21/20 00:07:14

Data:
  Replication health - Healthy
  Moving data - 0.000 GB
  Sum of key-value sizes - 0 MB
  Disk space used - 105 MB

Operating space:
  Storage server - 1.0 GB free on most full server
  Log server - 226.0 GB free on most full server

Workload:
  Read rate - 7 Hz
  Write rate - 0 Hz
  Transactions started - 3 Hz
  Transactions committed - 0 Hz
  Conflict rate - 0 Hz

Backup and DR:
  Running backups - 0
  Running DRs - 0

Client time: 01/21/20 00:07:14

fdb> 

```

## use the DB

```auto
$ fdbserver --version
FoundationDB 6.3 (v6.3.0)
source version
protocol fdb00b063010001

$ fdbcli --exec "writemode on; set poop emoji"
>>> writemode on
>>> set poop emoji
Committed (1366086432)

$ fdbcli --exec "get poop emoji"
`poop' is `emoji'

$ fdbcli --exec "writemode on; set poop emojis"
>>> writemode on
>>> set poop emojis
Committed (1382502961)

$ fdbcli --exec "get poop emoji"
`poop' is `emojis'

```

## processes

```auto
# pgrep -ilU foundationdb
77073 backup_agent
94492 fdbmonitor
72585 fdbserver

# pgrep -ilfU foundationdb
77073 /usr/local/bin/backup_agent \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --logdir=/var/log/foundationdb
94492 fdbmonitor \
    --conffile /usr/local/etc/foundationdb/foundationdb.conf \
    --lockfile /var/run/fdbmonitor.pid
72585 /usr/local/bin/fdbserver \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --datadir=/var/db/foundationdb/data/4500 \
    --listen_address=public \
    --logdir=/var/log/foundationdb \
    --public_address=auto:4500

```

## files

```auto
# tree /var/db/foundationdb/ /usr/local/etc/foundationdb/ /var/log/foundationdb/
/var/db/foundationdb/
└── data
    └── 4500
        ├── coordination-0.fdq
        ├── coordination-1.fdq
        ├── log2-V_3_LS_2-6f5d01fbbca6ef21906798ec22911edf.sqlite
        ├── log2-V_3_LS_2-6f5d01fbbca6ef21906798ec22911edf.sqlite-wal
        ├── logqueue-V_3_LS_2-6f5d01fbbca6ef21906798ec22911edf-0.fdq
        ├── logqueue-V_3_LS_2-6f5d01fbbca6ef21906798ec22911edf-1.fdq
        ├── processId
        ├── storage-d528650a51442d78738b28cc972a9e61-0.fdq
        └── storage-d528650a51442d78738b28cc972a9e61-1.fdq
/usr/local/etc/foundationdb/
├── fdb.cluster
├── foundationdb.conf
└── foundationdb.conf.sample
/var/log/foundationdb/
├── trace.127.0.0.1.4500.1579564792.dAiyK8.0.1.xml
└── trace.127.0.0.1.4500.1579564944.GhdRBR.0.1.xml

```

## switching to TLS

> This doesn’t work yet I’m just documenting what I did

```auto
# cat /usr/local/etc/foundationdb/foundationdb.conf
## foundationdb.conf
##
## Configuration file for FoundationDB server processes
## Full documentation is available at
## https://apple.github.io/foundationdb/configuration.html#the-configuration-file

[fdbmonitor]
user = foundationdb
group = foundationdb

[general]
trace_format = json
restart_delay = 60
## by default, restart_backoff = restart_delay_reset_interval = restart_delay
# initial_restart_delay = 0
# restart_backoff = 60
# restart_delay_reset_interval = 60
cluster_file = /usr/local/etc/foundationdb/fdb.cluster
# delete_envvars =
# kill_on_configuration_change = true

## Default parameters for individual fdbserver processes
# https://apple.github.io/foundationdb/configuration.html#foundationdb-conf-fdbserver
[fdbserver]
command = /usr/local/bin/fdbserver
public_address = auto:$ID
listen_address = public
datadir = /var/db/foundationdb/data/$ID
logdir = /var/log/foundationdb
# logsize = 10MiB
# maxlogssize = 100MiB
# machine_id =
# datacenter_id =
# class =
# memory = 8GiB
# storage_memory = 1GiB
# cache_memory = 2GiB
# metrics_cluster =
# metrics_prefix =

# TLS settings
# https://apple.github.io/foundationdb/tls.html
# https://forums.foundationdb.org/t/solved-correct-setup-of-tls-for-foundationdb/1037/4
tls_certificate_file=/usr/local/etc/foundationdb/fdb.pem
tls_verify_peers=Check.Valid=0

## An individual fdbserver process with id 4500
## Parameters set here override defaults from the [fdbserver] section
[fdbserver.4500]

[backup_agent]
command = /usr/local/bin/backup_agent
logdir = /var/log/foundationdb

[backup_agent.1]

```

Then stop & restart DB with new TLS configs:

```auto
#### stop all the things
# service foundationdb stop
# cd /usr/local/etc/foundationdb
# openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout private.key -out cert.crt
# cat cert.crt private.key > fdb.pem
#### switch to TLS cluster
# cat /usr/local/etc/foundationdb/fdb.cluster
foundationdb:foundationdb@127.0.0.1:4600:tls
# ls -AFGhl /usr/local/etc/foundationdb/
total 23
-rw-r----- 1 dch foundationdb 1.3K May 3 20:22 cert.crt
-rw-r----- 1 dch foundationdb 45B May 8 16:36 fdb.cluster
-rw-r----- 1 dch foundationdb 3.0K May 3 20:22 fdb.pem
-rw-r----- 1 dch foundationdb 1.6K May 9 09:11 foundationdb.conf
-rw-r----- 1 dch foundationdb 1.2K May 3 11:07 foundationdb.conf.sample
-rw-r----- 1 dch foundationdb 1.7K May 3 20:21 private.key
# service foundationdb start# pgrep -ilfU foundationdb
82184 /usr/local/bin/fdbserver --cluster_file --datadir --listen_address --logdir --public_address --tls_certificate_file --tls_verify_peers --trace_format
62203 /usr/local/bin/backup_agent --cluster_file=/usr/local/etc/foundationdb/fdb.cluster --logdir=/var/log/foundationdb
# sockstat -46l | grep fdb
foundationdb fdbserver82184 17 tcp4 127.0.0.1:4500 *:*
# ldd /usr/local/bin/fdbserver
/usr/local/bin/fdbserver:
	libeio.so.1 => /usr/local/lib/libeio.so.1 (0x802a13000)
	librt.so.1 => /usr/lib/librt.so.1 (0x802a1f000)
	libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x802a28000)
	libdevstat.so.7 => /lib/libdevstat.so.7 (0x802a2e000)
	libssl.so.111 => /usr/lib/libssl.so.111 (0x802a37000) <--- OpenSSL1.1.1 in base
	libcrypto.so.111 => /lib/libcrypto.so.111 (0x802acf000)
	libc++.so.1 => /usr/lib/libc++.so.1 (0x802d9e000)
	libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x802e6b000)
	libm.so.5 => /lib/libm.so.5 (0x802e8e000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x802ec0000)
	libthr.so.3 => /lib/libthr.so.3 (0x802eda000)
	libc.so.7 => /lib/libc.so.7 (0x802f06000)
	libelf.so.2 => /lib/libelf.so.2 (0x803312000)
	libkvm.so.7 => /lib/libkvm.so.7 (0x80332e000)

```

At this point the fdbserver process is not listening on TLS port and I’m not sure what I should do differently. TLS file permissions seem OK, & I’ve tried a bunch of `fdbserver` direct commandline invocations without luck.

Is the config correct in `fdb.cluster` and `foundationdb.conf`?

I’ll attach the json logs in the gist.

---

<div class="post-metadata">

**Author:** ![Dretch](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/dretch/32/1000_2.png) [@Dretch](https://forums.foundationdb.org/u/Dretch)\
**Post date:** [December 12, 2020, 4:01pm UTC](https://forums.foundationdb.org/t/enabling-tls-on-first-install-for-single-node-setup/2111/2 "2020-12-12T16:01:11Z")

</div>

I had a similar issue, but with FoundationDB 6.2.28-1 on Ubuntu 18.

There was a helpful log message in the systemd log: `service foundationdb status -n50`

```
Dec 12 15:45:30 gareth fdbmonitor[16107]: LogGroup="default" Process="fdbserver.4500": Launching /usr/sbin/fdbserver (16233) for fdbserver.4500
Dec 12 15:45:30 gareth fdbmonitor[16107]: LogGroup="default" Process="fdbserver.4500": ERROR: TLS state of public address auto:4500 does not match in coordinator list.
Dec 12 15:45:30 gareth fdbmonitor[16107]: LogGroup="default" Process="fdbserver.4500": Process 16233 exited 1, restarting in 66 seconds

```

Based on this message, I changed a line in foundationdb.conf from `public_address = auto:$ID` to `public_address = auto:$ID:tls` … and then FoundationDB was able to startup.

---

<div class="post-metadata">

**Author:** ![dch](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/dch/32/473_2.png) [@dch](https://forums.foundationdb.org/u/dch)\
**Post date:** [June 30, 2021, 10:51am UTC](https://forums.foundationdb.org/t/enabling-tls-on-first-install-for-single-node-setup/2111/3 "2021-06-30T10:51:48Z")

</div>

# getting started with FoundationDB on FreeBSD

With some help from #CouchDB folk, I have this working. Note the `:tls` at the end, and adapted ports, and _also_ the requirement for fdbcli to repeat the parameters already set in the config file. I suspect there is something wrong with parameter generation for the fdbserver process.

The `databases/foundationdb` port version was `6.3.13`. There is also  
a [7.0.0-master](https://freshports.org/databases/foundationdb-devel) port tracking upstream work reasonably closely.

## basic config

The port installs a basic `/usr/local/etc/foundationdb/foundationdb.conf`  
but you need to set up the cluster file yourself. A starting point is:

```auto
# echo foundationdb:foundationdb@127.0.0.1:4500 \
    | tee /usr/local/etc/foundationdb/fdb.cluster

```

Alternatively, use a `native TLS` setup directly:

```auto
# echo foundationdb:foundationdb@127.0.0.1:4600:tls \
    | tee /usr/local/etc/foundationdb/fdb.cluster

```

Add users to the foundationdb group so that they can read the cluster  
file, and access the DB, via `pw groupmod -m foundationdb $USER`.

## set permissions

```auto
# echo foundationdb_enable=YES > /etc/rc.conf.d/foundationdb
# zfs create zroot/var/db/foundationdb
# chown -R foundationdb:foundationdb \
    /var/db/foundationdb /usr/local/etc/foundationdb
# chmod -R ug=rw,o-rwx,ug+X \
    /var/db/foundationdb /usr/local/etc/foundationdb

```

## amend the config for TLS

If you didn’t use TLS above, the default port config is fine.

```auto
# /usr/local/etc/foundationdb/foundationdb.conf
## foundationdb.conf
##
## Configuration file for FoundationDB server processes
## Full documentation is available at
## https://apple.github.io/foundationdb/configuration.html#the-configuration-file

[fdbmonitor]
user = foundationdb
group = foundationdb

[general]
trace_format = json
restart_delay = 60
## by default, restart_backoff = restart_delay_reset_interval = restart_delay
# initial_restart_delay = 0
# restart_backoff = 60
# restart_delay_reset_interval = 60
cluster_file = /usr/local/etc/foundationdb/fdb.cluster
# delete_envvars =
# kill_on_configuration_change = true

## Default parameters for individual fdbserver processes
[fdbserver]
command = /usr/local/bin/fdbserver
public_address = auto:$ID:tls
listen_address = public
datadir = /var/db/foundationdb/data/$ID
logdir = /var/log/foundationdb
# logsize = 10MiB
# maxlogssize = 100MiB
# machine_id =
# datacenter_id =
# class =
# memory = 8GiB
# storage_memory = 1GiB
# cache_memory = 2GiB
# metrics_cluster =
# metrics_prefix =

# TLS settings
tls_ca_file=/usr/local/etc/foundationdb/cert.crt
tls_key_file=/usr/local/etc/foundationdb/fdb.pem
tls_certificate_file=/usr/local/etc/foundationdb/fdb.pem
# required if we are using self-signed certificates
tls_verify_peers=Check.Valid=0

# NB parameters set here override defaults from the [fdbserver] section
# each [fdbserver.XYZ] entry here will create an additional fdbserver
# process listening on that port, and inheriting existing TLS options.
[fdbserver.4600]

[backup_agent]
command = /usr/local/bin/backup_agent
logdir = /var/log/foundationdb

[backup_agent.1]

```

## create TLS certificates as required

```auto
$ cd /usr/local/etc/foundationdb
$ openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
  -keyout private.key -out cert.crt
$ cat cert.crt private.key > fdb.pem

```

## connect to the DB

As root, `service foundationdb start` and then run the rest as a normal  
user. Again, note that the user requires _read_ access to all of  
`/usr/local/etc/foundationdb` which is unusual on FreeBSD, especially  
for TLS certificates, and also _write_ access to the `fdb.cluster` file.

If you are not running TLS, then skip the additional command-line  
parameters.

```auto
$ fdbcli --tls_verify_peers Check.Valid=0 \
    --tls_key_file /usr/local/etc/foundationdb/private.key 

Using cluster file `fdb.cluster'.

The database is unavailable; type `status' for more information.

```

## initialise the DB

Now we have an fdb shell, it’s time to initialise an in-memory DB for  
testing, and write some data. If you run status a few times quickly  
after the `configure new single memory` command, you’ll also see FDB  
initialising and starting up - it’s not instant.

```auto
Welcome to the fdbcli. For help, type `help'.

fdb> configure new single memory
Database created

fdb> status

Using cluster file `fdb.cluster'.

Configuration:
  Redundancy mode - single
  Storage engine - memory-2
  Coordinators - 1
  Usable Regions - 1

Cluster:
  FoundationDB processes - 1
  Zones - 1
  Machines - 1
  Memory availability - 8.0 GB per process on machine with least available
  Retransmissions rate - 0 Hz
  Fault Tolerance - 0 machines
  Server time - 06/30/21 10:30:57

Data:
  Replication health - Healthy
  Moving data - 0.000 GB
  Sum of key-value sizes - 0 MB
  Disk space used - 105 MB

Operating space:
  Storage server - 1.0 GB free on most full server
  Log server - 4157.8 GB free on most full server

Workload:
  Read rate - 6 Hz
  Write rate - 0 Hz
  Transactions started - 2 Hz
  Transactions committed - 0 Hz
  Conflict rate - 0 Hz

Backup and DR:
  Running backups - 0
  Running DRs - 0

Client time: 06/30/21 10:30:57

fdb> 

```

## use the DB

```auto
fdb> writemode on
fdb> set poop emoji
Committed (219266389)
fdb> get poop
`poop' is `emoji'
fdb> set poop 💩  
Committed (252691029)
fdb> get poop
`poop' is `\xf0\x9f\x92\xa9'
fdb> 

```

## processes when using TLS

Note that in the `fdbserver` entry, parameters seem to be missing from  
the supplied options. This is … unexpected.

```auto
# pgrep -ilU foundationdb
18843 backup_agent
6891 fdbmonitor
44182 fdbserver

# pgrep -ilf fdb
6981 /usr/local/bin/fdbmonitor \
    --conffile /usr/local/etc/foundationdb/foundationdb.conf \
    --lockfile /var/run/fdbmonitor.pid \
    --daemonize
44182 /usr/local/bin/fdbserver \
    --cluster_file \
    --datadir \
    --listen_address \
    --logdir \
    --public_address \
    --tls_ca_file \
    --tls_certificate_file \
    --tls_key_file \
    --tls_verify_peers \
    --trace_format
18843 /usr/local/bin/backup_agent \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --logdir=/var/log/foundationdb \
    --trace_format=json

```

## processes without TLS

```auto
# pgrep -ilfU foundationdb
77073 /usr/local/bin/backup_agent \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --logdir=/var/log/foundationdb
94492 fdbmonitor \
    --conffile /usr/local/etc/foundationdb/foundationdb.conf \
    --lockfile /var/run/fdbmonitor.pid
72585 /usr/local/bin/fdbserver \
    --cluster_file=/usr/local/etc/foundationdb/fdb.cluster \
    --datadir=/var/db/foundationdb/data/4500 \
    --listen_address=public \
    --logdir=/var/log/foundationdb \
    --public_address=auto:4500

```

## files

```auto
# tree -AF /usr/local/etc/foundationdb/ /var/log/foundationdb/ /var/run/fdbmonitor.pid /var/db/foundationdb/
/usr/local/etc/foundationdb/
├── cert.crt
├── fdb.cluster
├── fdb.pem
├── foundationdb.conf
├── foundationdb.conf.sample
└── private.key
/var/log/foundationdb/
├── trace.127.0.0.1.18843.1625048558.1QF9z6.0.1.json
└── trace.127.0.0.1.4600.1625048558.jSIe2n.0.1.json
/var/run/fdbmonitor.pid [error opening dir]
/var/db/foundationdb/
└── data/
    └── 4600/
        ├── coordination-0.fdq
        ├── coordination-1.fdq
        ├── log2-V_4_LS_2-2ab07e66d565347b440298a035eaa3fa.sqlite
        ├── log2-V_4_LS_2-2ab07e66d565347b440298a035eaa3fa.sqlite-wal
        ├── logqueue-V_4_LS_2-2ab07e66d565347b440298a035eaa3fa-0.fdq
        ├── logqueue-V_4_LS_2-2ab07e66d565347b440298a035eaa3fa-1.fdq
        ├── processId
        ├── storage-50936d3cd197ddd6b529413c8420d20c-0.fdq
        └── storage-50936d3cd197ddd6b529413c8420d20c-1.fdq

```

## notes and references