Work around "Storage server running out of space (approaching 5% limit)" on your developer machine

Using FoundationDB
1

This one hits me regularly enough, so here is a quick tip if you are using fdb locally on your dev machine, which is running perpetually in a low free disk space state (less than 5% of the total capacity).

Using cluster file `C:\ProgramData\foundationdb\fdb.cluster'.

The database is available, but has issues (type 'status' for more information).

Welcome to the fdbcli. For help, type `help'.
fdb> status

WARNING: Long delay (Ctrl-C to interrupt)

Using cluster file `C:\ProgramData\foundationdb\fdb.cluster'.

Unable to start default priority transaction after 5 seconds.

Unable to start batch priority transaction after 5 seconds.

Unable to retrieve all status information.
...
Workload:
  Read rate              - 4 Hz
  Write rate             - 0 Hz
  Transactions started   - 2 Hz
  Transactions committed - 0 Hz
  Conflict rate          - 0 Hz
  Performance limited by server 208cfdeba9ddbf45: Storage server running out of space (approaching 5% limit).

The fdbserver is refusing to start new transactions as long as you are below 5% freespace by default. If you have 500GB/1TB ssd drive, this means that it triggers under 25GB/50GB free space.

You can work around this by changing a knob in your foundationdb.conf file to lower the threshold (to something low, like 0.1%).

DON’T DO THIS IN PRODUCTION (unless you know what you are doing!)

Simply edit foundationdb.conf and add a knob_min_free_space_ratio entry in the [fdbserver] section:

...
## Default parameters for individual fdbserver processes
[fdbserver]
public_address = auto:$ID
listen_address = public
parentpid = $PID
command=...
datadir=...
logdir=...
knob_min_free_space_ratio=0.001

Simply saving the file should restart your fdbserver process, and unblock you for good, or at least until you dip down below less than 500MB free space total.

3 Likes
2

The knob now seems to be knob_min_available_space_ratio

2 Likes
3

It was indeed changed recently and looking at the git timeline, it looks like it was done starting in v6.2.16.

I did not see any issue on my machine because in the mean time I added dedicated drives using one of these puppies and moved the data files out of my main partition.

Unfortunately, I cannot edit the original message!

So from 6.2.16 and up, it should look like:

...
## Default parameters for individual fdbserver processes
[fdbserver]
public_address = auto:$ID
listen_address = public
parentpid = $PID
command=...
datadir=...
logdir=...
knob_min_available_space_ratio=0.001

And again, it should only be done on a local development machine, NOT on production servers because it could impact how replication works!