So I just installed FoundationDB on my iMac with the intent of learning more about it, but when I got to the step to check the status, I get the following:
fdb> status
WARNING: Long delay (Ctrl-C to interrupt)
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.
Unable to start default priority transaction after 5 seconds.
Unable to start batch priority transaction after 5 seconds.
Unable to retrieve all status information.
Configuration:
Redundancy mode - single
Storage engine - memory-2
Coordinators - 1
Cluster:
FoundationDB processes - 1
Zones - 1
Machines - 1
Memory availability - 14.6 GB per process on machine with least available
Fault Tolerance - 0 machines
Server time - 06/05/20 17:34:30
Data:
Replication health - Healthy
Moving data - 0.000 GB
Sum of key-value sizes - 0 MB
Disk space used - 105 MB
Operating space:
Storage server - 1.0 GB free on most full server
Log server - 0.0 GB free on most full server
Workload:
Read rate - 4 Hz
Write rate - 0 Hz
Transactions started - 2 Hz
Transactions committed - 0 Hz
Conflict rate - 0 Hz
Performance limited by server c6785a5ea9a79d5d: Log server running out of space (approaching 100MB limit).
Backup and DR:
Running backups - 0
Running DRs - 0
Client time: 06/05/20 17:34:22
I tried re-installing, deleting the data and running configure new single memory, but neither changed the warnings I was getting above. Does anyone know what the issue might be? I’m on the latest of everything (macOS 10.15.5, FoundationDB-6.2.20).
Funny you ask… I checked while trying to debug this, and noticed: “Hey, I still have 180 GB remaining, that should be plenty!”, not realizing that 180 GB out of 4 TB would infact be less than 5%. Is there a configuration I can fiddle with to change this to say 1% (I’m only testing after all)? Or do I need to find a way to reclaim 20GB somewhere?
Thanks @gaurav! I wonder why my error didn’t mention the 5% limit…
That said, it doesn’t seem to be doing anything. I’ve stopped/started via launchd, and reconfigured a new database (configure new single memory).
My conf file:
## foundationdb.conf
##
## Configuration file for FoundationDB server processes
## Full documentation is available at
## https://apple.github.io/foundationdb/configuration.html#the-configuration-file
[general]
restart_delay = 60
## by default, restart_backoff = restart_delay_reset_interval = restart_delay
# initial_restart_delay = 0
# restart_backoff = 60
# restart_delay_reset_interval = 60
cluster_file = /usr/local/etc/foundationdb/fdb.cluster
# kill_on_configuration_change = true
## Default parameters for individual fdbserver processes
[fdbserver]
command = /usr/local/libexec/fdbserver
public_address = auto:$ID
listen_address = public
datadir = /usr/local/foundationdb/data/$ID
logdir = /usr/local/foundationdb/logs
knob_min_free_space_ratio=0.001
# logsize = 10MiB
# maxlogssize = 100MiB
# machine_id =
# datacenter_id =
# class =
# memory = 8GiB
# storage_memory = 1GiB
# cache_memory = 2GiB
# metrics_cluster =
# metrics_prefix =
## An individual fdbserver process with id 4689
## Parameters set here override defaults from the [fdbserver] section
[fdbserver.4689]
[backup_agent]
command = /usr/local/foundationdb/backup_agent/backup_agent
logdir = /usr/local/foundationdb/logs
[backup_agent.1]
Still getting:
WARNING: Long delay (Ctrl-C to interrupt)
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.
Unable to start default priority transaction after 5 seconds.
Unable to start batch priority transaction after 5 seconds.
Unable to retrieve all status information.
Configuration:
Redundancy mode - single
Storage engine - memory-2
Coordinators - 1
Cluster:
FoundationDB processes - 1
Zones - 1
Machines - 1
Memory availability - 10.4 GB per process on machine with least available
Fault Tolerance - 0 machines
Server time - 06/06/20 11:51:40
Data:
Replication health - Healthy
Moving data - 0.000 GB
Sum of key-value sizes - 0 MB
Disk space used - 105 MB
Operating space:
Storage server - 1.0 GB free on most full server
Log server - 0.0 GB free on most full server
Workload:
Read rate - 4 Hz
Write rate - 0 Hz
Transactions started - 1 Hz
Transactions committed - 0 Hz
Conflict rate - 0 Hz
Performance limited by server 0e1481eec1a7ec3c: Log server running out of space (approaching 100MB limit).
Backup and DR:
Running backups - 0
Running DRs - 0
Client time: 06/06/20 11:51:32
Alas, creating a few 40 GB files, duplicating them, then deleting them seems to have made enough space available… (260 GB now free, which I think is why I’m getting the 60GB for the Log Server (200 GB is “5%”, even though I set the percentage way lower)
Should I be worried that the Storage server Space here is just 1 GB?
Operating space:
Storage server - 1.0 GB free on most full server
Log server - 60.5 GB free on most full server
Fdb installs with default as Memory storage engine, that uses 1 gb by default. You should probably switch to disk storage engine. Typing this command on fdbcli should do that - configure ssd (please see the help on this command)
That makes sense. Thanks for all the help! (I’m still concerned that the 5% part didn’t seem to change, but I’ll be back when it actually becomes an issue )
I’m guessing maybe the knob name is changed. I didn’t see the knob “min_free_space_ratio” in the source code. You may want to try MIN_AVAILABLE_SPACE_RATIO knob by setting knob_min_available_space_ratioe.
The very first time I tried using fdb on my machine (2014-ish), I got the exact same thing happen, while trying to write a binding over the C library and I spent almost a week rewriting everything with no luck until I could get help from the team. When they told me to check the free disk space, I removed a random large file and the cluster instantly started to respond >_<
This has also happened to almost anyone on my team at one point, and that’s usually one of the first thing I tell people to check when installing on a new dev machine. I wish there was a way to setup fdb in a “developper mode” that would be tweaked a bit differently than “production mode”.
Since there are general warnings about not using the macOS one for production, I would assume that it should be set to “development” mode automatically during install? Combined with macOS’s tendency to have reclaimable space, this seems like it would be the right default. From what I understand, it seems like fdb wants the whole system to itself anyways, which will never be the case on macOS
)