The lack clear documentation, flakyness with the intel images, and lack of an arm image makes working on a mac with the docker images quite a challenging proposition (the natives installed packages work fine of course). Is there anything we could do to help here?
Creating documentation for the MacOS setup would definitely be appreciated. For the arm images, I’m not sure if there are any plans to create arm images in the short term, maybe @ammolitor has some knowledge about that.
To follow up from @newhook 's comments above, we were finally able to get this going by building arm binaries/packages using some of the work from the Lionrock repo mentioned above. I was able to run the compilation with QEMU on amd (but it took a REALLY long time) and to build the final ‘runnable’ image, for whatever reason I actually needed and arm64 runner to install the pkg (was getting some IO exception when doing dpkg -i) so we had to use self-hosted runners to actually get the whole job done.
I now am able to build multiarch (arm/amd) images for both. Code is here:
Thanks @panghy for the Lionrock trail you left.
Hi all.
I’ve been scouring these forums, looking for any possible way to run FoundationDB on my MacBook Pro (M1 Pro chip) and have come up completely blank 
I’ve tried downloading and installing the most recent arm64 client+server packages (both [odd] FoundationDB-7.3.27_arm64.pkg and [even] FoundationDB-7.3.26_arm64.pkg). I tried the x86_64 versions, too – no dice. Same with older versions of the arm64 pkg files.
I also tried downloading and compiling from source. I was able to produce working binaries, but no luck actually setting up a working server.
The closest I get is installing the native packages. Running fdbcli and issuing a status command immediately after installation yields at least a bit of information:
% fdbcli
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.
The database is available, but has issues (type 'status' for more information).
Welcome to the fdbcli. For help, type `help'.
fdb> status
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.
Unable to start batch priority transaction after 5 seconds.
Configuration:
Redundancy mode - single
Storage engine - memory
Log engine - ssd-2
Encryption at-rest - disabled
Coordinators - 1
Desired Commit Proxies - 3
Desired GRV Proxies - 1
Desired Resolvers - 1
Desired Logs - 3
Usable Regions - 1
Cluster:
FoundationDB processes - 1
Zones - 1
Machines - 1
Memory availability - 0.2 GB per process on machine with least available
>>>>> (WARNING: 4.0 GB recommended) <<<<<
Fault Tolerance - 0 machines
Server time - 12/05/23 22:45:54
Data:
Replication health - (Re)initializing automatic data distribution
Moving data - unknown (initializing)
Sum of key-value sizes - unknown
Disk space used - 105 MB
Operating space:
Storage server - 1.0 GB free on most full server
Log server - 0.0 GB free on most full server
Workload:
Read rate - 22 Hz
Write rate - 0 Hz
Transactions started - 8 Hz
Transactions committed - 0 Hz
Conflict rate - 0 Hz
Performance limited by server 61b2c4126c5a501e: Log server MVCC memory.
Backup and DR:
Running backups - 0
Running DRs - 0
Client time: 12/05/23 22:45:49
Attempting to run the status command again ends in abject failure:
fdb> status
WARNING: Long delay (Ctrl-C to interrupt)
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.
Timed out fetching cluster status.
Configuration:
Redundancy mode - unknown
Storage engine - unknown
Log engine - unknown
Encryption at-rest - disabled
Coordinators - unknown
Usable Regions - unknown
Cluster:
FoundationDB processes - unknown
Zones - unknown
Machines -
Machines - unknown
Data:
Replication health - unknown
Moving data - unknown
Sum of key-value sizes - unknown
Disk space used - unknown
Operating space:
Unable to retrieve operating space status
Workload:
Read rate - unknown
Write rate - unknown
Transactions started - unknown
Transactions committed - unknown
Conflict rate - unknown
Backup and DR:
Running backups - 0
Running DRs - 0
Client time: 12/05/23 22:47:14
If I sudo launchctl stop com.foundationdb.fdbmonitor then execute fdbcli and a status command, I can eke out a bit more information:
fdb> status
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.
Unable to start batch priority transaction after 5 seconds.
Unable to retrieve all status information.
Configuration:
Redundancy mode - single
Storage engine - memory
Log engine - ssd-2
Encryption at-rest - disabled
Coordinators - 1
Desired Commit Proxies - 3
Desired GRV Proxies - 1
Desired Resolvers - 1
Desired Logs - 3
Usable Regions - 1
Cluster:
FoundationDB processes - 1
Zones - unknown
Machines - 0
Fault Tolerance - 0 machines
Server time - 12/05/23 22:49:58
Data:
Replication health - (Re)initializing automatic data distribution
Moving data - unknown (initializing)
Sum of key-value sizes - unknown
Disk space used - 105 MB
Operating space:
Storage server - 1.0 GB free on most full server
Log server - 0.0 GB free on most full server
Workload:
Read rate - 38 Hz
Write rate - 5 Hz
Transactions started - 15 Hz
Transactions committed - 3 Hz
Conflict rate - 1 Hz
Performance limited by server ba3841f3f7667ac4: Log server MVCC memory.
Backup and DR:
Running backups - 0
Running DRs - 0
Client time: 12/05/23 22:49:52
What’s weird (to me) is that “operating space” is being reported as 1.0/0.0 GB free; I’ve definitely got more free disk space than that! Same with “memory availability”; this machine has a lot more than 0.2 GB available!
I didn’t spot any smoking guns in the logs, but would be happy to post anything (and everything!) that could help me (and likely others) run FoundationDB without hassle on the otherwise-amazing MacBook Pros with M-series chips!
(I’m running macOS Sonoma 14.1.2)
I’ve tried downloading and installing the most recent arm64 client+server packages (both [odd ]
FoundationDB-7.3.27_arm64.pkgand [even ]FoundationDB-7.3.26_arm64.pkg). I tried the x86_64 versions, too – no dice. Same with older versions of the arm64 pkg files.
I just tried the FoundationDB-7.3.27_arm64.pkg package and it worked fine for me (M1 MBP and same OS version).
The only thing that I changed was the amount of memory that should be used:
cat /usr/local/etc/foundationdb/foundationdb.conf
## foundationdb.conf
##
## Configuration file for FoundationDB server processes
## Full documentation is available at
## https://apple.github.io/foundationdb/configuration.html#the-configuration-file
[general]
restart-delay = 60
## by default, restart-backoff = restart-delay-reset-interval = restart-delay
# initial-restart-delay = 0
# restart-backoff = 60
# restart-delay-reset-interval = 60
cluster-file = /usr/local/etc/foundationdb/fdb.cluster
# kill-on-configuration-change = true
## Default parameters for individual fdbserver processes
[fdbserver]
command = /usr/local/libexec/fdbserver
public_address = auto:$ID
listen_address = public
datadir = /usr/local/foundationdb/data/$ID
logdir = /usr/local/foundationdb/logs
# logsize = 10MiB
# maxlogssize = 100MiB
# machine-id =
# datacenter-id =
# class =
memory = 8GiB # <<<<< Uncommented this line
# storage-memory = 1GiB
# cache-memory = 2GiB
# metrics-cluster =
# metrics-prefix =
## An individual fdbserver process with id 4689
## Parameters set here override defaults from the [fdbserver] section
[fdbserver.4689]
[backup_agent]
command = /usr/local/foundationdb/backup_agent/backup_agent
logdir = /usr/local/foundationdb/logs
[backup_agent.1]
After the change in the config file I restarted the fdbsevrer process: sudo pkill fdbserver. I haven’t done any further testing but the status result works and I can write and read from the cluster:
fdb> status details
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.
Configuration:
Redundancy mode - single
Storage engine - memory
Log engine - ssd-2
Encryption at-rest - disabled
Coordinators - 1
Desired Commit Proxies - 3
Desired GRV Proxies - 1
Desired Resolvers - 1
Desired Logs - 3
Usable Regions - 1
Cluster:
FoundationDB processes - 1
Zones - 1
Machines - 1
Memory availability - 4.9 GB per process on machine with least available
Fault Tolerance - 0 machines
Server time - 12/06/23 12:02:17
Data:
Replication health - Healthy
Moving data - 0.000 GB
Sum of key-value sizes - 0 MB
Disk space used - 105 MB
Operating space:
Storage server - 1.0 GB free on most full server
Log server - 1201.1 GB free on most full server
Workload:
Read rate - 23 Hz
Write rate - 0 Hz
Transactions started - 9 Hz
Transactions committed - 0 Hz
Conflict rate - 0 Hz
Backup and DR:
Running backups - 0
Running DRs - 0
Process performance details:
127.0.0.1:4689 ( 3% cpu; 18% machine; 0.000 Gbps; 3% disk IO; 0.1 GB / 4.9 GB RAM )
Coordination servers:
127.0.0.1:4689 (reachable)
Client time: 12/06/23 12:02:17
fdb> writemode on
fdb> set test test
Committed (552326904)
fdb> get test
`test' is `test'
What’s weird (to me) is that “operating space” is being reported as 1.0/0.0 GB free; I’ve definitely got more free disk space than that! Same with “memory availability”; this machine has a lot more than 0.2 GB available!
This could be some misreporting of some libraries.
@johscheuer thanks for hopping in so quickly to help a hapless internet stranger. I really (really!) wish I was on the other side of the “works for me” divide in this case 
Here’s what I’ve tried:
First, I totally uninstalled FoundationDB:
sudo /usr/local/foundationdb/uninstall-FoundationDB.sh
sudo rm -rf /usr/local/foundationdb
sudo rm -rf /usr/local/etc/foundationdb
Then, for good measure, restarted my machine.
I installed FoundationDB (server and client) from FoundationDB-7.3.27_arm64.pkg. I updated my config file per your recommendation:
% cat /usr/local/etc/foundationdb/foundationdb.conf
## foundationdb.conf
##
## Configuration file for FoundationDB server processes
## Full documentation is available at
## https://apple.github.io/foundationdb/configuration.html#the-configuration-file
[general]
restart-delay = 60
## by default, restart-backoff = restart-delay-reset-interval = restart-delay
# initial-restart-delay = 0
# restart-backoff = 60
# restart-delay-reset-interval = 60
cluster-file = /usr/local/etc/foundationdb/fdb.cluster
# kill-on-configuration-change = true
## Default parameters for individual fdbserver processes
[fdbserver]
command = /usr/local/libexec/fdbserver
public-address = auto:$ID
listen-address = public
datadir = /usr/local/foundationdb/data/$ID
logdir = /usr/local/foundationdb/logs
# logsize = 10MiB
# maxlogssize = 100MiB
# machine-id =
# datacenter-id =
# class =
memory = 8GiB
# storage-memory = 1GiB
# cache-memory = 2GiB
# metrics-cluster =
# metrics-prefix =
## An individual fdbserver process with id 4689
## Parameters set here override defaults from the [fdbserver] section
[fdbserver.4689]
[backup_agent]
command = /usr/local/foundationdb/backup_agent/backup_agent
logdir = /usr/local/foundationdb/logs
[backup_agent.1]
Then I executed sudo pkill fdbserver
Sadly, the problem persists:
fdb> status details
WARNING: Long delay (Ctrl-C to interrupt)
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.
Unable to start batch priority transaction after 5 seconds.
Unable to retrieve all status information.
Configuration:
Redundancy mode - single
Storage engine - memory
Log engine - ssd-2
Encryption at-rest - disabled
Coordinators - 1
Desired Commit Proxies - 3
Desired GRV Proxies - 1
Desired Resolvers - 1
Desired Logs - 3
Usable Regions - 1
Cluster:
FoundationDB processes - 1
Zones - 1
Machines - 1
Memory availability - 0.3 GB per process on machine with least available
>>>>> (WARNING: 4.0 GB recommended) <<<<<
Fault Tolerance - 0 machines
Server time - 12/06/23 12:36:46
Data:
Replication health - (Re)initializing automatic data distribution
Moving data - unknown (initializing)
Sum of key-value sizes - unknown
Disk space used - 105 MB
Operating space:
Storage server - 1.0 GB free on most full server
Log server - 0.0 GB free on most full server
Workload:
Read rate - 21 Hz
Write rate - 0 Hz
Transactions started - 8 Hz
Transactions committed - 0 Hz
Conflict rate - 0 Hz
Performance limited by server 55926f920eec545a: Log server MVCC memory.
Backup and DR:
Running backups - 0
Running DRs - 0
Process performance details:
127.0.0.1:4689 ( 2% cpu; 4% machine; 0.000 Gbps; 2% disk IO; 0.1 GB / 0.3 GB RAM )
Coordination servers:
127.0.0.1:4689 (reachable)
Client time: 12/06/23 12:36:38
And eventually settles back to:
fdb> status details
WARNING: Long delay (Ctrl-C to interrupt)
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.
Timed out fetching cluster status.
Configuration:
Redundancy mode - unknown
Storage engine - unknown
Log engine - unknown
Encryption at-rest - disabled
Coordinators - unknown
Usable Regions - unknown
Cluster:
FoundationDB processes - unknown
Zones - unknown
Machines -
Machines - unknown
Data:
Replication health - unknown
Moving data - unknown
Sum of key-value sizes - unknown
Disk space used - unknown
Operating space:
Unable to retrieve operating space status
Workload:
Read rate - unknown
Write rate - unknown
Transactions started - unknown
Transactions committed - unknown
Conflict rate - unknown
Backup and DR:
Running backups - 0
Running DRs - 0
Process performance details:
Coordination servers:
127.0.0.1:4689 (reachable)
Client time: 12/06/23 12:38:32
Interesting, running a tail on the logs shows a ton of continuous activity by the server, even as fdbcli status reports a timed-out/unknown status. (and I can’t interact with the server in any meaningful way)
Interesting, running a
tailon the logs shows a ton of continuous activity by the server, even asfdbcli statusreports a timed-out/unknown status. (and I can’t interact with the server in any meaningful way)
Are you able to share those logs? If those are too many, can you filter them by Severity (30+ should contain the important information).
@johscheuer thanks (again!) for your help.
I’ve pasted the last ~300kb of the logs to 1701958617.597571" DateTime="2023-12-07T14:16:57Z" Type="ProcessMetrics" ID="000 - Pastebin.com. (if there’s a better way to ship a full log file to this forum, I’m all ears)
Interestingly (ugh?), I couldn’t find any Severity exceeding 20 after grepping through the logs.
(if it helps at all, the file was /usr/local/foundationdb/logs/trace.127.0.0.1.4689.1701885084.vW9C2y.1.79.xml)
Something I’ve since tried:
Totally uninstalling FoundationDB, restarting my machine with “Reopen windows when logging back in” turned off (to ensure maximum resource availability), then reinstalling FoundationDB-7.3.27_arm64.pkg.
Same issues as before 
So… I dug out an original MacBook Air (M1, 2020). Installed FoundationDB-7.3.27_arm64.pkg and… everything just worked 
This is what it looks like right after the server starts:
fdb> status details
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.
Configuration:
Redundancy mode - single
Storage engine - memory
Log engine - ssd-2
Encryption at-rest - disabled
Coordinators - 1
Desired Commit Proxies - 3
Desired GRV Proxies - 1
Desired Resolvers - 1
Desired Logs - 3
Usable Regions - 1
Cluster:
FoundationDB processes - 1
Zones - 1
Machines - 1
Memory availability - 0.2 GB per process on machine with least available
>>>>> (WARNING: 4.0 GB recommended) <<<<<
Fault Tolerance - 0 machines
Server time - 12/07/23 15:42:13
Data:
Replication health - (Re)initializing automatic data distribution
Moving data - unknown (initializing)
Sum of key-value sizes - unknown
Disk space used - 105 MB
Operating space:
Storage server - 1.0 GB free on most full server
Log server - 65.0 GB free on most full server
Workload:
Read rate - 40 Hz
Write rate - 0 Hz
Transactions started - 22 Hz
Transactions committed - 1 Hz
Conflict rate - 0 Hz
Backup and DR:
Running backups - 0
Running DRs - 0
Process performance details:
127.0.0.1:4689 ( 3% cpu; 43% machine; 0.000 Gbps; 7% disk IO; 0.1 GB / 0.2 GB RAM )
Coordination servers:
127.0.0.1:4689 (reachable)
Client time: 12/07/23 15:42:13
…and once things settle down:
fdb> status details
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.
Configuration:
Redundancy mode - single
Storage engine - memory
Log engine - ssd-2
Encryption at-rest - disabled
Coordinators - 1
Desired Commit Proxies - 3
Desired GRV Proxies - 1
Desired Resolvers - 1
Desired Logs - 3
Usable Regions - 1
Cluster:
FoundationDB processes - 1
Zones - 1
Machines - 1
Memory availability - 0.5 GB per process on machine with least available
>>>>> (WARNING: 4.0 GB recommended) <<<<<
Fault Tolerance - 0 machines
Server time - 12/07/23 15:42:20
Data:
Replication health - Healthy
Moving data - 0.000 GB
Sum of key-value sizes - 0 MB
Disk space used - 105 MB
Operating space:
Storage server - 1.0 GB free on most full server
Log server - 65.0 GB free on most full server
Workload:
Read rate - 22 Hz
Write rate - 0 Hz
Transactions started - 6 Hz
Transactions committed - 0 Hz
Conflict rate - 0 Hz
Backup and DR:
Running backups - 0
Running DRs - 0
Process performance details:
127.0.0.1:4689 ( 3% cpu; 46% machine; 0.000 Gbps; 9% disk IO; 0.1 GB / 0.5 GB RAM )
Coordination servers:
127.0.0.1:4689 (reachable)
Client time: 12/07/23 15:42:20
What’s interesting is that, on my MacBook Pro, “Log server” operating space is consistently reported as 0.0 GB, which is clearly wrong. On the MacBook Air, 65.0 GB is reported, which is close-ish to the amount of actual free space on the disk.
Clearly, there’s something up with my main dev machine. I’ll keep digging, but any pointers or ideas are much appreciated.
Update
For future travelers: this issue was apparently caused by an issue with my APFS “Data” volume. Running “First Aid” on it through Disk Utility immediately fixed all of my problems.
x:person_shrugging:
(and thanks again for stepping in to help, @johscheuer)
Update #2
…and it stopped working again. I think the culprit is:
Operating space:
Storage server - 1.0 GB free on most full server
Log server - 0.0 GB free on most full server
My guess: when the operating space for “log server” is reported at 0.0 GB, things stop working. (when it worked earlier today for me, “log server” space was reported at 0.2 GB, and then 0.1 GB; the fdbcli status posted earlier by @johscheuer also showed ample log server space, as did my working MacBook Air)
Will keep digging, but would appreciate any pointers on how/why log space could be showing 0.0 GB on a machine with ample free disk space.
Final update
Things are working now, and I’m pretty sure I tracked down the problem:
Although I had free disk space, it wasn’t enough based on how minFreeSpace is calculated.
"First Aid” was a red herring; instead, I believe it freed up just enough disk space so that the operating space for “log server” cleared 0.0 GB.
This was further confirmed after I went on a deleting spree, trashing old iOS simulators, cache files, etc.; the “log server” operating space increased, and everything worked perfectly.
I also ran across Work around "Storage server running out of space (approaching 5% limit)" on your developer machine - #3 by KrzysFR which provides a work-around for running out of free space in a development environment.
My new config looks like:
% cat /usr/local/etc/foundationdb/foundationdb.conf
## foundationdb.conf
##
## Configuration file for FoundationDB server processes
## Full documentation is available at
## https://apple.github.io/foundationdb/configuration.html#the-configuration-file
[general]
restart-delay = 60
## by default, restart-backoff = restart-delay-reset-interval = restart-delay
# initial-restart-delay = 0
# restart-backoff = 60
# restart-delay-reset-interval = 60
cluster-file = /usr/local/etc/foundationdb/fdb.cluster
# kill-on-configuration-change = true
## Default parameters for individual fdbserver processes
[fdbserver]
command = /usr/local/libexec/fdbserver
public-address = auto:$ID
listen-address = public
datadir = /usr/local/foundationdb/data/$ID
logdir = /usr/local/foundationdb/logs
# logsize = 10MiB
# maxlogssize = 100MiB
# machine-id =
# datacenter-id =
# class =
# memory = 8GiB
# storage-memory = 1GiB
# cache-memory = 2GiB
# metrics-cluster =
# metrics-prefix =
knob_min_available_space_ratio=0.001 # <<<<< Add this line
## An individual fdbserver process with id 4689
## Parameters set here override defaults from the [fdbserver] section
[fdbserver.4689]
[backup_agent]
command = /usr/local/foundationdb/backup_agent/backup_agent
logdir = /usr/local/foundationdb/logs
[backup_agent.1]
This information seems like it might be useful to other developers; perhaps there’s a place for it in the wiki or “getting started” project documentation?
thank you so much, this worked and saved ton of debugging time for me 