I thought I’d ask here, perhaps somebody has a magic recipe. I was planning to migrate my app from RethinkDB to FoundationDB, and did some work towards that a couple of years ago. But as I wanted to pick up and complete the work, I found out that I can’t, because I am unable to run FoundationDB on my development machine, which is a Mac with an M1 (Apple Silicon) processor.
I’m puzzled, because FoundationDB seems to be developed at Apple, and I’m guessing most developers have Apple Silicon machines by now? Am I missing something?
In my case, I deploy on Linux, but my application is developed on a Mac in a local JVM, with a couple of supporting Docker containers. I can run the fdb servers in Docker (assuming arm64 is supported as an architecture, but if local .so libs are required, these need to be accessible natively to the JVM.
Darwin does not support KAIO, try to pass disable_posix_kernel_aio=1 to both fdbserver process and fdbclient (with set knob). If you use fdbcli also pass this knob.
I think mac m1 support is pretty much there for 7.1+, but you need to build from source currently. Several developers are using mac m1’s and building locally (basically all testing happens on linux though).
Well, that’s good news! At least I know I’m not the only one that wants to use FoundationDB for development on a Mac machine
BTW, making fdb fully work on MacOS is perhaps unnecessary — a working arm64 docker container would be plenty enough (the current official docker container is intel-only). It’s not about being native or particularly fast, it’s about being able to do local development at all.
I think you can get pretty far with the x86_64 MacOS package (.pkg file) and Rosetta2 then. I was able to install it (I had to click through a warning about Apple not being able to verify it which is a bit silly), and then I was able to load libfdb_c from the python bindings as long as I did arch -x86_64 python3
I’ve just revisited this problem. I wanted to migrate my SaaS application to FoundationDB (it’s based on RethinkDB right now), and I’ve actually done much of the interfacing a couple of years back, but the inability to run fdb on Apple Silicon is a showstopper. I can’t develop on my machine.
Here’s what I did today:
Took a closer look at the ‘tigrisdata/foundationdb’ images on Docker Hub. Unfortunately, their source isn’t available (anymore?) and I have no idea what’s actually inside.
Checked the official Docker images, but these don’t have arm64 versions.
Revisited the Github issues related to Apple Silicon support. It seems they were all closed without resolution.
Tried a workaround suggested in one of the Github issues (--knob_disable_posix_kernel_aio=1), that still results in a core dump.
Noticed that there is no longer a way for users to open Github issues.
Wanted to try downloading the latest native release again, but the webpage is down (404).
The problem of the webpage being down seems to have been reported 3 weeks ago…
I’m not sure what to make of this… I have to say, all this is rather disheartening. Recent discussions of distributed databases on HN do not even mention FoundationDB, as nobody seems to be even considering it. I really like FoundationDB, and I do understand that I am entitled to absolutely nothing, but I’m growing increasingly worried about painting myself into a corner with a dying project, like I previously did with RethinkDB.
Hm, we are running FDB 6.3 on Apple Silicon with no issues. Literally just downloading it from Release Release 6.3.24 · apple/foundationdb · GitHub and running it. If you’re talking about bindings specifically, we use Java and any x86 JVM would do.
Tried a workaround suggested in one of the Github issues (--knob_disable_posix_kernel_aio=1), that still results in a core dump.
Could you give some more additional information like what version of FDB you are using? If you want to run FDB on your Mac (with Docker) you have to use the even numbered images (the one with AVX disabled). I just tried this Support Apple Silicon · Issue #4111 · apple/foundationdb · GitHub out and it still works without any issues on my M1. If you share some more information on what version(s) you tried it’s easier to help you.
If you want to run FDB directly on you Mac you can also use the FDB package for arm64 FoundationDB-7.1.27_arm64.pkg.
I tried with multiple versions — the docker container version that I reported in the GitHub issue (back when reporting issues was possible for non-collaborators) was 7.1.23. I retried recently with foundationdb/foundationdb:7.1.27 and the result was the same (segfault).
I had no idea about the even-numbered images and AVX, how would one normally find out about that? A quick check shows that foundationdb/foundationdb:7.1.27 doesn’t crash with a segfault, so it might work under emulation.
I also had no idea about the native arm64 pkg, last I checked these were not available. I installed one from your direct link and it seems to work just fine!
I tried with multiple versions — the docker container version that I reported in the GitHub issue (back when reporting issues was possible for non-collaborators) was 7.1.23. I retried recently with foundationdb/foundationdb:7.1.27 and the result was the same (segfault).
I had no idea about the even-numbered images and AVX, how would one normally find out about that? A quick check shows that foundationdb/foundationdb:7.1.27 doesn’t crash with a segfault, so it might work under emulation.
I also had no idea about the native arm64 pkg, last I checked these were not available. I installed one from your direct link and it seems to work just fine!
They are fairly new and were added with 7.1.26, so not surprising that you had no idea about them
I tried the 7.3.26 with docker and I get Disk i/o operation failed. Any ideas?
docker run --rm foundationdb/foundationdb:7.3.26
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Starting FDB server on 172.17.0.2:4500
<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
<jemalloc>: (This is the expected behaviour if you are running under QEMU)
Error: Disk i/o operation failed
I tried a few other version, including 7.1.26, which failed with a different error.
This is all on an m1 max.
matthew@Matthew-Newhook fdbtop % docker run foundationdb/foundationdb:7.1.26
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Starting FDB server on 172.17.0.2:4500
/var/fdb/scripts/fdb.bash: line 68: 32 Killed fdbserver --listen-address 0.0.0.0:"$FDB_PORT" --public-address "$PUBLIC_IP:$FDB_PORT" --datadir /var/fdb/data --logdir /var/fdb/logs --locality-zoneid="$(hostname)" --locality-machineid="$(hostname)" --class "$FDB_PROCESS_CLASS"
Thanks for the help so far. The next problem that I ran into is that the fdbserver isn’t usable from the fdb apps on my local machine. I tried exposing port 4500, that didn’t work. I tried using host networking, but that didn’t work either as the host isn’t actually my mac. Is there a solution here, or is the only real solution to use the native package installer? I’d really rather not do that, because it would require all developers to install the package vs docker which works for everyone without any additional work.
Thanks for the help so far. The next problem that I ran into is that the fdbserver isn’t usable from the fdb apps on my local machine. I tried exposing port 4500, that didn’t work. I tried using host networking, but that didn’t work either as the host isn’t actually my mac. Is there a solution here, or is the only real solution to use the native package installer? I’d really rather not do that, because it would require all developers to install the package vs docker which works for everyone without any additional work.
The host network setup should work, as long as you’re using the IP address of the docker VM:
# Start the container with the FDB server, this will be blocking. Wait until the cluster is running
$ docker run --platform=linux/amd64 --net=host -ti --entrypoint=/var/fdb/scripts/fdb_single.bash --name=fdb-dev foundationdb/foundationdb:7.3.26
Starting FDB server on 192.168.64.2:4500
fdbserver pid is: 12
FDBD joined cluster.
Configuring new single memory FDB database
Database created
Using cluster file `/var/fdb/fdb.cluster'.
Configuration:
Redundancy mode - single
Storage engine - memory
Log engine - ssd-2
Encryption at-rest - disabled
Coordinators - 1
Desired Commit Proxies - 3
Desired GRV Proxies - 1
Desired Resolvers - 1
Desired Logs - 3
Usable Regions - 1
Cluster:
FoundationDB processes - 1
Zones - 1
Machines - 1
Memory availability - 8.0 GB per process on machine with least available
Fault Tolerance - 0 machines
Server time - 11/27/23 08:35:16
Data:
Replication health - (Re)initializing automatic data distribution
Moving data - unknown (initializing)
Sum of key-value sizes - unknown
Disk space used - 105 MB
Operating space:
Storage server - 1.0 GB free on most full server
Log server - 408.6 GB free on most full server
Workload:
Read rate - 0 Hz
Write rate - 0 Hz
Transactions started - 0 Hz
Transactions committed - 0 Hz
Conflict rate - 0 Hz
Backup and DR:
Running backups - 0
Running DRs - 0
Client time: 11/27/23 08:35:16
fdbserver --listen-address 0.0.0.0:"$FDB_PORT" --public-address "$PUBLIC_IP:$FDB_PORT" --datadir /var/fdb/data --logdir /var/fdb/logs --locality-zoneid="$(hostname)" --locality-machineid="$(hostname)" --class "$FDB_PROCESS_CLASS"
Now in a different terminal:
$ docker exec -ti fdb-dev cat /var/fdb/fdb.cluster > fdb-dev.cluster
$ fdbcli -C fdb-dev.cluster
Using cluster file `fdb-dev.cluster'.
The database is available.
Welcome to the fdbcli. For help, type `help'.
fdb> status details
Using cluster file `fdb-dev.cluster'.
Configuration:
Redundancy mode - single
Storage engine - memory
Log engine - ssd-2
Encryption at-rest - disabled
Coordinators - 1
Desired Commit Proxies - 3
Desired GRV Proxies - 1
Desired Resolvers - 1
Desired Logs - 3
Usable Regions - 1
Cluster:
FoundationDB processes - 1
Zones - 1
Machines - 1
Memory availability - 8.0 GB per process on machine with least available
Fault Tolerance - 0 machines
Server time - 11/27/23 09:35:25
Data:
Replication health - (Re)initializing automatic data distribution
Moving data - unknown (initializing)
Sum of key-value sizes - unknown
Disk space used - 105 MB
Operating space:
Storage server - 1.0 GB free on most full server
Log server - 408.6 GB free on most full server
Workload:
Read rate - 12 Hz
Write rate - 0 Hz
Transactions started - 7 Hz
Transactions committed - 0 Hz
Conflict rate - 0 Hz
Backup and DR:
Running backups - 0
Running DRs - 0
Process performance details:
192.168.64.2:4500 ( 6% cpu; 1% machine; 0.000 Gbps; 2% disk IO; 0.1 GB / 8.0 GB RAM )
Coordination servers:
192.168.64.2:4500 (reachable)
Client time: 11/27/23 09:35:25
Thanks but that doesn’t work for me because the docker VM host isn’t accessible. Looks like this is some sort of docker issue for me. I’ve been using rancher desktop. What OSX docker package are you using?
This seems to have worked. Took a while to put all the details together.
The lack clear documentation, flakyness with the intel images, and lack of an arm image makes working on a mac with the docker images quite a challenging proposition (the natives installed packages work fine of course). Is there anything we could do to help here?
