Unable to get started on a Mac M2

rabernat · November 14, 2023, 12:55am

Thanks for sharing FoundationDB with the community as an open-source project. It looks like a truly amazing piece of technology.

I recent learned about FoundationDB and became curious to try it out. However, I cannot find a combination of installations that works for both client and server! I am on an Apple M2 laptop using macOS 14.0. I want to use the Python API. This is very frustrating, and I am about ready to give up.

My Steps

I am attempting to follow the official Getting Started guide at https://apple.github.io/foundationdb/getting-started-mac.html.
This led me to the downloads page: https://apple.github.io/foundationdb/downloads.html
On that page, I am told to download version 6.3.23 https://github.com/apple/foundationdb/releases/download/6.3.23/FoundationDB-6.3.23.pkg

Attempt to use 6.3.23

I used the package installer to install both client and server
fdbcli --exec status works. The database is running. Yay!

% fdbcli --exec status
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster'.

Configuration:
  Redundancy mode        - single
  Storage engine         - memory-2
  Coordinators           - 1
  Usable Regions         - 1

Cluster:
  FoundationDB processes - 1
  Zones                  - 1
  Machines               - 1
  Memory availability    - 8.0 GB per process on machine with least available
  Fault Tolerance        - 0 machines
  Server time            - 11/13/23 17:27:36

Data:
  Replication health     - (Re)initializing automatic data distribution
  Moving data            - unknown (initializing)
  Sum of key-value sizes - unknown
  Disk space used        - 105 MB

Operating space:
  Storage server         - 1.0 GB free on most full server
  Log server             - 117.5 GB free on most full server

Workload:
  Read rate              - 16 Hz
  Write rate             - 0 Hz
  Transactions started   - 4 Hz
  Transactions committed - 0 Hz
  Conflict rate          - 0 Hz

Backup and DR:
  Running backups        - 0
  Running DRs            - 0

Client time: 11/13/23 17:27:36

Now I tried to use Python. The docs say

Python and C APIs were installed using the FoundationDB installer above.

However, I found this not to be the case, nor do I see how it would be possible on macOS. There is no longer a “system Python” on macOS. (I manage python environments using mamba.)

I installed the python bindings via pip install foundationdb
I attempted to connect to my database using the documented instructions for the python API (https://apple.github.io/foundationdb/api-python.html)

$ python -c 'import fdb; fdb.api_version(710); fdb.open()'
...
OSError: dlopen(/usr/local/lib/libfdb_c.dylib, 0x0006): tried: '/usr/local/lib/libfdb_c.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/lib/libfdb_c.dylib' (no such file), '/usr/local/lib/libfdb_c.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))
...
Exception: Unable to locate the FoundationDB API shared library!

Traceback (most recent call last):
  File "/Users/rabernat/mambaforge/lib/python3.10/site-packages/fdb/impl.py", line 1515, in <module>
    _capi = ctypes.CDLL(capi_name)
  File "/Users/rabernat/mambaforge/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(libfdb_c.dylib, 0x0006): tried: 'libfdb_c.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibfdb_c.dylib' (no such file), '/Users/rabernat/mambaforge/lib/python3.10/lib-dynload/../../libfdb_c.dylib' (no such file), '/Users/rabernat/mambaforge/bin/../lib/libfdb_c.dylib' (no such file), '/usr/lib/libfdb_c.dylib' (no such file, not in dyld cache), 'libfdb_c.dylib' (no such file), '/usr/local/lib/libfdb_c.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/usr/lib/libfdb_c.dylib' (no such file, not in dyld cache)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rabernat/mambaforge/lib/python3.10/site-packages/fdb/impl.py", line 1522, in <module>
    _capi = ctypes.CDLL(lib_path)
  File "/Users/rabernat/mambaforge/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/usr/local/lib/libfdb_c.dylib, 0x0006): tried: '/usr/local/lib/libfdb_c.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/lib/libfdb_c.dylib' (no such file), '/usr/local/lib/libfdb_c.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/rabernat/mambaforge/lib/python3.10/site-packages/fdb/__init__.py", line 72, in api_version
    import fdb.impl
  File "/Users/rabernat/mambaforge/lib/python3.10/site-packages/fdb/impl.py", line 1524, in <module>
    raise Exception("Unable to locate the FoundationDB API shared library!")
Exception: Unable to locate the FoundationDB API shared library!

At this point, I wondered if there was a mismatch between the version I had installed and the fdb.api_version(710) instruction. I retried with fdb.api_version(630).

From reading the traceback, I surmised that I might need an arm64 library. This was confusing…clearly the server is working. But the Python client is not? Anyway, there is no arm64 build for 6.3.23, so I moved to version 7

Before moving on, I completely deleted my installation (using the sudo /usr/local/foundationdb/uninstall-FoundationDB.sh script) and started fresh.

Attempt to use 7.3.27

I downloaded FoundationDB-7.3.27_arm64.pkg and installed both client and server using the installer.

Now my server doesn’t work at all. My problem is identical to the one documented here, with no resolution in the past 5 months:

On the bright side,

python -c 'import fdb; fdb.api_version(710); fdb.open()'

now works fine. However, of course, I can’t actually do any transactions with the database because it isn’t running.

Summary

I have a very common setup (dare I say, the most common?) for a developer, a recent model Mac laptop with an M2 chip, plus wanting to use Python. I attempted to install and use foundationdb by following the documented instructions carefully.

With 6.3.23, my server works, but my Python client does not (complains about “incompatible architecture”)
With 7.3.27, my Python client works, but my server does not!

I would appreciate any advice that will unblock me. My next ideas would probably be to either use Docker or build from source. I anticipate both would be time-sucking rabbit holes.

p.s. The forum would not let me include more than 2 links, so I used literals for my URLs.

newhook · November 14, 2023, 1:46am

I believe the 7.1.x versions are the latest stable versions. I’ve been using that on my Mac and in our Linux testing environment environment for the past few weeks.

rabernat · November 14, 2023, 2:00am

Thanks for the quick reply!

I have tried with serveral different 7.1.x versions including the latest release (7.1.42: FoundationDB-7.1.42_arm64.pkg). I also tried both even and odd releases, as apparently these differ in whether they are compiled with AVX instructions. Unfortunately, I have the same server problem for all 7.1.x versions I have tried (“The database is unavailable”).

Can you share the specific version and package installer you used to get 7.1.x running on an M2 Mac?

rabernat · November 14, 2023, 2:10am

Ok so I did some more digging, and I discovered that, if I install the x86_64 packages for 7.1.x, the server works! (e.g. https://github.com/apple/foundationdb/releases/download/7.1.42/FoundationDB-7.1.42_x86_64.pkg) This is similar to what I found with 6.3.23.

However, in this case, the Python client doesn’t work (“Unable to locate the FoundationDB API shared library!”)

So it seems like I have to choose between the client or the server.

rabernat · November 14, 2023, 2:48pm

Ok, so I managed to get things working by first installing the arm64 client and server and then installing the x86_64 client only!

I’m happy I found a workaround, but something feels quite broken here. It would be great if the documentation and distributed packages could provide a smoother onboarding experience. I don’t think my setup is weird or unique in any way.

newhook · November 14, 2023, 3:28pm

My mac is M1 Max. I’ve used both 7.1.43 and 7.1.37. When you say “client” above you are talking the python client? Does fdbcli work correctly? I can’t speak directly to the python bindings.

rabernat · November 14, 2023, 3:41pm

fdbcli “works” in for both architectures (no error messages), but can only connect to the database when I have installed the x86_64 server.

The Python bindings only work when I install the arm64 client.

mavenraven · December 25, 2023, 7:57pm

@rabernat FWIW, I have an M1, and FoundationDB-7.1.26_arm64.pkg works as expected without any need for the x86_64 package. I did have to reboot my machine for fbcli to sucesfully connect to the cluster though.

This is also not using the Python bindings, but instead the Java record layer.

Martin · February 25, 2024, 8:59pm

I also found the documentation to be currently misleading and many combinations did not work on an M3. This page was the top search response.
7.3.26 ARM worked for me both the server and client importing fdb in python. Release Release 7.3.26 · apple/foundationdb · GitHub

I believe the key is to use a non ARV ARM build. 7.3.27 is marked as latest currently, but 7.3.26 is the equivalent non-arv build.