What is the status of running a cluster on macOS with the current version of FoundationDB? I am aware of this message:
The macOS version of the FoundationDB server is intended for single-machine development use only; its use in multi-machine clusters is not supported. In the present release, the Linux version is the best-tested and most performant platform for multi-machine clusters.
… but with the most recent M1 Mac minis coming out (along with rumors of 10GBASE-T variants coming soon), they could make for a fairly dense, affordable, multi-machine cluster in a rack, not to mention the performance these machines could bring (and we can hope for an ECC RAM variant down the line ).
All that hypothetical stuff out of the way, I’m very much a Mac person (fairly clueless about the layout of operation of a Linux box), and I do have 4 Intel Mac minis at my immediate disposal (and about 2 dozen more in our own data center racks) I wanted to test a multi-machine setup with — the question is will it work? I figured performance may not be as optimized, but I was hoping the stance above has changed, at least a bit, over the years.
If running a multi-machine cluster directly on macOS is a no go, would running it via docker on macOS be a supported configuration?
My immediate concern is playing around with adding machines and getting comfortable operating a cluster, but if it was something that could work long term, I mean I can easily fit 8 of these in a 5U shelf, so why wouldn’t I want to
Is it safe to run a FDB production-cluster on macOS.
Does FDB support macOS on ARM.
The short answer to both questions is currently no. The long answer is a bit more complicated:
Is it safe to run a FDB production-cluster on macOS.
The biggest problem here is that we don’t run our simulation tests at scale on anything but Linux. Testing FDB is expensive (Snowflake runs thousands of CPU hours for FDB testing every day) and currently nobody is willing to pay the cost for macOS testing (it would probably be cheaper, I don’t think the same scale of testing is really necessary, but we would need to think more about this).
Does FDB support macOS on ARM.
Currently, FDB doesn’t compile on Apple Silicon (it does work on Linux+ARM - though this build also isn’t as well tested). I took a first crack on this. However, there are still a few problems:
Our coroutine library doesn’t work on Apple Silicon. I replaces it with boost coroutine2 (which is probably a good change in itself) but this will require extensive testing.
Currently that build crashes on Apple Silicon - so there’s still some debugging to do. Also, none of the bindings are getting compiled atm (except for the C bindings).
The current version of boost doesn’t compile properly on Apple Silicon (at least for me). So either this will need to depend on a pre-release build or we’ll have to wait for a future boost version.
We certainly plan to provide FDB on Apple Silicon for development - though it will like take some time. Currently I don’t know of any efforts to make this production ready though.
All this being said: it is not clear to me why an Apple Silicon cluster would be desirable? Don’t get me wrong: I am super impressed of what Apple was able to pull off here (and I ordered my MacBook Pro at the release day). However, running an Apple Mini cluster just doesn’t seem to be very practical.
Did you look at AWS A1 instances? I heard very good things about those (don’t know how they compare to Apple Silicon) and operating an AWS A1 cluster is almost certainly going to be much cheaper than buying and running a bunch of Mac Minis.
This was really just hypothetical/wishful thinking, but it makes sense that it would be a somewhat high priority since development will likely move to Apple Silicon Macs sooner rather than later.
Is it safe to run a FDB production-cluster on macOS
I was hoping things would have changed a bit over the past few years, but this makes sense.
There aren’t many reasons to run our server apps on macOS, but we do have some heavily optimized code that targets AVFoundation, and the Secure Enclave makes for a convenient built-in hardware security module for key signing, and ultimately, its what we are used to, hence the abundance of Mac minis at my disposal (the T2s really did scream with regard to H.265 encoding, especially considering the low power draw in general of the minis, and compact, though odd, shape in a server rack). Not that we would use them for our production deployment of FoundationDB, but certainly the easiest solution for me if it were supported (especially since we already have the empty rack space due to other, unrelated projects)
It seems like I can probably go ahead and try to set up a Mac cluster in my home office since I won’t be handling important data really - it’s sole purpose is to get to know what I’m doing with machines that are directly available in front of me, and not knowing anything about Docker really, I’m guessing that’s a decent plan B, at least for these “tests”?
I would expect that this would work… Let us know, I would be interested in that experience.
Another thing that I forgot to mention: on macOS FDB can’t use kernel AIO - which means that it will use EIO instead. This probably will mean that storage will run ~10% slower.
We’re also planning to run Joshua on a libc++ build. This could also uncover potential bugs (we had a bug once that was caused by libc++ and libstdc++ calling destructors in vectors in a different order - there might be other unknown subtleties).
If you have a lot of resources, we might also be able to help you use those for simulation testing
I’ll definitely report back based on the performance/any oddities I’m seeing, and I’ll give the Docker route a try as well. Although I don’t have a ton of time resources available, as soon as the development crunch phase ends in a few months, I’ll gladly accept your help getting a larger-scale Mac cluster properly tested, for science!
(You’d think Apple stewarding the project would be more interested than me in playing around with this, but I guess that would be all hush hush even if it were happening )
We are doing some development with FoundationDB and some people in the team would like to use M1 or the upcoming M2 for local deployment. Any timeframe as to when an arm64 build will be ready?
We’re currently working on FDB on ARM Linux (which is mostly working already but not officially supported yet as we still need to build test infrastructure). For macOS I am not aware of a timeline.
When I tried to get FDB running on my M1 mac I ran into problems with some dependencies, so I decided to wait until support for M1 improves. One of the tricky parts is our coroutine library which didn’t work at all on arm Macs. We recently switched to use boost coroutine but even that didn’t support macOS on ARM at the time (and I didn’t check whether support was added by now).
Generally speaking though: sadly macOS on ARM is low priority for us and AFAIK most FDB developers don’t have access to M1 Macs. So I don’t think support is coming very soon. However, we would be more than happy to accept pull requests
That’s good! The needs of my team are to be able to run FoundationDB inside a kubernetes cluster (either on the cloud or locally inside k3d), in Linux docker images. My understanding is that a working FoundationDB ARM64 build on linux would allow such deployment scenario even on M1. Correct me if I am wrong.
In fact, I was more referring to a potential, M2/M1X evolution of the M1 where the ARM64 will start to be interesting for development purpose (more memory, etc…). From this point, I think a properly working Linux ARM64 FoundationDB build would be quite welcome
Would be happy to contribute! (more on the Record Layer though since it’s the code base I am more familiar with…)
).
)