Inspired by pull request #850, I’d like to kick off a discussion about FoundationDB development workflows and making “improvements” in that area. I use quotes because I recognize the subjectivity inherent to this area.
I know Markus has mentioned that Snowflake’s using cmake for that purpose. I hope he’ll chime in here to discuss the ups and downs of that.
My pre-disposition is toward Bazel because it’s well supported within Dropbox. Out of the box, we get parallel builds, caching, integration with our CI and toolchains, and a simple python-like syntax.
In order to advance the discussion, I’ve separated most of the changes to existing code into a separate PR (#856). Alex indicated that it would be a welcome change separate from Bazel, and I think it will make adopting any other build system easier as well.
We are using cmake and our FDB 6 fork is ported to cmake already.
In general we are quite happy with cmake. My bazel-knowledge is limited but my understanding is that these are the benefits of cmake:
Better known by C+±devs. This is imho one of the biggest ones.
Very few dependencies
Meta-build (so it supports GNU make, Ninja, VS etc)
Easy things are much simpler to do in Bazel, but it seems that hard things are easier to do in cmake (but I might be wrong about this one)
I hope I can soon create a pull-request for our cmake stuff. This is what it currently supports:
automatic download of dependencies if they are not found on the host or if the software on the host is outdated
currently rpm (including debuginfo-packages) and deb packages are created with cmake. cmake also supports OS X and Windows formats, but I didn’t implement this yet
It builds the Java-client, the actorcompiler, the vexillographer, and then compiles all actor files and and then the cpp files.
CTest will execute all simulated tests and it can execute tests in parallel. It can also do reporting and in our case write test results and traces from tests into a Snowflake database (this part is probably only interesting for us and we don’t plan to open source this unless there’s some demand for that)
I think in the end the biggest benefit of cmake is that it is pretty much the standard build system for C++.
I’ve spent some time trying to dig through what bazel does and does not support. I’m personally a bit biased in favor of bazel, but I’m concerned it might not fit the requirements or desires that we have. I have no idea how much bazel has improved, so please let me know if any of these are inaccurate:
I would be happy about:
Sane, easy to write and extend build language
Easy to flip between static and dynamic linking for local development
Monorepo makes setting up an ASAN/MSAN build significantly easier, and valgrind is slow
I’m concerned about:
We need to support linux+Mac+windows. Linux is easy, Mac probably works, but it looks like windows needs a lot of special cases.
The libressl dependency will be awkward to handle for every blaze reason possible. It’s not bazel-ified, and bazel doesn’t have integration with pkg-config to be able to rely on the system libressl.
I’m a little unenthused about the monorepo approach without remote execution of build rules and remote cache. If we go with the model that everything required to run msan should be tracked by bazel, Changing from -O2 to -O0 to debug something would cause one to need to recompile libc++ and libressl, which hurts. Apparently remote caching now exists in bazel, but it’s not something we would likely feel good running for community developers, who would be stuck with very long builds.
Looking at the C++ roadmap for bazel, I see some of my concerns on there. I think our strict requirements are to be able to handle linux, Mac, and windows builds well, and be able to sensibly handle the libressl dependency[1]. One would likely be able to hack around most of everything else (e.g. lack of make install equivalent in bazel). As far as I’ve been able to see, CMake can handle our requirements well, today, so I think I’m still leaning towards an eventual rewrite into CMake.
If that ends up being the case, this would leave you with two ways forward:
Wrap a CMake build system in bazel. This seems maybe possible with a repository_rule, but I’m also reading a lot of complaints about it, so I have no idea. Better support for this is on the roadmap.
Maintain bazel as a parallel build system. If it means community involvement and participation, I’m not going to stand in the way of having bazel as a secondary build system, and we can talk further about how to most sensibly fit FDB into bazel’s view of the world, but we would require that any upstreamed changes work when built with CMake.
And although I’ve leapt on handling your PR, it’s not like I’m the voice of FDB, so other opinions/commentary are clearly welcome.
[1]: One can argue that we don’t have sane handling of libressl regardless of what we do. Our docker image contains a custom build of it, as we need a static library linked with -fPIC. Maybe that’s better than being in a monorepo, because you’ll only ever compile it once, but maybe not as you’d need a special version of it for asan/msan anyway…
Bazel is admittedly better for large shops and is less good when you need the platform configuration and detection of things like CMake and the autotools.
I think there is room to have Bazel coexist alongside other systems; if the code is kept simple and free from magic (a good thing for a system like fdb anyway), it should be easy for us to keep it up-to-date.
This topic came up during a meeting. It was agreed that best effort maintaining bazel should be a low maintenance burden, and as long as we have sufficiently clear documentation describing its level of support, it shouldn’t cause new developer confusion. Thus, no one seemed opposed, so it seems you’re welcome to have bazel support for as long as you have an interest in maintaining it.
I’ll comb through your PR for bazel in more detail once all the header changes have dropped out, and I’ll go put some thought into how FDB fits into bazel’s opinionated view of the world.