FoundationDB

FoundationDB on NixOS: packaging, notes and questions


(Austin Seipp) #1

Hello *,

First, sorry for the long post in advance. But I’d like to say that, as of yesterday, I’ve added a FoundationDB package to the NixOS Linux distribution, as well as a module for using FoundationDB in a declarative manner on NixOS.[1] I’m a software engineer and NixOS package maintainer, so I figure I’d drop by and announce it and ask some questions concerning packaging for “non-standard” Linux distributions.

I also think NixOS is perhaps the first distro to pick up a working FoundationDB build (made easier for reasons explained below), and I hope this can help other developers get it working where needed, for the time being.


Some background

NixOS is Linux distribution built on a declarative package management. Rather than running sets of commands to bring a system to some state, by modifying the existing state – you write a declaration specifying how you would like your machine to work, which is then realized for you. Multiple system states can co-exist, and you switch between them to go from one to another. The results of this design decision are profound in practice, including atomic system upgrades, transactional system rollbacks, reproducible builds (that properly track dependencies), supporting conflicting/multiple library versions, and more.

This turns out to be extremely useful for FoundationDB which has an… interesting build process, to say the least.

For example, the declarative description of the FoundationDB package for NixOS may end up as a better starting point for building it yourself[2] on foreign systems, vs something like the Docker image, once you learn to read it. In particular, this description:

  • Correctly captures build requirements. If a build requirement is not listed in the declarative description (under the name buildInputs), it is not available during compilation/build time. Thus, removal of ‘gawk’ means the ‘awk’ command will not be available, and the build will fail. This means every build dependency must be explicitly listed, and there are no cases where this would fail (for example, by failing to suggest to a user to apt-get install something.) Networking is removed, all builds are sandboxed with filesystem namespaces, and more.

  • Handles odd version dependencies. For example, the FoundationDB build includes a copy of Boost 1.52.0, as a header package. This package is private to the FoundationDB build and never exposed to any other packages, as it is otherwise too old to support. While Nix actually supports as far back as Boost 1.55, 1.52 is too old to use with our generalized Boost build infrastructure. Creating a custom variant of Boost is extremely easy.
    Similarly, NixOS natively handles multiple GCC versions, and the FoundationDB build is overridden explicitly to GCC 4.9 this in all configurations, as that’s what it supports.

  • Tells you precisely how results are installed; for example, you must build all and fdb_c in the Makefile targets, and then copy the resulting files out of a few different places.

  • Atomic. Any updates to any dependencies for FoundationDB (or the dependencies of its dependencies) will imply a rebuild of FoundationDB. Because every package, every dependency, is described is controlled in this single git repository, this effectively means Nix provides a nearly-transparent, completely reproducible build environment for FoundationDB. This is something not even Docker can easily provide, as even a single apt get install <foo> from an un-pinned source at any point in the creation of the image can be fatal. (This is why people typically standardize on Docker images and not Docker files – because all too often Dockerfiles are not actually reproducible and are poisoned like this.)

  • Tells you what needs patching. For example, Nix removes all .git directories from the source code when performing builds (it’s may be prohibitively large anyway), and many other build systems/CI systems for Linux distros tend to strip this as well – preferring explicitly packaged tarballs. This required a patch to the FoundationDB build system to not search for .git, as it otherwise required it for determining version revision information.


FoundationDB on NixOS

Using FoundationDB with the included NixOS module is pretty easy. With the latest master version of NixOS, you can enable it automatically on boot in just a few lines. For example, here’s a copy of my configuration, running FoundationDB on a 1950X ThreadRipper with 64GB of RAM (~5GB per process).

  services.foundationdb = {
    enable = true;
    dataDir = "/data/fdb/data";
    logDir  = "/data/fdb/logs";
    serverProcesses = 12;
    backupProcesses = 6;
    extraReadWritePaths = [ "/data/fdb/backups/" ];
  };

That’s it, and FoundationDB will automatically be installed – binary packages will be downloaded, put into place, and system services started all at once. This simple declaration has a lot of logic behind it[2]:

  • fdbmonitor and all fdbserver processes are controlled by a single systemd unit, called foundationdb.service, which is available via systemctl. It’s correctly started after networking interfaces are available and the system is booted.
  • Data directories and log directories are pointed explicitly onto fast NVMe drives (but default to /var/log/foundationdb and /var/lib/foundationdb like usual).
  • Auto-mouting semantics. Thanks to systemd’s RequiresMountsFor= directive, when FoundationDB is started, any filesystem mounts along the path of the data or log directories are automatically mounted upon server start, and a systemd dependency is implicitly added to ensure this mount happens first and succeeds. For my server, my main OS partition is on an NVMe boot drive, while my FoundationDB data is on a completely separate NVMe drive. This feature ensures the NVMe mount point is always available prior to server initialization.
  • The FoundationDB processes are heavily sandboxed, with explicit paths to read/write to (e.g. for backups, or for /etc/foundationdb). All forms of new permission elevation are denied, /dev is restricted, and /tmp is privately mounted in a way that is isolated from all other units. This is done through the use of systemd security features to put FoundationDB in its own control group and namespace on Linux. For example, the entire Linux system is effectively marked read-only from the POV of FoundationDB; any attempts to read or write outside of the log directory, data directory, or backup paths results in an explicit error. Further paths (like /boot) are not even readable at all. This is a security feature (which NixOS takes more seriously now), but also a sanity one to ensure FoundationDB is “being a good citizen” in a hostile world.
  • As expected, a new database (configure new single ssd) is also initialized at first startup.

The remaining options have sensible defaults, mostly following the ordinary default FoundationDB parameters. For example, you can also configure process class, storage memory, locality (zone/machine/dc/hall IDs), and more.[3]

You never need to run individual edits to an fdbmonitor configuration file, it’s all controlled here. Like Nix package builds, changes to this FoundationDB description are are atomic: any change either results in a restarted FoundationDB service, or, if the service fails (say, due to an invalid configuration parameter), the results are rolled back.

For NixOS users, I have also written a section in the manual about FoundationDB usage[4], which may be relevant or interesting to people here – I’d appreciate feedback! (Currently using a local copy I built myself, but this documentation will appear on https://nixos.org soon enough.)


Some questions

  • Is there any timeframe on support for newer GCC/boost libraries? Right now, while NixOS keeps copies of GCC 4.9, and the FoundationDB package provides Boost 1.52.0 (headers only), it would be nice to eliminate these hard dependencies. One reason is that GCC 4.9 is effectively on life support in NixOS; in fact there is almost nothing else that seems to explicitly require it. Had I not added FoundationDB recently, it’s possible GCC 4.9 would have been axed before too long, making adding it much harder.

    The other reason is just maintainabiliy; lots of specific workarounds ultimately hurt long-term maintainability for us, so working with upstreams to fix these problems is often desireable. (NixOS recently switched to GCC 7.3.0 as the default compiler, although 5.x and 6.x are also available.)

  • Is the above security policy for the FoundationDB server “correct”? In particular, while I prefer to enhance security of NixOS services where-ever possible that doesn’t break functionality (which has not seemed to happen with FDB), I’d like to ask if this is OK. More generally: In practice, having been an OSS maintainer of big projects, and a Linux distro maintainer, I know both sides of this: deviating from upstream policy enough is likely to create its own issues. However, these issues almost always make their way back upstream, not through the package maintainer. This creates an unfortunate series of events where primary upstream developers have to sort through downstream users, downstream packages, and downstream setup choices – and this isn’t always a good use of time on behalf of the developers.

    So I’d like to know if I’m doing something wrong or something you consider bad. If I am, I’ll gladly change this to be more in-line with what upstream expects, and it will be easier for everyone.

    In particular, FoundationDB does not seem to offer systemd unit descriptions for its packages (rather, it offers init.d-style scripts, which systemd distros such as recent RHEL, Ubuntu etc can transparently handle), so it’s possible none of this is wrong, but simply unfamiliar. (If you’d like, I’d be more than willing to help contribute upstream unit files, if you would like to provide them.)

  • Is there any explicit support policy on 3rd party distro packages? In a prior life, I developed and worked on proprietary Linux software that was sold to customers in a variety of (bizarre, horrifying) settings, so I understand the need for things like singular, static binaries, etc. This leads into a lot of the weird stuff in the build system, such as libc++ frobnication (something to do with symbol versioning?) and recent plans I read elsewhere to use __asm__ to mark glibc memcpy versions to link against.

    Of course, there’s nothing wrong with that! You have to do serious QA on a piece of software like FDB. But then the question is: what about packages you did not compile in this controlled environment? Should there be a notice “this isn’t an official upstream package”? Should there be any notice? Should downstream say something? Or should you say “Only packages provided by foundationdb.org have undergone our quality assurance, on these systems:” on the homepage? I’m not sure.

    I’m afraid my prior experience never lead into a proprietary project becoming OSS, so I’m not sure I can offer any guidance here on what to do. You’ve still got customers to support so you can’t just exactly obliterate all the stuff you have in place for this, but it adds a bit of tension between (formerly) proprietary product users and OSS users.

  • There is a lot of weird stuff going on in the build systems, relating to static libraries. I had to write a patch in order to fix this; in particular, it would seem as if the makefiles put ld flags like -lstdc++ inside of _LIBS variables for individual subprojects; but this is wrong, because _LIBS is for tracking files that will be generated by rules in the build system, not the actual flags that are needed at link time. That’s _LDFLAGS, and as you can see from the patch, it essentially just moves a bunch of xxx_LIBS += -lfoo to xxx_LDFLAGS += -lfoo in order to get things working.

    Without this patch, the net effect is that the build system will immediately fail; for example, Make will try to build fdbcli, but fdbcli needs the _LIBS dependencies to be built first. However, it has no idea how to build -ldl or whatnot because that’s not a rule, that’s a link-time flag. So you get a mysterious error "Error: cannot make dependency "-ldl", needed for bin/fdbcli" or whatnot.

    I do not know why this is needed. Of course, you all wrote the build system! So maybe my understanding of _LIBS is wrong. Perhaps I missed a subtle build dependency. Do _LIBS get transformed into _LDFLAGS by something? If so, where? If not, why does the build system list -ldl in _LIBS – when it seems to contain rule dependencies, not link flags? Should everything really be in _LDFLAGS like my patch has done? Has some subtle semantics of Make or the build tools changed? It’s unclear to me why this is needed, but it’s fishy.


Some suggestions

This list is incomplete, just a set of thoughts.

  • Please add a make install target. This should not rely on any particular packaging file, it should just run install on the files in the right places. Unless I completely missed it (I don’t think I did?), this is an oversight that makes 3rd party packaging much more annoying. In one particular annoyance, fdbrestore, fdbdr, dr_agent etc are all actual symlinks to one binary (backup_agent), and determine their operation based on the exe name. I didn’t realize this until I had dug in and read the source code! So instead I had to ‘extract’ the fdb .dpkg and look at the filesystem hierarchy. Then I mostly replicated that. But make install would have done it for me…

    Please note that it is essential that a command like make install respect the traditional PREFIX= and DESTDIR= environment variables, so that maintainers can install them into arbitrary places, which is vital. These environment variables do not need to have any special meaning to the FoundationDB source code, they merely need to exist, and be respected, when finally copying files around during make install. (FDB, as far as I can see, doesn’t rely so much on fixed hardcoded paths outside of /etc/foundationdb, and even that is fixable in the fdbmonitor file.) I’m sure you’re aware of it, but I only mention it because adding make install but not adding that will just have distro maintainers right back here. :slight_smile:

  • Please make libstdc++ frobnication, link-validate.sh, etc optional during build. First, on a system like NixOS, and, indeed, almost any Linux distro based around compiling packages for users, and shipping them those packages – validating symbol names is meaningless, because there is only one global glibc, and it is explicitly compiled against. There will never be older versions that mysteriously appear out of nowhere, only newer ones. Second, the particulars of frobnication aren’t the same everywhere, for example, in NixOS I just sheepishly hacked libstdc++_pic and replaced it with libstdc++ in the Makefile, because the _pic variant is an Ubuntu specific anomaly! Then I just disabled the link-validate check.

    These are surely vital for officially sanctioned static binary packages for your customers/users, but for distro maintainers, being able to turn this off would be excellent.

  • Please make the existence of .git optional at build time. Many distros and packaging systems prefer to use tarballs (with optional signatures or hashes) instead of direct git repositories, so .git is not always going to be available. This causes FoundationDB’s build to fail since it requires it. For prior projects, one way we did this was to add a sdist target to create a tarball of the source code without git, for every release. Then, put a .release-version file inside the tarball containing the actual versioning information, as part of make sdist.

  • The FoundationDB Documentation site map is missing many important pages, for example, such as the “Administration” page and “TLS” page, which do not seem to be available. The TLS page simply doesn’t seem to exist anywhere but the repo, while the Administration page can at least be found on the current documentation using the “Search” bar. But neither are linked from “Site Map”!

  • Please don’t use -Werror, anywhere, under any circumstances. In fact I suggest forgetting that flag has ever existed (as a distro maintainer, I almost wish it didn’t). :slight_smile: Ideally -Werror could be injected into the build system, for example during CI on systems you perfectly control, or FoundationDB developer machines. It’s basically always wrong otherwise, though, and using it should require a dance on part of the person doing it. I had to patch this out of ./Makefile for things to work.

    In particular, for NixOS, while the C++ compiler is GCC 4.9, i.e. old, glibc is far newer. That’s unsurprising: there can only be one glibc, but many compilers. This results in GCC throwing warnings (errors) due to more modern glibc headers than the one that the build system/CI system currently tests against. (For example, glibc headers over time have been augmented with things like warn_unused_result, meaning perfectly warning-free code can become warning-laden, without even touching the compiler, only glibc.)

    While the compiler version is fixed, you can pretty much never guarantee the glibc version is fixed. Maybe NixOS is using glibc 2.25 with GCC 4.9, while another system is only using 2.23 with 4.9. Unless you test every possible combination of a specific compiler version against an array of glibcs.

In particular, fixing these few issues would make the NixOS FoundationDB package far simpler, and it would allow me to remove several of my patches.


Testing, limitations

I’ve tested FoundationDB-on-NixOS with a 9 node cluster split 3x3 in geographically distinct data centers (Amsterdam, New York, Bangalore) with proper locality settings, in triple datacenter mode. Autocoordination automatically promoted 7 coordinators out of the 9 nodes in distinct datacenters to achieve this (odd numbers: 2 coords in 2 of these regions, 3 coords in the last region). This cluster then had inserts (fdbcli --no-status --exec 'writemode on; set fooX barX’, for many X values) performed on it repeatedly while taking nodes and coordinators offline at random intervals for random periods. (Nothing fancy to stress things, more like “I hit some commands at random intervals while watching Netflix to see what would happen”).

There are currently several limitations. The biggest one is that TLS support is NOT supported in the NixOS configuration right now. This isn’t due to a bug, just lack of time over the past weekend on my part. (The above 9-node/3-DC test was in fact encrypted – but using Wireguard as an overlay VPN in order to transparently encrypt data-in-flight. As a bonus, the wg command on any node gives a hand indication of the raw network bandwidth use between any two nodes). I expect to enable this soon after some testing of FDBLibTLS.so

The second major limitation is that only the C bindings are installed, not the Python, Ruby, Java, or Go bindings. Each of these will likely require help from fellow NixOS maintainers to properly maintain and build, as language-specific packages and bindings often require bespoke setup. I’m a C/Haskell programmer however, so I admit I’m unlikely to get to this soon without cajoling, since my needs are met…

NixOS just recently had its latest release, version 18.03, in late March. I have no plans to backport FoundationDB packages in any form to NixOS 18.03. The next release is NixOS 18.09, due around September, and I expect NixOS’s FoundationDB support will be quite featureful and ready by this time, and there is nothing to suggest its reversion. Until then, the semantics and default configurations may change!

DR mode, PITR for active clusters, recovery etc has not been tested extensively (I did not set up a second cluster in my tests), but ideally should work fine, as backup_agent seems to work fine too on my local machines.

There are probably some other bugs and problems I’ve missed, surely.


Links

[1] https://nixos.org
[2] https://github.com/NixOS/nixpkgs/blob/617db2df96a75f7808d544b57aa97d9859377e84/pkgs/servers/foundationdb/default.nix
[3] https://github.com/NixOS/nixpkgs/blob/18f28a6413e33416576f632367f0a4816c74c188/nixos/modules/services/databases/foundationdb.nix
[4] https://inner-haven.net/~aseipp/nix/fdb-manual/index.html#module-foundationdb


(David Scherer) #2

Fantastic work!

I like the principles behind NixOS a lot, at least by comparison to the insanity that is a “normal” Linux distribution.

I would guess that a PR to make newer compiler and/or boost versions work would be welcome.

I don’t think a contribution of systemd unit descriptions would be a bad idea, but realistically to include them the Apple team (who are the only ones in a position to do serious CI at the moment) will have to do some work to build and maintain tests. So I don’t know if they will be up for that.

In a perfect universe, every distro would build and package FoundationDB from source, and all of them would build byte-for-byte the exact same fdbserver binary that Apple builds and runs zillions of tests on. That would give everyone the best of all worlds. I’m not sure where this dream falls on the spectrum of challenging to impossible. It seems like it should be easier for NixOS than for most.

Failing that, running binaries from fdbserver.org will probably be the safest thing for production.

Is it really impossible to take a build dependency on a particular glibc header version? I ask because it sounds like that would resolve a number of your issues, make the build much more similar to the one that Apple is doing, and be at least a baby step toward the exact-same-binary world. It sounds like the official build is moving in the direction of building with a modern glibc and explicitly requesting old symbols as necessary, rather than building with an ancient glibc, so it wouldn’t be a matter of trying to pin an ancient version, just of trying to pin something other than the exact version of the actual .so your distribution uses.

I don’t know what’s going on with _LIBS vs _LDFLAGS; if it’s as broken as you say why does it build for anyone?

+1 on make install; I think this more or less “belongs” to packagers and maybe you should even write it and submit a PR

-0 on a build option to disable lowest-common-denominator build targeting features. I agree that this is not really necessary when the result of the build is intended to be used in the environment it’s built in. But it’s not clear to me that these things do any harm, and I prefer the ideal of everyone building the same binary unless they specifically want to do something different

+1 on make sdist or something similar

-1 on suppressing warnings during builds. I think the reasons you give why “perfectly warning-free code can become warning-laden” are, mutatis mutandis, reasons why “perfectly bug-free code can become bug-laden”, and thus why we should pin header versions. If the build is truly deterministic, only developers (and maintainers, but no one said it was an easy job) should see new warnings.

In terms of distro testing, I would make sure you are actually testing the C bindings. IIRC fdbcli doesn’t use them (it statically links the client). I guess backup_agent probably does, though.

I would think a little about what Nix “rollbacks” mean with respect to the data files. For example, it will not in general be possible to downgrade FoundationDB versions. Try to make sure that the worst case is temporary unavailability rather than, say, deleting data files or something.

I don’t speak for the team at Apple; this is all just, like, my opinion, man.

Dave


(Alex Miller) #3

Yay for a NixOS package! :tada:

Below are responses to some of your questions. The ones I didn’t answer (largely around policies), are simply because I don’t have answers.

FoundationDB build includes a copy of Boost 1.52.0

There was a recent PR which fixed the couple issues needed to support boost > 1.52.0. Our internal testing hasn’t been upgraded to use a newer boost, but you can probably fit in with your generalized boost infrastructure just fine now.

I’d be more than willing to help contribute upstream unit files

There was actually an open request for this in #241.

Is there any timeframe on support for newer GCC/boost libraries?

A PR was already merged to add support for boost >1.52.0, so if you pull master, you should be fine. I validated the build against 1.67.0.

Newer GCC/clang is a project that’s a background task of mine, so it’ll probably happen sometime in the next month or two, but I make no promises. If you have any patch (full or partial), I’d be happy to accept a PR.

So maybe my understanding of _LIBS is wrong.

Probably not. There’s probably some strange mixup of LIBS and LDFLAGS within the internals of the build system, and there’s not really anyone that enjoys touching it. I feel like I hit weird issues when trying to integrate statically linked libraries into the TLS Plugin build, but I didn’t stare at it for too long.

libstdc++ frobnication, link-validate, etc.

I think there’s an argument to be made that we should make a stronger separation between release and non-release builds. For the binaries that get offered from www.foundationdb.org, we’d like all those checks to pass, and to make sure that the binary can run on even ancient platforms. If you’re just building for your local machine, then all the extra effort likely doesn’t produce an observable difference.

I think we’d potentially want to consider having three different build modes: Developer, Distro, and Release. Developer would probably default to -g -Wall -Werror and none of the build strictness shenanigans (and this would be the default), Distro would drop -g -Werror and the dependencies on having a checkout, and release would be with all bells and whistles enabled.

I don’t really have well developed thoughts or opinions surrounding this though.

For NixOS users, I have also written a section in the manual about FoundationDB usage

I’m unclear if there’s any value in users having backup agents running by default, and I mean this as a genuine question. I suppose it might change depending on if your default install is geared towards being networked into a larger cluster or used for local testing and development.

And it’d be great if you wouldn’t mind filing issues for:

  • make install target
  • Optionally disable binary portability
  • Admin and TLS pages unfindable
  • libraries should be listed in _LIBS and not _LDFLAGS

-0 on a build option to disable lowest-common-denominator build targeting features. [snip] it’s not clear to me that these things do any harm

I think there’s a trade off here of how close to the release build a normal developer build is versus how many hoops one might have to jump through to get a building and testable fdbserver. I’d be a bit sad and disappointed if potential contributors get turned away because they don’t use the provided Dockerfile as a build image, and end up getting stuck on e.g. stdlibc++ frobnication. It seems a bit strange to me to do so much effort to support old glibc versions in a build for distro packaging, when one could easily slap on a glibc>2.14 dependency for the package instead.

That said, my view comes from not having as clear of a picture of what our distro story should be as your perfect universe outlines. If the goal is for everyone to generate the byte-for-byte same binary, it seems like enforcing Docker as The One True Way To Build would be a good idea.


(Austin Seipp) #4

I don’t think a contribution of systemd unit descriptions would be a bad idea, but realistically to include them the Apple team (who are the only ones in a position to do serious CI at the moment) will have to do some work to build and maintain tests. So I don’t know if they will be up for that.

In practice I guess it sort of depends on the scope of the unit file in question. Things like security enhancements generally need to be carefully reviewed by developers so they don’t impact functionality in some unforseen way; for example, filesystem path restrictions. On the other hand, the basic logic behind the current init script is very easy to replicate (maybe 10/15 lines of a unit file, which is much more readable/reviewable than init.d scripts) and probably would be low impact.

There is the argument that actually running under systemd could introduce other issues; that’s possible, but realistically I’m going to guess you would have found them by now: in practice on every modern Linux distro (RHEL 7, Ubuntu 16.04 and beyond) you are actually running FoundationDB under systemd already, it’s just the init script is run as a compatibility measure so you don’t see it. But the old style stuff is just for compatibility.

Failing that, running binaries from fdbserver.org 4 will probably be the safest thing for production.

So, it’s actually possible using some magic to install non-NixOS compiled binaries on NixOS. This is generally frowned upon when source is available however, for a number of reasons (including source auditing, better package integration if patches need to be applied to work around small deficiencies like FHS differences, better reliability since failures can often happen in a test suite or at compile time, e.g. from dependency updates, rather than at runtime in odd ways, etc). I imagine most Linux distros will adopt this exact same stance.

There is a policy in NixOS where some packages actually do have two versions: the NixOS compiled version, and a binary package directly from the upstream developers. For example, there are two variants of Firefox under this distinction (one compiled by us, another from mozilla.org). It’s unclear to me if this is the right path for FoundationDB on NixOS. I am not sure about other distros, but I think some of them have similar policies.

In a perfect universe, every distro would build and package FoundationDB from source, and all of them would build byte-for-byte the exact same fdbserver binary that Apple builds and runs zillions of tests on. That would give everyone the best of all worlds. I’m not sure where this dream falls on the spectrum of challenging to impossible. It seems like it should be easier for NixOS than for most.

So there are two things here, from my POV:

  • Can we achieve deterministic results in NixOS, or <arbitrary distro XYZ>?
  • Can we achieve deterministic results that are the same as the Apple team would have?

The answer to 1 is “yes, and fundamentally NixOS is very well designed for this, but it’s still ongoing work” (you only need see the work Debian, etc have paved the way with – reproducible-builds.org). Many others are also working towards it. NixOS has perfect reproducibility for a limited subset of things (including the kernel for a while now, I think) but there’s always more things to work on…

The answer to 2 is “impossible”. In fact, it’s more impossible for NixOS than other distributions, despite its design allowing perfectly hermetic, controlled build environments. Why? Because NixOS does not have a standard FHS layout, like 99% of all Linux distributions. Most crucially, this also applies to things like dynamic libraries.

This means that, barring every other possible technical challenge, the binaries can never be the same. You can trivially see this by just running ldd on fdbcli:

root@link> ldd $(which fdbcli) | grep ld
        /nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib/ld-linux-x86-64.so.2 => /nix/store/27x7pinqdsl9f3rpbm8bsszd9fhwq266-glibc-2.27/lib64/ld-linux-x86-64.so.2 (0x00007f5133319000)

That is, the dynamic linker ld-so is fundamentally in a different place than any Linux distribution. And things like this are part of the ELF header; it’s also not possible to “quotient” these changes out of the binaries in general since any string, in general, can affect buffer layout, alignment in the resulting object file, etc etc.

So any attempt to get bit-for-bit reproducibility is DOA, I’m afraid. That said, many Linux distributions are converging on reproducible builds, a la point 1. So what’s more likely is that individual distros will each be able to guarantee their FDB builds are fully deterministic, as well as the official FDB packages by the Apple team are. But none of these will match. Thus, it’s more like a family of reproducible binaries, rather than a singular one.

(The very short version of this for bystanders is that: when Nix compiles something, it hashes the thing it’s going to compile, as well as all the input dependencies to compile it. The results from the build are then installed into a directory which is keyed by this hash (27x7pinqdsl9f3rpbm8bsszd9fhwq266, in the case of glibc-2.27). This is how coexistance works; you can even have two copies of glibc 2.27 installed, for example, if one had a patch applied and the other didn’t – because they would result in two separate hashes.)

Is it really impossible to take a build dependency on a particular glibc header version?

It’s not impossible in theory, just more work and nothing in NixOS really does this right now as it stands, I think. NixOS could do it easier than most others, again, but even despite effectively being superpowered, we have limits to what we accept (or feel is acceptable). I think many people would frown upon it relative to just patching out -Werror (or whatever). Plus, all of these hacks do at some level have ongoing costs for someone (for example, me, and anyone else who helps maintain the packages, or you, the developers); it’s mostly just a matter of where the complexity should go.

For things like this, I’d say (with my maintainer hat on, not a developer one) that issues like this are best resolved upstream for FOSS projects. This is so that the actual fixes are localized, and so half-baked fixes do not proliferate among downstreams, or at least for no longer than absolutely necessary. This is, IME, the majority view for most distro maintainers, who tend to get involved upstream.

So there’s a dynamic that people are going to author packages, even without explicitly detailing everything as I’ve done on the forum. Some will probably even use my work as a basis; but their tools may not be as good. So vendoring glibc headers is way harder, or they have to do workarounds like this will proliferate among downstream users. There are a lot of negatives to this, that may not align with your vision of the project. But it will also require maintenance on your part. Drawing that line is hard, and it’s not even a technical problem, all the actual technical specifies are pretty easy otherwise…

I don’t know what’s going on with _LIBS vs _LDFLAGS; if it’s as broken as you say why does it build for anyone?

My notebook just has the word “GHOSTS” scribbled on a single page with a red marker in big, capital letters, and I’ve nothing else to go on right now I’m afraid.

-0 on a build option to disable lowest-common-denominator build targeting features. I agree that this is not really necessary when the result of the build is intended to be used in the environment it’s built in. But it’s not clear to me that these things do any harm, and I prefer the ideal of everyone building the same binary unless they specifically want to do something different

At the end of the day though I don’t really think any downstream ever will get the same binary (for reasons alluded to above), but I can understand this point in the sense anyone who checks out the code should use the same process you do.

-1 on suppressing warnings during builds. I think the reasons you give why “perfectly warning-free code can become warning-laden” are, mutatis mutandis, reasons why “perfectly bug-free code can become bug-laden”, and thus why we should pin header versions. If the build is truly deterministic, only developers (and maintainers, but no one said it was an easy job) should see new warnings.

Mmm, following the last point – there could be a basic trade off here, right? In particular, rather than patching a Makefile which is a bit annoying and error prone, maybe there could be a file like build/local-build.mk that is optionally sourced by the Makefiles with some specific settings? And the default could be the current settings.

For example, by default -Werror could be enabled, but you could alternatively let the user do something like:

$ cat 'EXTRA_CFLAGS = -Wno-werror=<some minor warning that is benign>' > build/local-build.mk
$ make # build as usual

This would mean overrides are opt-in. So I could just do this instead in the build expression for NixOS, rather than sheepishly hack around it blindly. Thoughts?

In terms of distro testing, I would make sure you are actually testing the C bindings. IIRC fdbcli doesn’t use them (it statically links the client). I guess backup_agent probably does, though.

I do have some small app I’m working on that seems to work fine but admittedly I’m currently writing some C++ wrapper code that’s distracted me a bit. :slight_smile: I’ll post any issues I might have later.

I would think a little about what Nix “rollbacks” mean with respect to the data files. For example, it will not in general be possible to downgrade FoundationDB versions. Try to make sure that the worst case is temporary unavailability rather than, say, deleting data files or something.

In general this is no different than any other database; Nix doesn’t fundamentally change the fact you have to do migrations carefully, it just gives you tools to ensure careful rollback of system packages – but not necessarily the data. Backups, etc etc etc. But the vast majority of rollbacks are really something like “I accidentally mistyped a config file parameter and the server got mad and errored out on startup, so I rolled back”. In general any data administrator will have to be careful here, though…

But I’ll be sure to test this and write something in the manual to this effect; “How to upgrade FoundationDB”.

Also we should certainly support multiple major release branch versions of FDB in NixOS, and we already do this for most database packages. For example, we’ll certainly want multiple foundationdb51 for 5.1.x, foundationdb60 for 6.0.x branches etc, and in the NixOS configuration users pick what package they want – so users can carefully control what version they use, and where they’ll upgrade it, independent of the actual system (this is how Postgres works for example, in NixOS, and some people still even use PostgreSQL 9.3 happily, with the very new-and-latest NixOS versions). So I’ll put this on my TODO list.


(Alex Miller) #5

I put some thought into this last night, and this is where I ended up as well. I think having a developer-oriented package that builds from source and sets up a local FDB instance to use is great. It’d be good to have an additional package (or option on the former package) that just downloads the binary from foundationdb.org and sets up the configuration, with a note that the binary package is preferred for production use because the precise binary being used has going through a stricter QA process.

I think this could be interpreted better as the .text sections of the ELF files would be identical, and not the entire ELF file itself. I believe that works around the counter-arguments that you laid out above.