I maintain the nodejs foundationdb bindings. As the API evolves and changes, I’m trying to figure out what I should be setting FDB_API_VERSION to in the header.
It looks like:
Setting it to a low value will restrict what API features my bindings can expose
Setting it to a high value (eg 600) will stop my binding building without a new version of the foundationdb client library; which might be a pain for users. (And can you even use an older libfdb_c.dylib with a program that uses a newer header version?)
Obviously for the in-tree bindings (C, python, java, go) the header version tracks the current version. But what about for bindings like mine?
And each binding seems to be following a different strategy:
As you noted, the in-tree bindings release a new version as part of each new major or minor release of the software. Each of these has had a corresponding bump of the API version, although this isn’t strictly required if the API hasn’t changed at all. The expectation then is that users will use a version of the bindings that is less than or equal to the version of their native client. Or put another way, users can upgrade their bindings once the cluster had been upgraded.
Also, it is possible to use old clients with new bindings via the multi-version client (and in so doing, you could potentially connect a newer binding version to an older cluster). However, this does require that you also have a sufficiently new version of the native client.
Right so, if I use API version 600 in my node foundationdb bindings, someone using an older version of foundationdb can use my library provided:
The node fdb bindings are compiled against version 6 of the library
They have a copy of libfdb_c version 6.0.x alongside the version of libfdb_c they need to actually connect to their cluster
Is that right?
And if I follow the lead of how the swift bindings work and use header version 500, then:
Wouldn’t need a more recent libfdb_c
But they also wouldn’t be able to use any of the newer API features (like offsets in value versionstamps)
And if I make it configurable through the call to fdb_select_api_version_impl then I might run into dynamic linker errors or undefined behaviour if / when the C API actually changes
What I’m hunting for here is a recommendation / best practice for out-of-tree bindings. Thoughts?
That all sounds right. You could certainly follow the same approach as the in-tree bindings, offering releases at different versions with the latest API header version used as of that release. In fact, this is what we did historically with the Go bindings when they were maintained in a separate repository.
I’ll have to defer to the maintainers of the other bindings to provide their experiences with the other approaches.
Yeah thats what I’m doing so far; but the problem is that if you’re using an older version of the foundationdb cluster code you don’t get unrelated new features / bug fixes in the bindings.
In big setups its pretty common to migrate database version infrequently. So its annoying if that results in people getting stuck on old, buggy versions of my bindings code. I guess so long as they don’t mind keeping a couple copies of libfdb_c around its not so bad.
That also provides more justification for automatically downloading an appropriate version of libfdb_c from the internet if its not found locally / its an old version. That is, so long as you have internet access we should be able to fix these sort of issues automatically.
The assumption at the time the corresponding code was written in the C# binding (v1.x?) was that you would update the binding alongside the client library anyway (since there was a fast rate of release and new features at the time). And during the long hiatus, there were no new versions anyway so this wasn’t an issue.
This may be less true now that the code is more mature and that the rate of release as slowed down a bit.
The reason the .NET binding calls fdb_get_max_api_version and pass it back as the header version is because it would represent what the current client library was loaded at runtime. If the binding was newer than the client library, it would have to be careful and only call the old APIs, and if the binding was older than the client library, then the client library would handle emulating the older API surface.
I vaguely remember talking about that with someone a long time ago, but I can’t be sure if this is what was recommended to me, or if it’s something I improvised on the spot
What do you mean by that? Do you require the correct header files at compile time? The .NET binding does not have this issue (dll is loaded and “inspected” at runtime) so that may be why I’m not having the same issues as you and went this route? Do you have any other way to interop with the library in a more “dynamic” way?
I agree, and this is a long-standing debate At the time, I was not able to ship the client library with the binding because of the closed license, but this is no an issue anymore.
The downside to that is that if the FDB C api gains / loses function arguments, your API won’t work properly. But maybe thats not a big issue with the way C# is calling into the library.
But that said, the reality is that foundationdb hasn’t added or removed any function arguments from the C API in awhile. I could probably get away with doing something similar to the C# bindings and relax the allowed version range through a whitelisted set of versions. (Eg, make it work with any libfdb_c from 5.0.0 up to 6.0.0 by querying the library version and using fdb_select_api_version_impl). But when arguments are added / removed, we’ll need a hard break. (Suddenly instead of 5.0.0-6.1.0 users will need exactly version 6.2.0 after an update). But I could probably get nodejs to download that from the website if its missing at startup before loading the native module.
I really like your idea of a version API. It would make this stuff a bit easier. But I feel like right now this is a mess. It makes sense in a world where the bindings are is tightly bundled with the installed version of foundationdb; but with the bindings in a separate repo there’s too many versions floating around:
Versions of your local fdb_c dynamic library (the one you link, as well as any others accessible to the multi-version API)
Version of the node/c#/whatever bindings library
API version (the one passed to fdb_select_api_version)
Header version (#define FDB_API_VERSION 600 or set via fdb_select_api_version_impl)
Version of the FDB cluster you’re connecting to
Each of those has its own rules about what other versions it needs to match. (Eg, what API version range does the C# bindings support?) Its way too complicated.
Ironically no! Nodejs bindings are much less dynamic than they are in C# and ruby. This is because:
Javascript code only has access to the main thread, so even with dynamic linking we still need a non-javascript fdb future → JS adapter
Javascript doesn’t have destructors, so we need to hook into V8’s GC for cleanup. This is only possible from C.
There’s a remarkably involved chunk of c++ code needed to make the nodejs bindings work. Most of this code needs to be rewritten against the new n-api at some point too. This code is compiled against foundationdb by the node-fdb CI process for windows/mac/linux on nodejs 8/10/11 and all 9 of those version pairs are bundled in the node package. If you’re on a different OS (eg freebsd), or a different nodejs version (6, or electron) it’ll compile the native code instead on package installation. (Which requires the fdb header files and a working local compiler toolchain).
At the time, I was not able to ship the client library with the binding because of the closed license, but this is no an issue anymore.