Foundationdb FDB_API_VERSION version from bindings

Hi!

I maintain the nodejs bindings for foundationdb. Because the nodejs bindings are maintained and published out-of-tree, there’s a decoupling between which version of the node bindings people are using and which version of the foundationdb database people are using. So versioning is tricky. There are 4 different version numbers in play:

  • The version of the foundationdb database (server and client). Eg 6.2.15
  • The version of the nodejs foundationdb library. Eg 1.1.2.
  • The ‘runtime API version’ the client selects
  • The ‘header API version’ which node-foundationdb is compiled against

Every time I come back to this, I’m always confused about how those numbers all interact. For example, the foundationdb library sets the C “header version” via FDB_API_VERSION. How is this related to the application version?

Looks like runtime version <= header version, and header version <= libfdb_c.

We recently added support for API version 630 to the nodejs library, which included updating the header version (via #define FDB_API_VERSION 630). Unfortunately, this has caused problems for users running older versions of foundationdb.

Q:

What header version should I be setting in the node foundationdb bindings?

Should I:

  1. Tell users to match node-foundationdb with the version of the foundationdb database they have installed? This is unfortunate coupling. I don’t have the time or interest in maintaining a tableau of versions of the bindings for each version of the fdb API. Is there a chart anywhere showing which versions will work with which other versions? I’m happy to set a minimum supported fdb version for node-foundationdb, so long as its something reasonable.
  2. Use fdb_select_api_version_impl and force the issue? Would this cause weird issues?
  3. Be pessimistic with API versions, and just wait 6 months - 1 year before supporting new API features in node-foundationdb, to maintain more compatibility with older database versions?
  4. ???
1 Like

The header version declares what API your binding code expects from the fdb_c native client library. You’ll need this to match the latest version to get some of the latest changes, but that also means you can’t use it with old client versions that don’t expose the newest API. The runtime version is intended to determine what API the client application expects from the client library (and possibly from your bindings, if you want to use it that way). The idea is that an application declaring API version X should continue to work the same way if you upgrade your database or your bindings.

The approach that the official bindings take is that a new version of the bindings is released with each new FDB version. The new bindings can only be used on the new FDB version because of the header version requirement.

If you don’t want to manage multiple versions, your other ideas could be made to work. You could have your bindings stick with the oldest version that you want to support and delay introduction of new API-version dependent changes.

You could also try to pick a header version dynamically, but that might require you to have some extra logic to deal with changes in the underlying library behavior when you do. For example, if a C function signature were to change, you would have to call it with the correct arguments depending on the version you chose. This is going to be easier to accomplish in some languages than others.

Sorry I didn’t reply earlier. I hear all that. I understand the design and the reasons behind it. I knowing how it works isn’t enough to solve the problem I’m having. This just isn’t a decision I want to be making on behalf of my users. Essentially my choice as a library author is to either only support recent API versions (and break backwards compatibility regularly). Or support old versions and delay support of newer features by years. Either option is vaguely mediocre.

I feel like there’s got to be a 3rd choice here somewhere. But I’m not seeing it, and it’s unfortunate.

Yes, I could solve this through sheer willpower and man hours, maintaining testing and support for every fdb version. But this is all volunteer work and I don’t have the bandwidth to do that.

I hear what you’re saying, and I agree that this design requires non-trivial effort to support multiple versions of the database. I think a big part of the issue here is that our protocol between the bindings and FDB is defined using a C interface, which is not the most flexible or conducive to being modified. It is technically possible to write a client that dynamically chooses a header version for the C API (the multi-version client does this using dlopen and function pointers), but it is often rather painful to do and maintain.

If we had a better protocol, doing this dynamically could be quite a bit easier. One way to accomplish that would be via a proxy layer that provides some sort of flexible RPC interface. I saw a post a while back about an implementation of this idea, which possibly could be leveraged. I’m not sure if there are other similar projects being worked on in this vein that could help. You could even imagine having a very simple layer run locally per native client that wrapped the C API and exposed it in a more usable manner. The layer would have the same version coupling concerns you mentioned, but perhaps with a smaller amount of logic that maintenance would be easier than doing it for each complex language binding.

For the native client itself, I’m not aware of any plans that would significantly change its API to help here. If you had any ideas that you’d like to be considered, though, we’d be happy to hear them.

Maybe a better approach for the node-foundationdb bindings would be to have a separate npm module for each API version. (Eg foundationdb-620, foundationdb-630, and so on). Each version could be maintained in a branch and then gradually sunsetted over time. The old api versions will keep being forward-compatible with new versions of foundationdb. But new api binding versions don’t work with old versions of fdb.

Then new features (& new tests) only get added to the latest version. It’d be a bit of a maintenance & deployment headache but it should work well enough for users.