FoundationDB

Creating New Bindings

bindings

(Alec Grieser) #1

There has already been some discussion about adding new bindings, as well as a few projects or requests for bindings in languages other than the ones currently offered in the foundationdb repository. I wanted to write up what, from my perspective, is the work that a binding-writer would have to do to get a fully-working binding implemented as well as what work could be done to make the foundationdb repo more amenable to interacting with bindings that live outside its tree. (This is intended to be separate from discussion related to which bindings should live in-tree and which should live outside the repo.)

I believe this to be the work required:

  • Wrap the C API. Our C “bindings” are intended to provide a bridge between HLLs and our native code (and are in that sense distinct from other bindings which are intended to be used by layers or applications to write more interesting things, hence the scare quotes). For the most part this is fairly straightforward (for example, writing a “Transaction” interface that makes calls to all of the C functions beginning with “fdb_transaction” with the appropriate arguments). The most difficult one is probably “get_range”, as there is logic that must exist in the bindings to do things like pipelining requests, etc. It is also probably necessary that a third party binding’s build system somehow get access to the FDB header files. We publish those within our client packages (look inside /usr/local/include/foundationdb on macOS) and they are in our repo (except for fdb_c_options.g.h, which is generated by our build process), but maybe there should be an easier way to get them.
  • Integrate with fdb.options. There is a file within the fdbclient directory called fdb.options. It defines a few different options that can be used to tweak transaction/database/network behavior, and it is here where we define which kinds of atomic operations are supported. In theory, one could hand-code all of these things, but it would be a pain, and it would be a nuisance to update the hand-written code every time there was a new atomic op or a new option. The bindings living in tree for the most part use a tool within the fdbclient directory called “vexillographer” which generates a few different files for each binding. The go bindings read the fdb.options file directly and create generated.go using translate_fdb_options.go, which might be closer to the approach that a third-party binding might want to use. One would of course also need access to fdb.options and possibly fdb_c_options.g.h, but that could probably be accomplished the same way that the binding gets the header files (see above).
  • Implement the Tuple, Subspace, and Directory Layers. These are somewhat optional in that you can use FDB without these layers, and everything will work, but the impetus for including them in every layer is that they are common enough constructs that many applications will want them, and it is nice if things like data serialization is consistent across languages so that you can use tools written in one language to inspect what is done in another. (It also means that you aren’t locked into one language or the other, so you could change your stack without the underlying on-disk representation needing to change.) The subspace layer is fairly straightforward–it’s essentially “add a byte-array prefix to keys” with some helper methods. The tuple layer is the suggested way for serializing data in a manner that preserves ordering across the serialization boundary. We have a specification on the different supported types and how they should be serialized. The directory layer is essentially a string-interning thing to turn long paths into short byte prefixes. It’s somewhat complicated, so perhaps the best thing to do is look at an existing implementation, barring more documentation or a more formal specification.
  • Write tests for the bindingtester. We have a randomized tester, called, um, well, the “bindingtester”. Each binding implements a stack-based tester that essentially reads a specification of operations to do from the database, and this lets us test to make sure that all of the bindings do the same thing. We have specifications on what the tests should do within the spec subdirectory of the bindingtester. Our current bindings are integrated into the bindingtester through the “known_testers.py” file. It’s possible that there is some work needed to make it easier for third-party bindings to be found and run, but it’s also possible that the easiest thing for a binding-writer to do (at least initially) is to add their binding to the known_testers.py file and see if things work. The bindingtester can be invoked by running: python bindingtester.py ${tester_name}, and it has additional arguments to allow for reproducible tests, compare against another binding, etc. It is a randomized tester, though, so to really get confidence in the tests its best to run it many, many times.

I think that’s the overview of what it takes, but perhaps I’m missing something or eliding details in a way that makes this hard to follow. But I hope this helps.


Swift language support
(Will Wilson) #2

Looks like Rust bindings are underway here:


(Henri Asseily) #3

I created way back in '13 Perl bindings for FoundationDB, the code is in CPAN.
I’ll see how much work needs to be done to upgrade to the newly-returned-to-us FoundationDB.
My version of the bindings used FDB_API_VERSION=21 :slight_smile:

The Perl bindings are here: https://metacpan.org/pod/Fdb


#4

Is anyone working on elixir bindings?


(David Scherer) #5

API version 21 should still work! But I would highly recommend getting up to date anyway :-)


(Henri Asseily) #6

No :slight_smile: It won’t compile with the current headers, some of the constants don’t exist any more. It fails with a use of undeclared identifier ‘FDB_DB_OPTION_DUMMY_DO_NOT_USE’.


(David Scherer) #7

Yeah, if the binding builds with the C header (as opposed to something like Python ctypes) you would need an old version of the C header file to build (the C header by definition only supports a particular “header API version”), but then it ought to work in principle. The current source supports API versions down to 13!

Also although I have no recollection of what that is, it sounds like referring to an identifier with DO_NOT_USE in its name is a recipe for trouble :-)


(Henri Asseily) #8

Ok good to know :slight_smile: I’ll start work on upgrading to the latest as soon as work permits, and filter out anything DO_NOT_USE.


(Henri Asseily) #9

Ok basic tests are passing on API version 510, I committed version 0.3 of the Perl bindings at:

https://github.com/hasseily/Fdb