There has already been some discussion about adding new bindings, as well as a few projects or requests for bindings in languages other than the ones currently offered in the foundationdb repository. I wanted to write up what, from my perspective, is the work that a binding-writer would have to do to get a fully-working binding implemented as well as what work could be done to make the foundationdb repo more amenable to interacting with bindings that live outside its tree. (This is intended to be separate from discussion related to which bindings should live in-tree and which should live outside the repo.)
I believe this to be the work required:
-
Wrap the C API. Our C âbindingsâ are intended to provide a bridge between HLLs and our native code (and are in that sense distinct from other bindings which are intended to be used by layers or applications to write more interesting things, hence the scare quotes). For the most part this is fairly straightforward (for example, writing a âTransactionâ interface that makes calls to all of the C functions beginning with âfdb_transactionâ with the appropriate arguments). The most difficult one is probably âget_rangeâ, as there is logic that must exist in the bindings to do things like pipelining requests, etc. It is also probably necessary that a third party bindingâs build system somehow get access to the FDB header files. We publish those within our client packages (look inside
/usr/local/include/foundationdbon macOS) and they are in our repo (except forfdb_c_options.g.h, which is generated by our build process), but maybe there should be an easier way to get them. -
Integrate with fdb.options. There is a file within the fdbclient directory called fdb.options. It defines a few different options that can be used to tweak transaction/database/network behavior, and it is here where we define which kinds of atomic operations are supported. In theory, one could hand-code all of these things, but it would be a pain, and it would be a nuisance to update the hand-written code every time there was a new atomic op or a new option. The bindings living in tree for the most part use a tool within the fdbclient directory called âvexillographerâ which generates a few different files for each binding. The go bindings read the fdb.options file directly and create generated.go using translate_fdb_options.go, which might be closer to the approach that a third-party binding might want to use. One would of course also need access to
fdb.optionsand possiblyfdb_c_options.g.h, but that could probably be accomplished the same way that the binding gets the header files (see above). - Implement the Tuple, Subspace, and Directory Layers. These are somewhat optional in that you can use FDB without these layers, and everything will work, but the impetus for including them in every layer is that they are common enough constructs that many applications will want them, and it is nice if things like data serialization is consistent across languages so that you can use tools written in one language to inspect what is done in another. (It also means that you arenât locked into one language or the other, so you could change your stack without the underlying on-disk representation needing to change.) The subspace layer is fairly straightforwardâitâs essentially âadd a byte-array prefix to keysâ with some helper methods. The tuple layer is the suggested way for serializing data in a manner that preserves ordering across the serialization boundary. We have a specification on the different supported types and how they should be serialized. The directory layer is essentially a string-interning thing to turn long paths into short byte prefixes. Itâs somewhat complicated, so perhaps the best thing to do is look at an existing implementation, barring more documentation or a more formal specification.
-
Write tests for the bindingtester. We have a randomized tester, called, um, well, the âbindingtesterâ. Each binding implements a stack-based tester that essentially reads a specification of operations to do from the database, and this lets us test to make sure that all of the bindings do the same thing. We have specifications on what the tests should do within the spec subdirectory of the bindingtester. Our current bindings are integrated into the bindingtester through the âknown_testers.pyâ file. Itâs possible that there is some work needed to make it easier for third-party bindings to be found and run, but itâs also possible that the easiest thing for a binding-writer to do (at least initially) is to add their binding to the
known_testers.pyfile and see if things work. The bindingtester can be invoked by running:python bindingtester.py ${tester_name}, and it has additional arguments to allow for reproducible tests, compare against another binding, etc. It is a randomized tester, though, so to really get confidence in the tests its best to run it many, many times.
I think thatâs the overview of what it takes, but perhaps Iâm missing something or eliding details in a way that makes this hard to follow. But I hope this helps.
