What's new since 3.0.7 for the low level client API?

So… that’s interesting :slight_smile:

I am of course very eager to resume development for the .NET Binding (https://github.com/Doxense/foundationdb-dotnet-client), but would be interested to know what’s new and/or changed since the last public version (3.0.7)?

Is the API mostly unchanged?

The API has not changed much.

You will need to integrate with fdbclient/vexillographer to automatically generate options based on the fdb.options files.

You may also want to integrate with our bindings/bindingtester, which runs the same set of operations in different bindings and checks that they produce the same result.

The most significant new API change is the addition of version stamp operations. Version stamp operations will put the commit version and batch number of a transaction into a key or value. Check out the atomic operations documentation of any language binding for more details.

Basic support for version stamp operations will happen automatically when you are integrated with vexillographer, however we also added an API call for getting the version stamp of a transaction after it has committed (fdb_transaction_get_versionstamp).

I just wanted to say hi, Christophe! Long time no see.

Happy to be back :slight_smile: I will be able to delete so many code that was attempting to do what I can now do in 10 lines of code

I assume you’re referring to the C API which you had been wrapping, correct? Most of the API changes made here since 3.0.7 were implemented via options and atomic operations, which is why Evan mentioned that you’ll need to integrate the changes from fdb.options. Some more notable examples include the multi-version client (https://apple.github.io/foundationdb/api-general.html#multi-version-client) and versionstamp operations. Adding versionstamps to keys and values is done through atomic operations, but there was also a function added to the API (fdb_transaction_get_versionstamp) to get the versionstamp of a transaction after it’s been committed.

We added a couple other new functions as well. One is fdb_error_predicate which allows you to test whether an error is of a certain type (e.g. retryable). Another (fdb_get_client_version) returns a string containing version information of the fdb_c client. We don’t typically expose this latter one through our bindings, but rather use it to support the multi-version client.

The above functions and options are documented at https://apple.github.io/foundationdb/api-c.html, with the exception of fdb_get_client_version.

We’ve also had a couple behavior changes to our existing API. The two that immediately jump to mind are that transactions no longer reset after commit and that the AND and MIN atomic ops are now equivalent to a set if performed on a key not in the database. A more careful combing through of the release notes (https://apple.github.io/foundationdb/release-notes.html) should yield any other changes if I’m forgetting any.

Additionally, though this is somewhat orthogonal, we added new types to the Tuple class, so a fully-featured binding that worked at 3.0.7 should probably update to include the new types (float, double, UUID, boolean, and nested tuple), though it isn’t strictly required. We document our tuple types and the on-disk protocol here: https://github.com/apple/foundationdb/blob/master/design/tuple.md

I think I already added most of the new tuple types (I remember requesting the addition of the UUID type, and also talking about storing doubles). I definitely remember the double take on the embedded tuples (requiring handling nulls as 00 at the top, or 00 FF when embedded).

Unfortunately, it seems that we diverged slightly after that, since my embedded tuples (with 00 FF encoding) are still using the old 03/04 prefix/suffix, while it seems they are now using 05.

I also see that false/true are mapped to 0x26/0x27 while at the time they were encoded as integer 0 or 1. Is this a “recent” change as well?

  1. The multi-version client seems a bit more involved, since at the time there could only be one network thread per process, so the whole API is defined as a static class. With the notion of multiple possible version, does this need that I would potentially need to have multiple “FDB” class, each created with a desired API version? (new Fdb(version: 300) vs new Fdb(version: 500))

I’m not exactly sure if a single application would ever need to use multiple version at the same time? I could see a cluster update happening while an application is running, but I feel that if the protocol has changed, it means that the app would at least need to be updated? (I guess I’m missing the use case here).

  1. Versionstamps are looking very usefull. Are there any details on them? I’m seeing the java implementation (https://github.com/apple/foundationdb/blob/master/bindings/java/src/main/com/apple/foundationdb/tuple/Versionstamp.java) but is scare in detail for now. Is the only difference between the 80 bits and 96 bits version the two additional bytes at the end? I guess I will need to see them in action to get a feel.

I’ll probably need to create a custom struct to represent this type in .NET because 80 bits is somewhat in between two chairs (it would be a waste to use a Guid or Decimal that consumes 128 bits…)

  1. The transaction does not reset after commit is indeed a thing to know! :slight_smile: Do you think that it would be the role of the binding to do it automatically? Or do you wan’t to force consumer of the API to know about this? (not sure how this was handled by other bindings). Is this something that is dependent on the API version selected? (ie: 200/300 reset the transaction, 400+ does not)

  2. I looked at the vexillographer code, but I guess I would need to the see final generated code for java or python, in order to get what it’s doing. Looking at code that generate code is not always ideal :slight_smile: (And I see the irony of using a .NET project to generate Java binding code :wink: )

1 Like

I noticed the UUID type is now officially in the spec as well, instead of a placeholder

I presume 128 bit big-endian GUID with no requirement for 0x00 encoding etc?

Also there appears to be 128bit IEEE double as well

The 80 bit version stamp threw me too. I’d love to hear the background on the precision decision as well please.

You should be careful with the 128-bit UUID: it was designed with Python and Linux in mind at the time, and you could be surprised to know that the layout of System.Guid in .NET is NOT compatible! (and also not lexicographically ordered). I had to create a custom Uuid128 and Uuid64 type in .NET that is compatible with the RFC and with all the other FDB bindings for proper interop.

You should probably check that your favorite language respect the proper ordering between the text representation and the binary representation of your ‘guid’ type. A list of random uuid should sort the same way either in text format or in binary format (this was not the case with .NET’s System.Guid)

1 Like
  1. The main purpose of the multi-version client is to support upgrades from one version to another without downtime. You may remember that the cluster and fdb_c client versions must have matching protocol versions for them to communicate, and the multi-version client allows the clients to essentially preload the new client version before the cluster version is changed.

A client would load both versions of the fdb_c client (the one pre and post upgrade) and connect to the cluster, and when the cluster changes version the client automatically switches to using the upgraded version. Even with this feature, you can still only use one API version.

All of the logic for supporting this occurs in the fdb_c library, so all you should need at the binding level to support it is the relevant new options. And in fact, you technically don’t even need that because network options can now be set by environment variable as well.

  1. You should be able to find documentation of how the versionstamp operations work on one of the API reference pages (e.g. https://apple.github.io/foundationdb/api-c.html). Specifically, there are capabilities to set versionstamped keys and values as well as the new C function I mentioned that lets you know the versionstamp of a committed transaction.

  2. I believe we decided to change this behavior because of the possibility that a stray operation on a transaction might accidentally end up occurring after the commit, leading to issues with conflict ranges or mutations not applying correctly. Our bindings all follow suit and don’t reset the transaction after commit. This behavior is API versioned, so a transaction at an old API version will still reset the transaction.

  3. For what it’s worth, we had been using vexillographer back in 3.0.7 and previously as well. While we recommend using the fdb.options template to generate your options and atomic operations, etc, you could continue manually implementing the options as before.

Unfortunately, it seems that we diverged slightly after that, since my embedded tuples (with 00 FF encoding) are still using the old 03/04 prefix/suffix, while it seems they are now using 05.

Yeah, that’s right. We went with 05/00 instead of 03/04 because the 05/00 sorts prefixes correctly. For example, (("hello",),) would serialize to \x03\x02hello\x00\x04 and then (("hello", "world"),) would serialize to \x03\x02hello\x00\x02world\x00\x04, which means (("hello", "world"),) would sort before (("hello",),) when serialized. Using \x00 as the terminator fixes that problem.

I also see that false/true are mapped to 0x26/0x27 while at the time they were encoded as integer 0 or 1. Is this a “recent” change as well?

Yeah, that was introduced in release 5.0. In principle, an application can choose to use 1 and 0 to mean true and false, and everything would still work. The extra type lets them specify boolean-ness in a type safe way (and saves a byte on “true”, for what that’s worth).

Is the only difference between the 80 bits and 96 bits version the two additional bytes at the end?

Yep, that’s the only difference. The extra 16 bits let the user specify an order of operations within a transaction, and the first 80 bits are set by the database to enforce an order between transactions. So the 96 bit versionstamp can be used to globally order things across all transactions in a way that is consistent with a valid serialization order.

I’ll probably need to create a custom struct to represent this type in .NET because 80 bits is somewhat in between two chairs

Sure. As I’m sure you’ll find digging through the source, there is also some work needed to be done on the binding implementor’s part to handle cases where the versionstamp hasn’t been set and will be passed into a versionstamp mutation. I believe we refer to those versionstamps as “unset” in our API.

How did you handle the data migration when you changed the tuple format? For example, when you changed ‘true’ from 0x25 to 0x27, how did you handle migrating applications that used 0x25 before? Full backup and restore? Or did you had custom code at the tuple encoder level to “stick” with 0x25 for old keys, and 0x27 for new ones? (this would be a nightmare)

That is not a backwards compatible change indeed… We’ll have to copy the old class and keep it around (and somehow have precommit hooks to make sure the new one isn’t accidentally used).

I assume this is just an artifact of starting new?

Did nested tuples exist in the standard bindings in 3.0.7?

The changelog for version 5.0 says

"Tuples now support single- and double-precision floating point numbers, UUIDs, booleans, and nested tuples. "

which implies that these were added to the standard binding for the first time. So there was no migration, and there shouldn’t be a compatibility problem for anyone except Christophe ( and .NET users ).

It’s probably my fault - Christophe ran his designs for these things past me long ago (and in some cases I suggested designs to him), and I wrote them down so that if/when we implemented them later we could be compatible with what he’d done. But I probably didn’t catch the nested tuple ordering problem, and when the team went to actually implement nested tuples they (or their test infrastructure) did notice.

A small change I’m seeing in unit test: previously fdb_get_error would return strings like "success", "operation_failed", etc… (lower_case), but now it is returning textual strings "Success", "Operation failed", …

I’m don’t suppose this would break a lot of things, since this is usually used for the Description or Message part of exception thrown, but it does trigger some tests that were checking the content of exception messages.