# Implementing VersionStamps in bindings **URL:** \ **Category:** FoundationDB Core\ **Created:** [April 24, 2018, 7:05pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250 "2018-04-24T19:05:11Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![KrzysFR](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/krzysfr/32/43_2.png) [@KrzysFR](https://forums.foundationdb.org/u/KrzysFR)\ **Post date:** [April 24, 2018, 7:05pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/1 "2018-04-24T19:05:12Z")
I’m currently adding support to VersionStamps in the .NET Binding. I’m a bit confused by some discrepencies between the documentation, code implementation, and behaviour I’m seeing. Also, I was not able to find a lot of documentation on this subject, and no concrete example. ### 10 bytes or 12 bytes? **UPDATED** from answers below. There are two _flavors_ of Versionstamps: 80-bits and 96-bits long. The former is what the database understand, and the later is a client-side convention. - The **80-bits versionstamps** are 10 bytes longs, and are composed of 8 bytes (Transaction Version) followed by 2 bytes (Transaction Batch Order). They are ordered and guaranteed to be unique per transaction. They are handled by the `FDB_MUTATION_TYPE_SET_VERSIONSTAMPED_KEY` and `FDB_MUTATION_TYPE_SET_VERSIONSTAMPED_VALUE`. - The **96-bits versionstamps** are actually an 80-bit versionstamp followed by 2 extra bytes called _User Version_, so 12 bytes in total. These two bytes can be used if a transaction wants to insert more than one key. These two bytes are not _seen_ by the database, and are just a convention at the binding level. Both Python and Java bindings seem to have take the route of only exposing 96-bits versionstamps at the API level, and use a default User Version of 0. So these versionstamps will always be 12 bytes. The shorter 10-bytes ones are not exposed. When the application wants to create a key using a versionstamp, it does not know the actual value yet, so the pattern is to have a _placehold_ stamp, which then gets overwritten when setting a versionstamped key or value at commit time. These stamps are called _incomplete_ stamps in some bindings. The binding tracks the offset in the binary key or value where this placeholder stamp is located, and pass this info to the database, who then replace the bytes with the actual stamp. After the transaction has committed, the application can guery the _actual_ stamp used via `fdb_transaction_get_versionstamp`. So for example, if your layer uses keys like `('foo', )`, using the Tuple Layer, the serialized binary key would look something like this: ```auto ('foo', ) => < 02 'foo' 00 33 xx xx xx xx xx xx xx xx xx xx 00 00 > ``` _note: currently, bindings will use xx = FF as placeholders but this could be anything_ The prefix `02 'foo' 00` corresponds to the encoding of the `'foo'` string, the byte `33` is the type header for 96-bits version stamps in the Tuple Encoding and is not part of the stamp. The 10 ‘xx’ are placeholders for where the actual stamp will be placed, and the last `00 00` are the user version (0 by default). When calling the `SetVersionstampKey` method, you need to pass an additional value which is the offset in the key _where_ the stamp is located. This is done by adding 2 extra bytes at the end, containing the offset in little-endian. These 2 bytes are not actually _part_ of the key, and will be removed by the `SetVersionstampedKey` method. Since the location of the stamp in our example above is at offset 6, the actual byte array passed to the `SetVersionstampedKey` method will be: ```auto tr.SetVersionstampedKey( < 02 'foo' 00 33 xx xx xx xx xx xx xx xx xx xx 00 00 06 00>, 'hello world') ``` At commit time, the last two bytes are removed, and the 10 bytes are the specified offset are filled by the database. If the transaction commits at version `0x0123456789ABCDEF` with batch order `0x1234`, the key will become: ```auto < 02 'foo' 00 33 01 23 45 67 89 AB CD EF 12 34 00 00> = 'hello world' ``` In practice, the batch order will usually be 0 or a low number (depends on the number of concurrent transactions). The first 10 bytes are controlled by the database, and the last two bytes (‘00 00’) are controlled by the user, Original question: > [@](#): > > The first is that a VersionStamp seems to be 10 or 12 bytes depending on where I look. > > - The fdb.options states that `A versionstamp is a 10 byte, unique, monotonically (but not sequentially) increasing value for each committed transaction. The first 8 bytes are the committed version of the database. The last 2 bytes are monotonic in the serialization order for transactions.` > - The Java binding does use 12 bytes total, with the first 10 being the transaction number, and the last 2 the user version, and mention the Transaction Number as explicitely being 10 bytes. > - The Python binding a similar thing (but is devoid of comments) > - The database seems to overwrite 10 bytes in my keys. > > My initial tests with a 8 bytes transaction and 2 bytes user version showed the last 2 bytes systematically overwritten with 00 00 in a way that was non-obvious: I’m passing a temp key with 8 x 0xFF followed by 2 user version bytes `FF FF FF FF FF FF FF FF 12 34` and when I read back the key, I see something like `00 00 XX XX XX XX XX XX 00 00`. > > My initial thought that was my user version 0x1234 was overwritten with zeroes. BUT looking at the doc, and if the Transaction number is indeed 10 bytes, AND by some random chance the last two bytes of a Transaction Version are always 0, it would mean that VersionStamps are indeed 10 bytes + 2 bytes (and not 8 + 2) and that the “user version” part is strictly a client-side convention, and that the VERSIONSTAMP\_KEY mutation only care about the first 10 bytes (the next 2 bytes are ignored). > > So which is it? 12 bytes (10 + 2) or 10 bytes (8 + 2) ? Are the last 2 bytes of the transaction always 0? > > It looks like both JAVA and Python bindings have opted to only expose the 96 bits Versionstamps, while the core of fdb only knows about the 80 bits versionstamps… ### Custom Serialization Required The way Java and Python deal with Versionstamps and tuples is a little bit … weird. I’m not a big fan of having to need a custom method to build tuples that contain a versionstamp (due to the need to get the byte offset where it starts). This does not seem to play well with other serialization mechanics (like for ex combined with subspace prefixes, or other custom encodings). I was wondering if another approach would be better: using a specifc byte pattern, that is used to mark the location of a stamp (client side), and when such a byte array is passed to the VersionStampKey mutation, it would look for this pattern, and obtain the position that way. =\> no need for special code paths, any binary encoding scheme can simply output this pattern anywhere it wants, and it will be recognize at the last step. Obvious problem is what if this pattern is used by random chance by the key itself? It cannot be something trivial like all zeros, or all FF. Maybe `$VERSTAMP!$` or something like that? There is precendents with - for example - the multipart content encoding ([RFC1341(MIME) : 7 The Multipart content type](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html)) which explicitely define what is the expected chunk separator. Most implementations may choose a constant (or random) separator, and check if it is not contained in the message itself. If it is, choose another marker. We could maybe choose to have a default token to mark the spot where a VersionStamp is, but have a mechanism (somewhere on the transaction? or as an extra paramter to the VersionStampKey helper method) to specify what was the exact token used. ```csharp // use default token var key = AcmeLib.SerializeKey(("foo", 123, VersionStamp.Incomplete(42), 456)); // uses default token // -> <'foo',123,$VERSTAMP!$42,456> tr.VersionStampKey(key, ....); // risk of collision var key = AcmeLib.SerializeKey(("foo", 123, "Oh no, I have a $VERSTAMP!$ inlined", VersionStamp.Incomplete(42), 456), token: "ABCDEFGHIJ"); // -> <'foo',123,'Oh no, I have a $VERSTAMP!$ inlined',ABCDEFGHIJ42,456> tr.VersionStampKey(key, ..., token: "ABCDEFGHIJ"); ``` We could even decide to generate a random token per new transaction, and ensure that it does not happen twice in the same key. If it does, then the transaction would fail, retry (with a NEW random token), and the probability that the next token would be _also_ contained in another key of the same transaction would be very low. ```csharp db.Run((tr) => { var token = tr.GetStampToken(); // -> "Aoew!4='£K" var key = AcmeLib.Serialize(("foo", 123, VersionStamp.Incomplete(123), 456), token); // -> < 'foo',123,Aoew!4='£K42,456 > tr.VersionStampKey(key, ....); }); ``` ### fdb\_transaction\_get\_versionstamp **UPDATED** This method can be used to obtain the actual value that the database will insert into the key (or value) instead of the temporary placeholder. This value will be an 80-bit value that is the same for the whole transaction. If the transaction needed multiple ids, the way is to use an 96-bit timespan, with the last 16 bits being a user-provided integer. This method must be called BEFORE the call to `fdb_transaction_commit` and the Future will be resolved AFTER the transaction commits successfully (or fail). An exemple of Java code: > [@alloc](#): > > ```java > CompletableFuture vsFuture = tr.getVersionstamp(); > tr.commit().join(); // blocking call to wait on commit > byte[] versionstamp = vsFuture.join(); // non-blocking call to get versionstamp > > ``` This may have some impact on code that use async operations (.NET, Typescript with async/await, Java with CompletableFutures, etc…) especially when combined with retry loops (where the code does not manage the transaction itself, and in particular is not the one who invoke the commit method). Example of patterns that will fails: > [@spullara](#): > > Deadlocks, for good reason: > > ```java > byte[] versionstamp = open.runAsync(tx -> { > byte[] key = ds.get("test").packWithVersionstamp(Tuple.from(Versionstamp.incomplete())); > tx.mutate(SET_VERSIONSTAMPED_KEY, key, "test".getBytes()); > return tx.getVersionstamp(); > }).get(); > > ``` > > Cannot access closed object: > > ```java > byte[] versionstamp = open.runAsync(tx -> { > byte[] key = ds.get("test").packWithVersionstamp(Tuple.from(Versionstamp.incomplete())); > tx.mutate(SET_VERSIONSTAMPED_KEY, key, "test".getBytes()); > return CompletableFuture.completedFuture(tx); > }).get().getVersionstamp().get(); > > ``` > > OR > > ```java > byte[] versionstamp = open.runAsync(tx -> { > byte[] key = ds.get("test").packWithVersionstamp(Tuple.from(Versionstamp.incomplete())); > tx.mutate(SET_VERSIONSTAMPED_KEY, key, "test".getBytes()); > return CompletableFuture.completedFuture(tx); > }).thenApply(Transaction::getVersionstamp).get().get(); > > ``` Some possible solutions for Java: > [@spullara](#): > > Works: > > ```java > byte[] versionstamp = open.run(tx -> { > byte[] key = ds.get("test").packWithVersionstamp(Tuple.from(Versionstamp.incomplete())); > tx.mutate(SET_VERSIONSTAMPED_KEY, key, "test".getBytes()); > return tx.getVersionstamp(); > }).get(); > > ``` > > Awkward but works: > > ```java > byte[] versionstamp = open.runAsync(tx -> { > byte[] key = ds.get("test").packWithVersionstamp(Tuple.from(Versionstamp.incomplete())); > tx.mutate(SET_VERSIONSTAMPED_KEY, key, "test".getBytes()); > return CompletableFuture.completedFuture(tx.getVersionstamp()); > }).get().get(); > > ``` Original question: > [@](#): > > This is one maybe an error on my end, but when I call `fdb_transaction_get_versionstamp` I get the error 2017 with message `Operation issued while a commit was outstanding`, even though the commit was successfull: the data is stored in the database, and I can even get the commit version before calling get\_versionstamp. > > It returns an already completed Future, with error 2017 (UsedDuringCommit). > > Any idea? ---
**Author:** ![dave](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/dave/32/89_2.png) [@dave](https://forums.foundationdb.org/u/dave)\ **Post date:** [April 24, 2018, 7:39pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/2 "2018-04-24T19:39:03Z")
As far as the FDB core is concerned, versionstamps are 10 bytes: 8 bytes of version + 2 bytes transaction number within version. As far as the tuple format is concerned, versionstamps are 12 bytes: 8 bytes of version + 2 bytes transaction number within version + 2 bytes operation counter within transaction The latter format is obviously designed so that you can fill it in successfully with the help of the built in FDB operations. But FDB itself doesn’t know anything about the format of the last 2 bytes. Does that help? ---
**Author:** ![KrzysFR](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/krzysfr/32/43_2.png) [@KrzysFR](https://forums.foundationdb.org/u/KrzysFR)\ **Post date:** [April 24, 2018, 8:02pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/3 "2018-04-24T20:02:08Z")
Yes. So what is the “operation counter within a transaction”? in all my tests, the 2 bytes at ofset 8 and 9 are always 0. What are the conditions required to see a non-zero value there? At first glance it seems redundant with the user version (extra 2 bytes). ---
**Author:** ![dave](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/dave/32/89_2.png) [@dave](https://forums.foundationdb.org/u/dave)\ **Post date:** [April 24, 2018, 11:42pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/4 "2018-04-24T23:42:41Z")
It _is_ the same as the user version. VVVVVVVVTTUU 8 bytes version (filled by FDB) 2 bytes transaction# within version (filled by FDB) 2 bytes “user version” (_you_ should usually set this to 0 for the first versionstamped item you insert during a transaction, 1 for the second, etc) The reason that the last 2 bytes can’t be filled in automatically is because you very well may want to insert logically-the-same versionstamped item in more than one _index_, in which case it’s critical that it get the exact same (12-byte) versionstamp in each place. But the API can’t distinguish that from inserting two different versionstamped items, in which case you probably want to give them different versionstamps to preserve their order. ---
**Author:** ![alloc](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/alloc/32/9_2.png) [@alloc](https://forums.foundationdb.org/u/alloc)\ **Post date:** [April 25, 2018, 4:57am UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/5 "2018-04-25T04:57:06Z")
> [@KrzysFR](#): > > So which is it? 12 bytes (10 + 2) or 10 bytes (8 + 2) ? Are the last 2 bytes of the transaction always 0? Hm, yeah, this might have been an instance where brevity was chosen instead of clarity. Maybe a 12 byte “versionstamp” should have been an ExtendedVersionstamp or something to indicate it is different from a 10 byte versionstamp? ¯\_(ツ)\_/¯ > [@KrzysFR](#): > > Obvious problem is what if this pattern is used by random chance by the key itself? It cannot be something trivial like all zeros, or all FF. Maybe $VERSTAMP!$ or something like that? This can be improved somewhat by the fact that it will be the magic string preceded by the versionstamp type code, which might make it less likely to collide (or, well, maybe not). We ultimately decided that a magic string was more error prone than the extra methods were bad ergonomically, so we went with what’s in the codebase now. To handle subspaces, we added methods to pack a tuple with a prefix so that the prefix length gets added in to the offset correctly. It doesn’t quite work for suffixes, but that seems to be less common. > [@KrzysFR](#): > > So what is the “operation counter within a transaction”? What are your order of operations? The get\_versionstamp method is somewhat weird, but you have to call the future before it is committed, and then that future will be ready only after the commit. So something like: ```java CompletableFuture vsFuture = tr.getVersionstamp(); tr.commit().join(); // blocking call to wait on commit byte[] versionstamp = vsFuture.join(); // non-blocking call to get versionstamp ``` (But, like, in C# instead of Java.) I think you can get used\_during\_commit if you call `getVersionstamp` after commit rather than before. > [@KrzysFR](#): > > in all my tests, the 2 bytes at ofset 8 and 9 are always 0. What are the conditions required to see a non-zero value there? Are you only ever running one transaction at a time? You will only see a non-zero “batch version” (we call it) if there are multiple transactions being committed together at a single version, which can only happen if there are concurrent commits. The easiest way is to probably fire off multiple transactions and then wait for all of them. > [@dave](#): > > The reason that the last 2 bytes can’t be filled in automatically is because you very well may want to insert logically-the-same versionstamped item in more than one index, in which case it’s critical that it get the exact same (12-byte) versionstamp in each place. Just to expand on this a little, with versionstamps in particular, it’s often the case that you will want a forward index and a reverse index, i.e., `(keyspace_1, key) -> version` and `(keyspace_2, version) -> key`. There are a couple of reasons for this, but the most obvious is that as you don’t know what key you wrote if you only write the one with a version, then to remove that key, you have to somehow figure out what that version was. So you can either scan the keyspace\_2 (slow) or look it up in the other index (fast). The other less obvious reason is that if you get a `commit_uknown_result` error, then if all you had were the index with versions in `keyspace_2`, it would be really easy to add multiple entries in `keyspace_2` in subsequent retries by mistake. Having `keyspace_1` around let’s you either detect that you are in a retry loop and not write it again (essentially letting you know that your commit succeeded) or, if index maintenance is done correctly, lets you clean up `keyspace_1` and `keyspace_2` within the retry loop. But all of that depends on the versionstamp having the same user version each time for a logical record within the database. ---
**Author:** ![KrzysFR](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/krzysfr/32/43_2.png) [@KrzysFR](https://forums.foundationdb.org/u/KrzysFR)\ **Post date:** [April 25, 2018, 8:11am UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/6 "2018-04-25T08:11:51Z")
I have updated the original question about stamp size from these answers. ---
**Author:** ![KrzysFR](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/krzysfr/32/43_2.png) [@KrzysFR](https://forums.foundationdb.org/u/KrzysFR)\ **Post date:** [April 25, 2018, 8:50am UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/7 "2018-04-25T08:50:37Z")
> [@alloc](#): > > What are your order of operations? The get\_versionstamp method is somewhat weird, but you have to call the future before it is committed, and then that future will be ready only after the commit. So something like: > > CompletableFuture\ vsFuture = tr.getVersionstamp(); > tr.commit().join(); // blocking call to wait on commit > byte versionstamp = vsFuture.join(); // non-blocking call to get versionstamp > > (But, like, in C# instead of Java.) I think you can get used\_during\_commit if you call getVersionstamp after commit rather than before. Ok so this is a bit confusing. I’m calling the getVersionstamp method AFTER the commit has completed, but I’m getting an error `Operation issued while a commit was outstanding` which seems wrong to me: The commit was already completed, so it was not “_while_” and there was no “_outstanding_” commit? Maybe the wording of the error message is wrong? > […] but you have to call the future before it is committed, and then that future will be ready only after the commit I’m having some issues with this API, because it will seem a little bit weird - from a .NET coder’s point of view - when dealing with tasks, AND it does not appear to compose well with retry loops and multiple layers. My test creates and commits the transaction manually, but typical application will never do that and go through one of the retry loops (the `db.run(...)` in Java): ```csharp Task SomeControllerMethod(....) { //... check args ... await db.WriteAsync((tr) => { // traditional write operations tr.ClearRange(....); tr.Set(..., ...); // new Versionstamp API tr.SetVersionstampedKey(MAKE_KEY_WITH_VERSION_STAMP(), ....); }, HttpContext.Cancel); //... return View(....); } ``` _obviously, the db code would be inside some Business Logic class, and not inlined in the controller!_ If the code wanted to obtain the actual Versionstamp, as well as some other result extracted from the database, both at the same time, it will look very ugly: ```csharp //... Task stampTask; // out of scope Slice result = await db.ReadWriteAsync((tr) => { var val = await tr.Get('SOME_KEY', ....); tr.SetVersionstampedKey(MAKE_KEY_WITH_VERSION_STAMP(), ..); stampTask = tr.GetVersionstamp(); return val; }, cancel); Versionstamp stamp = await stampTask; // need an extra await here! //... var data = DoSomethingWithIt(result, stamp); return View(new SpomeViewModel { Data = data, ... }); ``` Having to hoist a task outside the scope and do an additional await looks _weird_ in moden .NET code. The retry loop could return the Versionstamp task alongside the result like this, but again it does not look nice: ```csharp (Slice result, Task stampTask) = await db.ReadWriteAsync((tr) => { Slice val = await tr.Get('SOME_KEY', ....); tr.SetVersionstampedKey(MAKE_KEY_WITH_VERSION_STAMP(), ..); return (val, tr.GetVersionstamp()); }, cancel); Versionstamp stamp = await stampTask; // still need an extra await here ``` I _think_ what the user would expect to happen is that the retry loops returns the Versionstamp directly not a Future: ```csharp (Slice result, Versionstamp stamp) = await db.ReadWriteAsync((tr) => { Slice val = await tr.Get('SOME_KEY', ....); tr.SetVersionstampedKey(MAKE_KEY_WITH_VERSION_STAMP(), ..); return (val, tr.GetVersionstamp()); }, cancel); ``` The inner lambda would have signature `Func>>`, which is a mouthfull, The retryloop method wants to return a `Task`, and not a `Task>` so some generic magic needs to be done. It may have some effects on the overall API because, in .NET, you cannot have overloads whose signature only differ by the return value. So it would not be easy to have an overload of `ReadWriteAsync(...)` that returns plain results, and another one which also return a tuple with the resolved timestamp. You’d probably have to change the method name, or add some arguments to disambiguate. I _could_ have a `ReadWriteWithVersionStampAsync() => Task<(TResult, VersionStamp>)` overload, but then it may cause issue when composing multiple libraries: > Let’s say in an HTTP request controller, the outer scope starts a retry loop, and then pass along the transaction to some business logic, which then calls into other Layers (Document, Blob, Index, …). If one layer is refactored somehow, and starts using versionstamps. It would have an impact on the outerscope (in the HTTP controller code) because it needs to _know_ to call GetVersionstamp, _before_ the commit, and extract the value _after_ it has suceeded, and _outside_ the scope of the retry loop. How do I pass back the actual versionstamp into the original layer (which has long since returned and be garbage collected). It looks like you need dedicated transactions that are fully handled by the Layer code, and will not be able to compose with other layer inside a single transaction ? ---
**Author:** ![ajbeamon](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ajbeamon/32/13_2.png) [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)\ **Post date:** [April 25, 2018, 1:52pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/8 "2018-04-25T13:52:30Z")
> [@KrzysFR](#): > > Let’s say in an HTTP request controller, the outer scope starts a retry loop, and then pass along the transaction to some business logic, which then calls into other Layers (Document, Blob, Index, …). If one layer is refactored somehow, and starts using versionstamps. It would have an impact on the outerscope (in the HTTP controller code) because it needs to know to call GetVersionstamp, before the commit, and extract the value after it has suceeded, and outside the scope of the retry loop. I may be misunderstanding what you’re saying here, but I think this is a good illustration for why get\_versionstamp works the way it does. If one of the layers needs the versionstamp but isn’t responsible for committing, it can call get\_versionstamp anyway and the returned future will be set when the transaction ultimately commits at a higher level. > [@KrzysFR](#): > > How do I pass back the actual versionstamp into the original layer (which has long since returned and be garbage collected). If the layer is going to use the versionstamp of a transaction, it has to be around in some form after the commit completes. If it’s not, what’s the use-case for needing it? ---
**Author:** ![KrzysFR](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/krzysfr/32/43_2.png) [@KrzysFR](https://forums.foundationdb.org/u/KrzysFR)\ **Post date:** [April 25, 2018, 9:07pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/9 "2018-04-25T21:07:24Z")
> [@ajbeamon](#): > > If the layer is going to use the versionstamp of a transaction, it has to be around in some form after the commit completes. If it’s not, what’s the use-case for needing it? You’re right, but I guess the issue comes from how you would write this using using idiomatic .NET with async/await. Let’s say I have a very simple WEB API controller, that create versionstamped keys but doesn’t need to know their value. This is nice and simple with the current API. _note: I’m assuming that encoding of version stamps at the tuple layer is magical and just works. outside the scope of this sample_ ```auto public class SomeApiController { #region Stuff.. // all initialized from the Web Application via HttpContext and DI... private IFdbDatabase Db; private CancellationToken Cancellation; private IDynamicKeySubspace Location; private string DoSomethingWithIt(Slice data) => "hello"; #endregion // REST EndPoint public async Task SomeRestMethod(Guid id) { // here is the "business logic" var data = await this.Db.ReadWriteAsync(async (tr) => { // read a key var val = await tr.GetAsync(this.Location.Keys.Encode("A", id)); // create stamped keys tr.SetVersionStampedKey(this.Location.Keys.Encode("B", tr.CreateStamp(1)), Slice.FromString("some_value")); tr.SetVersionStampedKey(this.Location.Keys.Encode("B", tr.CreateStamp(2)), Slice.FromString("some_other_value")); return val; }, this.Cancellation); // return return new SomeResult { Foo = DoSomethingWithIt(data) }; } } ``` I would NOT want to write something like this: ```auto await this.Db.ReadWriteAsync(async (tr) => { // read a key var val = await tr.GetAsync(this.Location.Keys.Encode("A", id)); // create stamped keys tr.SetVersionStampedKey(this.Location.Keys.Encode("B", tr.CreateStamp(1)), Slice.FromString("some_value")); tr.SetVersionStampedKey(this.Location.Keys.Encode("B", tr.CreateStamp(2)), Slice.FromString("some_other_value")); tr.GetVersionStampAsync() .ContinueWith((t) => { // this runs somewhere on the ThreadPool, at any time in the future! var stamp = t.Resut; DoSomethingWithId(data, stamp); }); // => if it fails, nobody will know about it! ); }, this.Cancellation); ``` The Task continuation runs on another thread, maybe later, long after the HTTP context has been collected. And also it as no way to send it back to the client and could throw exceptions into nowhere Now if I want to pass the actual versionstamp outside the scope of the retry loop and back to the controller while the HTTP context is still alive, I could try to change it like this, which is at least still using async/await: ```auto // REST EndPoint public async Task SomeRestMethod(Guid id) { // first part of the business logic (that talks to the db) (Slice data, Task stampTask) = await this.Db.ReadWriteAsync(async (tr) => { // read a key var val = await tr.GetAsync(this.Location.Keys.Encode("A", id)); // create stamped keys tr.SetVersionStampedKey(this.Location.Keys.Encode("B", tr.CreateStamp(1)), Slice.FromString("some_value")); tr.SetVersionStampedKey(this.Location.Keys.Encode("B", tr.CreateStamp(2)), Slice.FromString("some_other_value")); return (val, tr.GetVersionStampAsync()); }, this.Cancellation); // need another await VersionStamp stamp = await stampTask; // <-- this is ugly! var foo = DoSomethingWithIt(data, stamp); // hidden in here is the second part of the business logic // return return new SomeResult { Foo = foo }; } ``` But the actual business logic is split in two: the retry loop is doing the first part of the job, but then the actual “finish” is in this _DoSomethingWithIt_, which has to _know_ that I used two stamps with user version 1 & 2. Also, the outer controller code has to act as the middle man, and also do the task resolving to get the stamp. =\> this is very tied to the implementation (using FoundationDB) and may not be easy to abstract away. After playing a bit with it, I think the ideal would be to add an _`onSuccess`_ handler on the retry loop logic, that gets called once the transction commits, and is passed the result of the inner handler, plus the resolved versionstamp. It can then consume and post-process the stamp. ```auto // REST EndPoint public async Task SomeRestMethod(Guid id) { var foo = await this.Db.ReadWriteAsync( handler: async (tr) => { // this part runs inside the transaction, and can be retried multiple times // read a key var val = await tr.GetAsync(this.Location.Keys.Encode("A", id)); // create stamped keys tr.SetVersionStampedKey(this.Location.Keys.Encode("B", tr.CreateStamp(1)), Slice.FromString("some_value")); tr.SetVersionStampedKey(this.Location.Keys.Encode("B", tr.CreateStamp(2)), Slice.FromString("some_other_value")); return val; }, success: (val, stamp) => { // this parts runs at most once, after the transaction has committed succesfully. return DoSomethingWithIt(val, stamp); }, ct: this.Cancellation ); // no additional fdb-specific logic here! return new SomeResult { Foo = foo }; } ``` Everything stamp relative is handled inside `ReadWriteAsync()`, and the result is the complete post-processed thing. If I refactor this further, then no more fdb-logic is visible inside the controller itself ```auto public class SomeApiController { #region Stuff.. private IFdbDatabase Db; private CancellationToken Cancellation; #endregion // REST EndPoint public async Task SomeRestMethod(Guid id) { var engine = new MyBusinessLogicEngine(/*...*/); var result = await engine.DealWithIt(this.Db, id, this.Cancellation); // no additional fdb-specific logic here! return new SomeResult { Foo = result }; } } // library in a different assembly somewhere public class MyBusinessLogicEngine { private IDynamicKeySubspace Location; public Task DealWithIt(IFdbDatabase db, Guid id, CancellationToken ct) { return db.ReadWriteAsync( handler: async (tr) => { // this part runs inside the transaction, and can be retried multiple times // read a key var val = await tr.GetAsync(this.Location.Keys.Encode("A", id)); // create stamped keys tr.SetVersionStampedKey(this.Location.Keys.Encode("B", tr.CreateStamp(1)), Slice.FromString("some_value")); tr.SetVersionStampedKey(this.Location.Keys.Encode("B", tr.CreateStamp(2)), Slice.FromString("some_other_value")); return val; }, success: (val, stamp) => { // this parts runs at most once, after the transaction has committed succesfully. // second part of the business logic return "hello:" + val + ":" + stamp; }, ct: ct ); } } ``` But now, the method in the Business Logic class takes in a Database instance, and not a Transaction, so it cannot compose well with _another_ layer that could read/write some keys at the same time. I’ve seen this issue happening a lot in the last few years: the web controller has the db and is supposed to orchestrate everything in a single transaction to reduce latency. But all the various other libraries underneath will try open their own transactions in parallel. Sometimes by laziness, but sometimes by necessity (like a Blob Layer that has to upload more than 10 MB, or here deeply nested code that needs execute _after_ the transaction completes, but _before_ the controller gets back the result). So by “not composing” well, I mean that now code inside retry loops has to take the responsibility of handling the lifetime of the transaction, and/or interleave part of its code back up the callstack, while keeping all its own scope (arguments, variables, context objects) it allocated alive long enough. ---
**Author:** ![spullara](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/spullara/32/125_2.png) [@spullara](https://forums.foundationdb.org/u/spullara)\ **Post date:** [April 25, 2018, 10:08pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/10 "2018-04-25T22:08:06Z")
I ran into a similar problem today. I am able to use Database.run() to get the versionstamp as it returns the CompletableFuture\ and then I can wait for it and get the value. However, if I use Database.runAsync() it gets pretty ugly but I think I found the pattern for it. Not really reporting a bug, just agreeing that it is can be a little painful to get the versionstamp out of there if you need it. Works: ```java byte[] versionstamp = open.run(tx -> { byte[] key = ds.get("test").packWithVersionstamp(Tuple.from(Versionstamp.incomplete())); tx.mutate(SET_VERSIONSTAMPED_KEY, key, "test".getBytes()); return tx.getVersionstamp(); }).get(); ``` Deadlocks, for good reason: ```java byte[] versionstamp = open.runAsync(tx -> { byte[] key = ds.get("test").packWithVersionstamp(Tuple.from(Versionstamp.incomplete())); tx.mutate(SET_VERSIONSTAMPED_KEY, key, "test".getBytes()); return tx.getVersionstamp(); }).get(); ``` Cannot access closed object: ```java byte[] versionstamp = open.runAsync(tx -> { byte[] key = ds.get("test").packWithVersionstamp(Tuple.from(Versionstamp.incomplete())); tx.mutate(SET_VERSIONSTAMPED_KEY, key, "test".getBytes()); return CompletableFuture.completedFuture(tx); }).get().getVersionstamp().get(); OR byte[] versionstamp = open.runAsync(tx -> { byte[] key = ds.get("test").packWithVersionstamp(Tuple.from(Versionstamp.incomplete())); tx.mutate(SET_VERSIONSTAMPED_KEY, key, "test".getBytes()); return CompletableFuture.completedFuture(tx); }).thenApply(Transaction::getVersionstamp).get().get(); ``` Awkward but works: ```java byte[] versionstamp = open.runAsync(tx -> { byte[] key = ds.get("test").packWithVersionstamp(Tuple.from(Versionstamp.incomplete())); tx.mutate(SET_VERSIONSTAMPED_KEY, key, "test".getBytes()); return CompletableFuture.completedFuture(tx.getVersionstamp()); }).get().get(); ``` ---
**Author:** ![ajbeamon](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ajbeamon/32/13_2.png) [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)\ **Post date:** [April 26, 2018, 6:05pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/11 "2018-04-26T18:05:47Z")
> [@KrzysFR](#): > > (Slice data, Task\ stampTask) = await this.Db.ReadWriteAsync(async (tr) =\> > { > // read a key > var val = await tr.GetAsync(this.Location.Keys.Encode(“A”, id)); > > ``` > // create stamped keys > tr.SetVersionStampedKey(this.Location.Keys.Encode("B", tr.CreateStamp(1)), Slice.FromString("some_value")); > tr.SetVersionStampedKey(this.Location.Keys.Encode("B", tr.CreateStamp(2)), Slice.FromString("some_other_value")); > > return (val, tr.GetVersionStampAsync()); > }, this.Cancellation); > > // need another await > VersionStamp stamp = await stampTask; // <-- this is ugly! > > ``` I think this is one of the expected ways that you would use a versionstamp. The versionstamp result cannot be known while the transaction is uncommitted, so the return value of a call to get the versionstamp is a future to be set after the commit succeeds or fails. I believe it’s the case that this future is set as part of the commit, so in that particular example (and all of Sam’s examples, assuming `open` is a database), the future will already be ready at the point where you are waiting on it. It seems possible to provide an API in the bindings that could accomplish the same thing in this case without requiring a second wait if you wanted (e.g. by having another version of the run loop that returns a versionstamp or something, though it could only work on a database). Also, I don’t know if there’s a reason that get\_versionstamp must be called before commit, but if not then perhaps it could be changed so that you could get the versionstamp future after commit as well. It doesn’t really help in these retry loop examples you and Sam gave, though, because you don’t have access to the transaction outside of the loop. > [@KrzysFR](#): > > tr.GetVersionStampAsync() > .ContinueWith((t) =\> > { // this runs somewhere on the ThreadPool, at any time in the future! > var stamp = t.Resut; > DoSomethingWithId(data, stamp); > }); // =\> if it fails, nobody will know about it! It sounds like you are trying to design for a case where you have a layer API that has some sort of request that takes a transaction (i.e. doesn’t do the commit itself) but wants to do work both in and after the transaction and not return to the root caller until it’s done with all of that. I agree that our FoundationDB client and our other bindings don’t provide that capability, so at least based on the current API you would have to design your layer’s API accordingly. For example, you may have to make multiple calls into the API or return a value from your request function that will signify once the request is fully complete. Commit hooks could be useful in this and other scenarios, so it may be worth posing that as its own specific feature request. ---
**Author:** ![KrzysFR](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/krzysfr/32/43_2.png) [@KrzysFR](https://forums.foundationdb.org/u/KrzysFR)\ **Post date:** [April 26, 2018, 6:52pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/12 "2018-04-26T18:52:34Z")
I think one the confusing aspect is that the method returns a Future like most other operations on a transaction, but it cannot be resolved until after the commit Future has resolved, unlike any other operations. The only other similar thing I can think of are Watches, which outlive the transaction scope. Also, there is already one property, the commited version, which only exists after the commit, but this one is not exposed as a Future: you call `fdb_transaction_get_committed_version` and get the result immediately (or an error if you called it too soon). Couldn’t we have a similar system for the stamps? Maybe if requesting the versionstamp has some overhead when committing, it could require some option to be set, like `AUTO_RESOLVE_VERSIONSTAMPS`? Once the commit succeeds, then querying the versionstamp accessor would return the value (or an error) just like `fdb_transaction_get_committed_version` does today. This would at least give a more straight-forward view of the workflow: you do your thing with the transaction (get, set, clear, …) and maybe versionstamp some keys. You need the actual value used, it will be something that you call _after_ a successfull commit. Not half-before, half-after. Like you said, if the Future is resolved at the same time as the commit’s Future, then it could be a simple value accessor, no ? Then, for retry loops, the remaining issue to resolve, is how can we create a small window of execution that happens after the commit, but before the retry loop yields to the caller (and destroy the context). This could be done with a `onSuccess` lambda executed after commit, or maybe you could register callbacks on the transaction instance itself? I’m not fond of the later, because it can lead to bad-practice pattern (in .NET, probably similar in other languages) were the callback will create a new scope that will capture all the variables and state of the outer scope, and could keep a lot of objects and state alive for no reason that the GC cannot collect. ```auto await db.ReadWriteAsync((tr) => { // outer scope that can allocate large keys and values (byte[]) byte[] evil_buffer = new byte[100_000_000_000]; // will be captured by inner scope! // serialize keys, call tr.SetVersionStampedKey(...), etc... tr.OnCommitted((state) => { // inner scope var commitVersion = state.CommitVerison; var stamp = state.VersionStamp; // Do something with it! } }, ...); ``` The inner scope will capture all the variables in the outer scope, so the `evil_buffer` may be kept alive for more time than required (GC cannot guess if it is still alive). _This is a current limitation of the Roslyn .NET compiler that merge all scopes inside a method into a single container that aggregates all the state in the heap in the same instance. Not sure about Java or other_ The only way I know out of this, is to extract each inner scope into a different method, and call them. This leads to broken code with pieces of scopes everywhere. _I’ve been bitten by this so many times, maybe that’s why I’m so uneasy with this API pattern!_ 🙂 ---
**Author:** ![KrzysFR](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/krzysfr/32/43_2.png) [@KrzysFR](https://forums.foundationdb.org/u/KrzysFR)\ **Post date:** [April 26, 2018, 6:58pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/13 "2018-04-26T18:58:43Z")
I have a PR opened with my current work in progress here [https://github.com/Doxense/foundationdb-dotnet-client/pull/72](https://github.com/Doxense/foundationdb-dotnet-client/pull/72) I’m trying out an alternative way of serializing of versionstamps, without active support required from the encoders (tuple layer, etc…), using the random tokens idea I described above. It looks like it will simplify things a lot. ---
**Author:** ![ajbeamon](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ajbeamon/32/13_2.png) [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)\ **Post date:** [April 26, 2018, 7:44pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/14 "2018-04-26T19:44:50Z")
> [@KrzysFR](#): > > you call fdb\_transaction\_get\_committed\_version and get the result immediately (or an error if you called it too soon). Couldn’t we have a similar system for the stamps? In most (all?) of the examples posed in this thread, there was no transaction object to make this call on after the commit because of the use of the retry loops. ---
**Author:** ![KrzysFR](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/krzysfr/32/43_2.png) [@KrzysFR](https://forums.foundationdb.org/u/KrzysFR)\ **Post date:** [April 26, 2018, 8:29pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/15 "2018-04-26T20:29:35Z")
> [@ajbeamon](#): > > In most (all?) of the examples posed in this thread, there was no transaction object to make this call on after the commit because of the use of the retry loops. That’s true, I was thinking that the retry loop machinery would be able to do this automatically. My current thinking is that using one the the vesionstamped atomic operations would set a flag. Once the transaction need to be commited, and if this flag is set, the retry loop would also request the versionstamp as well, and hide the task internally. The result would then be exposed to the caller via some mechanism, either via a second ‘onSuccess’ lambda that is invoked with the result of the loop body plus the resolved stamp, or via some other way (shared ‘context’ object? a fat ‘result’ object with multiple properties?) If getting the resolved versionstamp from the transaction is cheap enough, I can do this automatically even if the caller doesn’t need it. If this has a non-trivial cost, then I would need to add some setting or option to trigger this. If the only cost is having to create a Future handle, then I could maybe try to optimize this case by delaying the allocation of all the interop machinery in the binding (only alloation the FutureHandle, not the tasks and callbacks). This could make it so that I can use the existing low level C API, and attempt to guide users of the .NET binding into patterns that are less prone to allocations and potential deadlocks. I would not want to artificially slow down the very fast path of code that just wants to add a message on some queue (using versionstamps) and that doesn’t care about the stamp itself. If the overhead is a couple % who cares, but if it is more, then I really need to make this opt-in. ---
**Author:** ![ajbeamon](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ajbeamon/32/13_2.png) [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)\ **Post date:** [April 26, 2018, 9:27pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/16 "2018-04-26T21:27:41Z")
Ah, I see. I thought your request was for a change to the C API, so my comment was in regard to that. Certainly the bindings can provide other features on top of that to make various patterns easier. I don’t think that there’s much extra cost to requesting the versionstamp needlessly, as it appears that the call just returns a future that’s being set regardless. ---
**Author:** ![josephg](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/josephg/32/84_2.png) [@josephg](https://forums.foundationdb.org/u/josephg)\ **Post date:** [April 27, 2018, 4:49am UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/17 "2018-04-27T04:49:52Z")
For what its worth, this is all quite fun with the new nodejs bindings too. I suspect that with the introduction of promises & async/await the new node bindings are going to be similar to the C# bindings in many ways. Deadlocks: ```javascript const stamp = await db.doTransaction(async tn => { tn.setVersionstampedValue('x', Buffer.from([1,2,1,2,1,2,1,2,1,2])) return tn.getVersionStamp() }) // DEADLOCKS ``` … When an async function returns a promise, it automatically unwraps the inner promise before returning. Thus that code waits for the `tn.getVersionStamp()` promise internally, which deadlocks. But you can trivially avoid that behaviour using awful hacks, like wrapping the stamp in an array: ```javascript const stampArr = await db.doTransaction(async tn => { tn.setVersionstampedValue('x', Buffer.from([1,2,1,2,1,2,1,2,1,2])) return [tn.getVersionStamp()] }) const stamp = await stampArr[0] // WORKS ``` I’m not sure what the best approach is here. JS might following what you end up doing in the C# bindings. ---
**Author:** ![dave](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/dave/32/89_2.png) [@dave](https://forums.foundationdb.org/u/dave)\ **Post date:** [April 27, 2018, 7:27pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/18 "2018-04-27T19:27:13Z")
I don’t think that switching to have a synchronous get\_versionstamp() after commit would actually make any of these issues easier. If you do want that, I think you can have it by just _blocking_ on the future after the transaction commits; I think that is in fact guaranteed to complete quickly enough to tolerate for even a single threaded event loop. I think the composable way to build operations with versionstamps is roughly like this (apologies if my hand-written C# doesn’t compile) ``` public Task< (LogicalResult, Task) > doThing( DatabaseOrTransaction dbtx, Guid id ) { return dbtx.ReadWriteAsync( async (tr) => { var read = tr.GetAsync( this.Location.Keys.Encode("A", id) ); tr.SetVersionStampedKey( this.Location.Keys.Encode("B", tr.CreateStamp(1)), id ); LogicalResult lr = computeLogicalResult( id, await read ); var fullId = computeFullId( id, tr.GetVersionStampAsync() ); return (lr, fullId); } ); } private async Task computeFullId( Guid id, Task stamp ) { return new FullId { part1 = id, part2 = await stamp }); } ``` Of course you can make computeFullId() an async lambda if you don’t want to name it. You should be able to combine multiple things like this in a transaction, including doing them in parallel if they don’t otherwise conflict. You can easily call it on a database as well. The logic for exactly what the versionstamp means (and even that a versionstamp is used at all) is safely located inside doThing(). The caller is responsible for not waiting on the Task until you are definitely outside a transaction context, but that is just a fundamental requirement of using versionstamps (or watches, which are very similar). And for the same reason I think it is kind of fundamentally needed for the return value of an operation like doThing() that you want to be composable to have two parts in some sense - one that you can safely use inside another transaction that is composing doThing(), and one that can only be accessed outside a transaction. ---
**Author:** ![KrzysFR](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/krzysfr/32/43_2.png) [@KrzysFR](https://forums.foundationdb.org/u/KrzysFR)\ **Post date:** [May 9, 2018, 4:27pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/19 "2018-05-09T16:27:15Z")
@alloc What is the impact of [#242 - Unify SET\_VERSIONSTAMPED\_KEY and SET\_VERSIONSTAMPED\_VALUE API](https://github.com/apple/foundationdb/pull/242) and [#148](https://github.com/apple/foundationdb/issues/148) for bindings? Looks like, depending on the API version, we will have to specify the offset as 2 or 4 bytes. ---
**Author:** ![alloc](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/alloc/32/9_2.png) [@alloc](https://forums.foundationdb.org/u/alloc)\ **Post date:** [May 9, 2018, 4:49pm UTC](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250/20 "2018-05-09T16:49:37Z")
That’s right. If you are a binding maintainer and have decided to implement versionstamps in tuples, the suggestion would be to have it choose whether to add two or four bytes based on the API version. Here’s the line where that’s done in the Java bindings: [https://github.com/apple/foundationdb/blob/f3093642b3c1babe93aacbaee8f60b4008662d52/bindings/java/src/main/com/apple/foundationdb/tuple/TupleUtil.java#L583](https://github.com/apple/foundationdb/blob/f3093642b3c1babe93aacbaee8f60b4008662d52/bindings/java/src/main/com/apple/foundationdb/tuple/TupleUtil.java#L583) The benefit of this change is that once that’s done, the same code can be used to encode data for `SET_VERSIONSTAMPED_KEY` mutations and `SET_VERSIONSTAMPED_VALUE` mutations rather than having one mutation take two bytes and the other four. (The alternative alternative would be to have `SET_VERSIONSTAMPED_VALUE` mutations have a weird limitation that meant that you could place a versionstamp only in the first 65 kB of the value.) I don’t think it should be too painful, but maybe I’m wrong. [Next page](https://forums.foundationdb.org/t/implementing-versionstamps-in-bindings/250.md?page=2)