FoundationDB

Use case for append_if_fits?


(Sam Pullara) #1

I’ve been looking at this new mutation:

Appends param to the end of the existing value already in the database at the given key (or creates the key and sets the value to param if the key is empty). This will only append the value if the final concatenated value size is less than or equal to the maximum value size (i.e., if it fits). WARNING: No error is surfaced back to the user if the final value is too large because the mutation will not be applied until after the transaction has been committed. Therefore, it is only safe to use this mutation type if one can guarantee that one will keep the total value size under the maximum size.

Unlike the other mutations, this caveat seems incredibly limiting since I will have to check the length, possibly in another key though, before calling append. What kind of layer features do you expect could be implemented with this mutation type? Maybe tracking the size using watch off to the side and hoping?


(David Scherer) #2

The motivating use case is/was high performance internal metrics collection. See MetricLogger.actor.cpp line 278. I’m not sure, but I think that in this case the data model means that there is only one possible writer for the key, so that the writer knows exactly how long the value it’s appending to is, and the use of an atomic operation is not the usual high concurrency but just reducing network communication overhead when appending a small value to a large one. Even if you had to actually read the value in the transaction, you would still save the network and transaction logging bandwidth to write the entire value in a conventional compare-and-set.

As you say, you could safely use it if some global invariant means that the length is a function of other things in the database you can check more cheaply (say, there are a bunch of keys that will always be the same length). Or you could have a data model such that if the key hits the maximum length it isn’t wrong (maybe you are updating a compact representation of data you have in a less compact form elsewhere, and your read code could go read the hard way if it detect that the value has overflowed). Or some invariant could just make an overflow impossible in the first place.


(Sam Pullara) #3

Ok, that is more or less what I expected. I suppose another nice use case would be cases where you know the period of the appends, their size and that the key is dependent on the timestamp such that you know it won’t overflow.