Should mutations performed via fdbcli always update the \xff/metadataVersion key?

Using FoundationDB
1

This may be controversial, but this is something that could prevent a lot of issues and headaches.

tl;dr: fdbcli should - by default - update the \xff/metadataVersion after committing any mutation to the cluster, in order to not break Layers using that key to cache metadata.

I’ve added support for caching metadata to all my layers, using the \xff/metadataVersion key to decide when they should throw any previous cached data. All my layers (and including the Directory Layer of the .NET binding) use this to cache data that does not change frequently and is expensive to query on every transaction (prefix of directory subspaces, schema and index definitions of a collection of documents, etc…) and everything is based on the \xff/metadataVersion being changed when an admin updates the schema.

My main issue is that, whenever I use fdbcli to quickly patch something - and most frequently it consists on performing a large clearrange on the db - it completely breaks that contract and borks all the layers that use caching, because the metadata version was not changed!

Welcome to the fdbcli. For help, type `help'.
fdb> writemode on
fdb> get \xff/metadataVersion
`\xff/metadataVersion' is `\x00\x00\x0f[F\x93p\x09\x00\x00'
fdb> clearrange \x00 \xFF
Committed (16884943757645)
fdb> get \xff/metadataVersion
`\xff/metadataVersion' is `\x00\x00\x0f[F\x93p\x09\x00\x00'
fdb>

In this example, I have completely wiped the database, including the directory layer!, but the metadata version has not changed. Following this command, any layer that had already cached any metadata will blissfully ignore this, and continue writing data into the keyspace, at least for a while, until some tool or command is used to change the value.

This very situation happened to me minutes before a demo, and it took some time for me to figure out what happens (and the fix was simply to bounce all the pods in the kubernetes cluster… not great).

I think that, by default, any mutation performed by fdbcli, should touch the \xff/metadataVersion key at commit time. This would ensure that any layer using caching would at reload its cache and see the change.

If there are situations were the admin using fdbcli does not want this to happen, maybe there could be a different argument to writemode, like writemode unsafe? I think that it should be very clear that randomly setting or clearing ranges in the keyspace, without touche the metadata version can be very dangerous!

Any opinions on this?

Anyone wondering why sometimes the caching feature of a layer did not work properly (maybe inducing data corruption), and that would now make the connection with a random “clearrange” issued via fdbcli when troubleshooting ?

2

fdbcli is an administrative tool - to me, using it to write actually data sounds insane in the first place (sorry for the wording). Do I miss something? Can you provide us with a use-case?

The only use-case for writing with fdbcli for us has been some very simple testing and demos. Otherwise we would write code to safely change data. fdbcli also doesn’t support any layers and it auto commit by default. Writing a Python scripts if you want to change data is probably a much better way to go anyways…

3

The use case is more during development/initial testing or as a last resort to reinstall everything from scratch to respond to unplanned events. We are still in development phase, so we regularly update the schema / change the way data is stored in the database by a layer under active development, and do a lot of clearrange \x00 \xFF in the test cluster between updates or when jumping from one branch to another.

I agree that doing that in production would be unsafe, but it would be especially even more unsafe if you have layers that use metadata caching.

That’s true, but unfortunately there is a lack of tooling at the moment, and fdbcli is usually the only thing we have. When you are remotely troubleshooting some issue, during an outage or minutes before a demo, inside some docker image, and the only tool you have at hand is fdbcli, it is to forget about all the potential damage. (_minimal docker images, that does not include python nor the python fdb bindings)

I think that since metadata caching is relatively new, maybe not a lot of layers are using it, and this issue may not be very visible? I’m also probably more impacted because I have implemented metadata caching in the Directory Layer itself (.NET binding) while other bindings have not, and have created a pattern in the .NET binding so that all the layers will directly or indirectly using metadata caching, so any accidental change will completely break everything…

Regarding python scripts, I think it’s just moving the problem: if the creator of the script is not aware of this issue, and forget to touch the metadata version key, the same thing could happen? Or inversely, if the owner of some layer decide to add metadata caching at some point to speed things up, it could break already written scripts or due to mis-communication between people, nobody would remember to fix the script accordingly.

–

I’m not sure exactly sure how to solve this issue globally: there’s nothing that can prevent anyone using fdbcli, a custom python script, or any other tool, to unknowingly break something that relies on the metadataVersion key for caching.

Current metadata caching solutions relies absolutely on the fact that a single global key will be changed, which may not happen in some situations.

It looks to me that the only robust way to build caching in complex layers, would be to have a way to ask if a range of keys has been changed since a specific read version, or create a write conflict range - using a read version in the past that would conflict the transaction at commit time. This would be immune to any change made by any tool or binding, while still being mostly efficient (would either induce latency at the start of the transaction, or induce conflicts at commit time). It would also decouple all the layers from a single global key, to one key (or range of keys) specific to that layer.

4

But now you’re trying to solve a debugging problem by potentially breaking production behavior for other people (for example one might implement a canary using fdbcli and they certainly wouldn’t want to update the metadata version key).

I still think that doing this is too much magic. A set in fdbcli is kind of expected to behave like a set Transaction::set - so this is imho not intuitive.

Also what exactly should fdbcli write into the value? I am not very familiar with that feature (and documentation is basically non-existent) but I thought a user could potentially add custom data there?

It seems to me the proper solution to your problem is to add the functionality of clearing the database to your application.

5

Unless I missed something, this key is supposed to contain a versionstamp that you update via an atomic operation. Since it is a “shared” key that any layer could mutate, any custom data written to this key would probably be clobbered by the next layer at any time.

Also, true I could write whatever script/tool to avoid creating problems and avoid fdbcli altogether.

But if caching directory subspaces starts being available in more bindings, and/or people start using more layers that use this key, I don’t know how to prevent others from doing this mistake. Especially since fdbcli is the only tool shipped out of the box and there are multiple forum/SO posts that explicitly point to doing a clearrange in fdbcli to clear the database (as the most common scenario I can think of).

It seems to me that asking everyone to never make any mistake is not a good strategy. I feel it should be “safe” by default. If, like you say, people who know better will always use custom scripts, then it looks like the only ones who will use fdbcli are newcomers that have not yet read this thread warning about this potential gotcha ! Why not make it so that they don’t learn it the hard way?

There is already the “writemode on” command required to make it a little bit harder to set things (it used to not be required a long time ago). I’m not sure what motivated the devs to make mutations opt-in in fdbcli? Would it be possible to make “on” change the metadata version key by default?

6

Well, I ended up scrapping the use of this key for the Directory Layer (and other layers) because this issue and others made me rethink the caching strategy (see Deferred value checks as an alternative to the metadataVersion key for caching in layers)

I still think that tools shipped out of the box should be “safe” by default, though.

I hope this won’t bite too many people in production!