We have been working on a PR that introduces a framework that we call special-key-space. Special keys are keys prefixed with \xff\xff. The framework associates all these keys with their corresponding client functions. In particular, each function will be registered to a range under (\xff\xff, \xff\xff\xff). The framework will proxy calls to corresponding functions and aggregate results if cross-range read happens. (A more detailed documentation for the context here.)
This post wants to get an agreement on how this should behave across different versions.
Currently, our proposal is to obey the API_VERSION that each client passes when initializing the client like this:
Each range-module would be tagged with an API-version when it was introduced
The overall framework would then ignore all modules whose API-versions are larger than the one the client set.
Each submodule could read the API-version as well, which would allow it to behave differently depending on the API version. This would allow for a schema change
Any other ideas are welcome to discuss and any concerns should be solved before implementation.
I just wrote this on the GitHub issue created for this, but I’ll repeat here as well. It may not be required that newly added features be disallowed in older API versions, as that isn’t required by the contract of API versioning and is not something we have enforced for most other features that get added. It is, however, a reasonable choice to exclude these in older versions if we want to do so.
Usually one of the primary motivations for not excluding a new feature in old versions is that it adds implementation complexity unnecessarily. Based on your proposal, though, it sounds like this may not be a big concern here. I just wanted to throw this out there in case it was relevant.
I think the problem here is that C functions and key ranges are a bit different in the sense that functions don’t accidentally show up. For keyspaces this is a bit different, if someone does getRange("\xff\xff/", "\xff\xff\xff") the client might not expect the result set to change across different API versions.
I haven’t looked at the design of the special key-space feature, but is it going to support doing arbitrary range reads over it? I don’t think existing special keys allow you to do this, rather you need to target the special key specifically.
Yes that is a change we did there (and we can change this if people think it is a bad idea). The thinking was:
It is much more consistent with the way the normal keyspace behaves
It doesn’t break old code as it enhances functionality without restricting it.
Personally I would prefer to allow this but restrict which ranges are exposed into this keyspace than throwing exceptions if one tries to access something that doesn’t exist. Otherwise the following can happen:
Wee add \xff\xff/foo in api version 700
Someone sets api version to 620 but adds code somewhere that gets \xff\xff/foo. Let’s assume that for testing the user uses fdb 7.0
In production this user runs 6.2 - but now this will cause weird exceptions.
I don’t think this is desirable which is why I would prefer to keep this as stable as possible over version.
@alloc expressed similar concerns before. Can you give us your opinion on this (and other stuff regarding this proposal) please.
One downside of allowing arbitrary range reads is that new “keys” may be introduced that appear in your existing range reads, and these keys don’t have the typical costs associated with a normal key. If we take the status key as an example, injecting that into someones range read would add an expensive status call to every such read, and may be unexpected.
My personal thinking is that it would be reasonable for what I think you are calling modules to support arbitrary interactions within the module, but supporting operations querying across modules is something that we should carefully consider whether it makes sense.
Yeah, it is definitely right that unexpected “keys” appearing in your range is troublesome.
My point is that this may not happen in the special-key-space framework unless you do something like getRange(\xff\xff, \xff\xff\xff).
The way it works now is every time you bundle a new feature to a range, it behaves like:
It only allows registering to a range
Any new range should not be overlapped with any registered ranges
In particular, if you want to register a function to \xff\xff/status/submodule/, \xff\xff/status/submodule/\xff and \xff\xff/status/json, \xff\xff/status/json/\xff is already registered, then this behavior is not allowed.
Consequently, the unexpected behavior of a module is only going to happen when the underlying function is changed. In this case, the developer who makes the change should let users know the change. For existing single key get, we can register them to a singleKeyRange.
A potential drawback here is that the existing registered range takes whole control of the range and you cannot add new sub-modules to it. However, you can still change the implementation or split the range if you do need to add stuff in the range. Every range added should be reviewed carefully so conflicts will not happen in most scenarios.
The other question I have is whether doing a range read of \xff\xff to \xff\xff\xff is legal, as doing something like that spans multiple modules, and the introduction of new ones could be a costly surprise. Likely someone wouldn’t be reading the whole range, but with any range that spans multiple modules, a new one could be introduced in between.
This is the thing I was considering the merits of, and I think the possibility of someone doing it is one reason Markus gave for not including new modules at old versions.
Doing a range read over (\xff\xff, \xff\xff\xff) is by default legal now, which may introduce unexpected results as you said. Yeah, if we exclude new features in older versions, we can avoid the issue. Some additional thoughts here
Yeah, as you said, the implementation complexity is not much concern here. The versions can be easily integrated within keys.
The motivation to allow range read over modules are scenarios like this:
We use \xff\xff/transaction/, \xff\xff/transaction/\xff to expose all transaction-related information. For example. \xff\xff/transaction/conflicting_keys, \xff\xff/transaction/mutations, \xff\xff/transaction/read_conflict_range, \xff\xff/transaction/write_conflict_range and so on.
I think it makes sense that sometimes you want only one of them and sometimes you want all of them by just calling getRange(\xff\xff/transaction, \xff\xff/transaction\xff). To achieve this, we may need to allow arbitrary range reads, which also makes it behave like general FDB calls. Actually, the goal of the framework is to make special keys’ calls like general FDB calls.
If the arbitrary range reads are still concerned, we can split the whole space into several top-level modules which can have their own sub-modules. Range read within a top-level module is allowed but cross-module range read between top-level modules is not allowed. However, this makes the framework more complex.
I think is what I was sort of imagining (e.g. \xff\xff/transaction would be a module), and that modules could enforce their own rules about what’s reasonable within them. The nice thing about this is that we avoid these behavior changes that only manifest at runtime and that would require thoughtful consideration for people upgrading, and if all else were equal I think it would be my preference. If it’s significantly more complex to do, though, then maybe it’s not worth it.
Right, I agree you shouldn’t add new ranges to old API versions if you are allowed to read arbitrary ranges. If it’s not legal, then I don’t think this particular concern is as important.
Unfortunately, with this only being detectable at runtime, we still have the risk of there being some surprises, but that’s probably acceptable with sufficient documentation of the changes between versions.
I think a missing facet of this is whether it’s ever not a misuse to read across modules. If there is no valid use, then disallowing it seems like a more friendly approach to me. If there is a valid use, then that use is going to be potentially subjected to these bad performance problems and would be a separate category of “risky but reasonable” actions.
Yeah, like discussed above. Although the current implementation is not, we can definitely add restriction on cross-module interaction if there is no real use case.
I guess that is fair. Although one problem with that is that the boundaries of modules might sometimes look a bit artificial - unless we want to build a tree of modules, but I don’t want to go there. I think there are multiple use-cases (although this depends on what we will expose through this mechanism in the future which is hard to predict):
Debugging: it would be useful to be able to read the whole range just to test stuff.
We will probably have multiple modules within the transaction keyspace (a range that allows the user to query the internal state of a transaction). Currently this only contains the conflict set - eventually I would like to have the read-set, write-set, options, etc in there as well. Reading the whole range is something that makes sense and should be cheap enough as it only reads local memory.
We plan to split the status json stuff into a flattened keyspace. This would allow us to ask for only a subset of that which can be implemented faster. But in that case if one still wants the whole status in a flattened way, they should be allowed to do it in one query (and it would also be much faster).
There are probably more usecases - but these are the ones I can think of right away. If this doesn’t convince you I will think a bit more about this
Oh I do agree that all of those types of queries are useful to be able to do as ranges. I’m not saying that range reads in this key-space are bad, just that range reads across modules seems like a less clear win, depending on how you define modules. Also keep in mind that I’m coming from a position of not having looked at the current design and am just imagining how something might work rather than how it might or actually have been built.
My initial thought with this was that it could be one module (with sub-modules?) that allows reading the arbitrary portions of it.
Same here, I could imagine this being a module where the whole range or parts of it could be read arbitrarily.
My main reservation is about when range reads end up invoking unrelated actions, and I’m not sure I see a strong use-case for that. Basically what I’m trying to say is that all else being equal, I would prefer that unrelated actions not be triggerable by the same operation, but I’m also sensitive to the fact that it may be difficult to accomplish that in practice. If that’s the case, I’m not sure that this protection would merit adding a lot of extra complexity.
I think I got your idea.
At present, every range registered with its function class can be seen as a module and cross-module range read is legal.
However, as you suggest and we discuss above, if we want to have the protection for unexpected results from cross-module behavior and preserve the ability to do range read to some extent, we can have a more formal definition of module.
Like \xff\xff/transaction/, \xff\xff/status/json can all be modules. And within them, range reads are allowed. Cross-module reads are forbidden.
I think this at least makes sense for me. The concerns are to have more documentation about how these modules are defined and restrictions on them and also less consistent with the general kv interface.
But if this protection overweights the concerns, I think we can add this module-level restriction to the current implementation in the future.
Sure we can do that - I just don’t like it. It is very different from what a user might expect.
We also allow users to query ranges across directories. So if all of FDB has a flat keyspace and the clients are responsible to to divide this up, I don’t think we should do something different here.
But I would like to get input from a layer developer on this - so far only FDB Core people have taken part in this discussion
There’s some truth to that, though arguably there are multiple behaviors here that could be unexpected and I think that potentially triggering unanticipated actions in an operation would be one as well. I tend to like failing a request that may have unintended consequences if there’s not a good use case to allow the risky requests.
I’m confused, there was some discussion above about these use-cases where I had discussed that I thought these made sense to be queried within themselves (e.g. status and transaction stats individually), and I still support that. My question is about use-cases where you would need to read across disparate modules.
