# A new tool for managing layer metadata **URL:** \ **Category:** FoundationDB Core\ **Created:** [March 2, 2019, 2:24am UTC](https://forums.foundationdb.org/t/a-new-tool-for-managing-layer-metadata/1191 "2019-03-02T02:24:25Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Evan](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/evan/32/104_2.png) [@Evan](https://forums.foundationdb.org/u/Evan)\ **Post date:** [March 2, 2019, 2:24am UTC](https://forums.foundationdb.org/t/a-new-tool-for-managing-layer-metadata/1191/1 "2019-03-02T02:24:25Z")
I am moving discussion from [Added a metadata version key by etschannen · Pull Request #1213 · apple/foundationdb · GitHub](https://github.com/apple/foundationdb/pull/1213) over to the forums. A common pattern in layers is that there is a rarely changing schema that needs to be validated within each transaction. These reads of a small amount of data can be a burden on clusters, because the reads are concentrated on a very small section of key space. The PR adds a key to the system keyspace which has its value sent to the client with every read version. This means that clients can see the value of this key without communicating with storage servers. **Q1:** > [@](#): > > Does this mean that a `get('\xff/metadataVersion')` is guaranteed to complete synchronously? or can it still block, like for example if the read version is not yet know by the transaction? If the read version is not known yet, it will block until the read version comes back from the proxy. If you already have a read version, it is guaranteed to be synchronous. **Q2:** > [@](#): > > There is only one key for the whole cluster, so does this mean that layers have to be conservative and not change the key too often? This should not be too big of a concern. Each change will cause every client to invalidate their cache, so if you have 1000 clients, changing it will cause 1000 reads to the location where metadata is stored. **Q3:** > [@](#): > > Could this metadata version key be used by the directory layer to signal important changes, for example when deleting or changing the prefix of a directory subspace? or is it not one of the indended uses of this feature? This is a plausible use case. @ajbeamon might know more about how easy it would be to implement. ---
**Author:** ![mengxu](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/mengxu/32/893_2.png) [@mengxu](https://forums.foundationdb.org/u/mengxu)\ **Post date:** [March 2, 2019, 6:55am UTC](https://forums.foundationdb.org/t/a-new-tool-for-managing-layer-metadata/1191/2 "2019-03-02T06:55:42Z")
In the GitHub PR, I didn’t quite get how this metadata key should be used to mitigate the hot key problem. > [@](#): > > > The metadata version key “\xff/metadataVersion” is a key intended to help layers deal with hot keys. The value of this key is sent to clients along with the read version from the proxy, so a client can read its value without communicating with a storage server. Can anyone (@alloc @Evan ) give a simple example to explain how this version key should be used? ---
**Author:** ![ryanworl](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ryanworl/32/440_2.png) [@ryanworl](https://forums.foundationdb.org/u/ryanworl)\ **Post date:** [March 2, 2019, 1:21pm UTC](https://forums.foundationdb.org/t/a-new-tool-for-managing-layer-metadata/1191/3 "2019-03-02T13:21:12Z")
The shortest example I can think of is if you have a document store that can add and drop indexes at runtime. You need to maintain a list of which indexes are active, and one way is to read a key during every transaction that holds the list. That way any time an index is added or removed you will see it right away. If every transaction has to read a single key, you will eventually overload the storage servers with that key. It also taxes every transaction with some small amount of latency (although you could probably hide it by optimistically assuming the schema is valid and checking it yourself sometime during the transaction later). Note that you still need to manage the backfilling and deleting of indexes in the background, you’ll just observe the state transitions thereof on all clients immediately. The alternative is to somehow cache the schema. The goal is to operate the cache such that any client only has either the current version or the previous version of the schema. If you can maintain that, you can use the online schema change protocol from F1. This is possible in FDB but takes some amount of code every layer that wants online schema changes has to write. A bug in this code is almost guaranteed to create inconsistencies in the data, such as index entries that point to nothing. This change sends a third type of version (other than read and commit), the “‘metadata version” of a transaction, back to the client when they begin a transaction. This lets clients cache the schema and invalidate it based on the metadata version being different than what they’ve got cached. When a client detects a change, it can read the actual schema key and continue serving requests. This means only the transactions at that time need to read the actual schema key. ---
**Author:** ![mengxu](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/mengxu/32/893_2.png) [@mengxu](https://forums.foundationdb.org/u/mengxu)\ **Post date:** [March 3, 2019, 4:48am UTC](https://forums.foundationdb.org/t/a-new-tool-for-managing-layer-metadata/1191/4 "2019-03-03T04:48:18Z")
Thank you very much for the example! Now I got the idea. As a summary: To solve the hot keys, we can cache the keys in the layers (e.g., RecLayer or DocLayer). To keep the cached data consistent with the data in DB (storage servers), we need to invalidate the cache whenever the data is changed. The metadata key can be used as a mechanism to notify the layers/clients that their cached data may have changed, and they need to invalidate their cached data. ---
**Author:** ![KrzysFR](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/krzysfr/32/43_2.png) [@KrzysFR](https://forums.foundationdb.org/u/KrzysFR)\ **Post date:** [March 3, 2019, 4:24pm UTC](https://forums.foundationdb.org/t/a-new-tool-for-managing-layer-metadata/1191/5 "2019-03-03T16:24:57Z")
Will we also be able to set a watch on this key and be notified when it changes? (for monitoring tool, etc…) ---
**Author:** ![mbhaskar](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/mbhaskar/32/277_2.png) [@mbhaskar](https://forums.foundationdb.org/u/mbhaskar)\ **Post date:** [March 4, 2019, 3:54pm UTC](https://forums.foundationdb.org/t/a-new-tool-for-managing-layer-metadata/1191/6 "2019-03-04T15:54:49Z")
@Evan considering this key is in system key space, do we need to set the option `ACCESS_SYSTEM_KEYS`, on the transactions reading or writing this key? I hope that’s not the case at least for read transactions, as that could be every transaction. ---
**Author:** ![ryanworl](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ryanworl/32/440_2.png) [@ryanworl](https://forums.foundationdb.org/u/ryanworl)\ **Post date:** [March 4, 2019, 4:02pm UTC](https://forums.foundationdb.org/t/a-new-tool-for-managing-layer-metadata/1191/7 "2019-03-04T16:02:54Z")
> The key has been given special treatment in `NativeAPI` and `ReadYourWrites` (RyW being the relevant one here I think?) ---
**Author:** ![Evan](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/evan/32/104_2.png) [@Evan](https://forums.foundationdb.org/u/Evan)\ **Post date:** [March 5, 2019, 5:24am UTC](https://forums.foundationdb.org/t/a-new-tool-for-managing-layer-metadata/1191/8 "2019-03-05T05:24:48Z")
As Ryan figured out, you do not need to call ACCESS\_SYSTEM\_KEYS to read or modify this key. I did not add support for watching this key efficiently, meaning if you watch the key, the watch will go to the database like any other watch. Is there an actual use case for using the watch API? In some sense the whole point of the key is to watch for changes in a transactional manor, and the watch API creates a future which is separated from the transaction that created it. ---
**Author:** ![KrzysFR](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/krzysfr/32/43_2.png) [@KrzysFR](https://forums.foundationdb.org/u/KrzysFR)\ **Post date:** [March 5, 2019, 8:37am UTC](https://forums.foundationdb.org/t/a-new-tool-for-managing-layer-metadata/1191/9 "2019-03-05T08:37:17Z")
> [@Evan](#): > > Is there an actual use case for using the watch API? This would be for monitoring tools or data visualization tools that could be notified when the “schema” is changed and automatically refresh the page. Currently, you need to actively read the key to notice that it changed, which would require a monitoring tool to do polling. I don’t think that this would be a problem if the watch would go to the database for this particular use case. ---
**Author:** ![kocolosk](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/kocolosk/32/412_2.png) [@kocolosk](https://forums.foundationdb.org/u/kocolosk)\ **Post date:** [April 3, 2019, 8:09pm UTC](https://forums.foundationdb.org/t/a-new-tool-for-managing-layer-metadata/1191/10 "2019-04-03T20:09:21Z")
This looks like a cool enhancement. I’m wondering if there’s been any further discussion about Q3 (the use of `metadataVersion` by the DirectoryLayer). It feels like a very natural integration, and one that I think we could use to good effect to model databases in Apache CouchDB. /cc @ajbeamon ---
**Author:** ![ajbeamon](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ajbeamon/32/13_2.png) [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)\ **Post date:** [April 3, 2019, 11:30pm UTC](https://forums.foundationdb.org/t/a-new-tool-for-managing-layer-metadata/1191/11 "2019-04-03T23:30:21Z")
I don’t think it would be too hard to add support for it, though I’m not sure yet exactly what the API for it should look like. I created an issue in GitHub to track it: >