I feel like this should be better documented as a pretty powerful strategy for caching data structures in memory with periodic refreshes (given structures that are large like lists of things or a hashtable):
- Write the information to a prefix that one would scan to load
- Write the information with a Versionstamp somewhere in the prefix (write it after the part that’s usually the cache key, for instance, say you need to have a cache of all friends of a user, put the Versionstamp after the user ID but before the friend IDss)
- (Optionally, have something that garbage collects those Versionstamp rows by doing GRV and purging older writes that were made in 2)
- On initial load, read from disk the non-Versionstamp region and remember the read version that it was read with (you always have that as part of the read transaction, a lower bound would have been fine so even batch readers would work: https://github.com/wavefrontHQ/foundationdb-utils/blob/master/src/main/java/com/wavefront/fdb/utils/BatchReader.java)
- In periodic refreshes, scan the Versionstamp region by constructing a prefix that has the Versionstamp of the last known read version (the tricky part).
public Versionstamp toVersionstamp() { return Versionstamp.fromBytes(ByteBuffer.allocate(Versionstamp.LENGTH). order(ByteOrder.BIG_ENDIAN). putLong(version). putInt(0xffffffff). array()); }
- Enjoy reading exactly the rows you need to update your data structure with (for instance in Java, one can use Caffeine with a refresh() function that does automatic period background refreshes). One can also include delta (both additions and deletions) in the Versionstamp region.