Versionstamp for batch loads + background refresh use cases

Using FoundationDB
1

I feel like this should be better documented as a pretty powerful strategy for caching data structures in memory with periodic refreshes (given structures that are large like lists of things or a hashtable):

  1. Write the information to a prefix that one would scan to load
  2. Write the information with a Versionstamp somewhere in the prefix (write it after the part that’s usually the cache key, for instance, say you need to have a cache of all friends of a user, put the Versionstamp after the user ID but before the friend IDss)
  3. (Optionally, have something that garbage collects those Versionstamp rows by doing GRV and purging older writes that were made in 2)
  4. On initial load, read from disk the non-Versionstamp region and remember the read version that it was read with (you always have that as part of the read transaction, a lower bound would have been fine so even batch readers would work: https://github.com/wavefrontHQ/foundationdb-utils/blob/master/src/main/java/com/wavefront/fdb/utils/BatchReader.java)
  5. In periodic refreshes, scan the Versionstamp region by constructing a prefix that has the Versionstamp of the last known read version (the tricky part).
  public Versionstamp toVersionstamp() {
    return Versionstamp.fromBytes(ByteBuffer.allocate(Versionstamp.LENGTH).
        order(ByteOrder.BIG_ENDIAN).
        putLong(version).
        putInt(0xffffffff).
        array());
  }
  1. Enjoy reading exactly the rows you need to update your data structure with (for instance in Java, one can use Caffeine with a refresh() function that does automatic period background refreshes). One can also include delta (both additions and deletions) in the Versionstamp region.