Neon: a versioned quad store that works like git requires a lock

(Amirouche) #1


I created a versioned quad store on top of wiredtiger. The thing is that it requires a lock to work that protects a datastructure that is shared between all threads working against the same branch. In the context of foundations this would me there is distributed lock shared by all clients and a datastructure. What do you think I can use to replicate this lock using foundationdb? I am thinking about using REDIS. Maybe you know better.

The code is at github with the documentation.

Feel free to give feedback on that idea of this kind of layer. Would you use something like that given it’s a single writer multiple reader on a branch but think that append only transactions can be merged in master easily.

If there is no interest in such a layer I will fallback on non-versioned quadstore, that doesn’t require a lock to work properly or maybe I will create the quadstore to get started with Foundation API anway, not sure yet.

(David Scherer) #2

You can certainly make locks in FDB, and furthermore you can do things to the database atomically with releasing the lock, which is pretty vital to making locks async fault tolerant. But you will have to think about what happens when the holder of the lock fails.

You should also consider an optimistic strategy: just go ahead modifying your shared data structure and trust FDB to abort any such modifications that conflict. It’s not necessarily more scalable than locking, you will still be latency bound, but it’s simpler.

(Amirouche) #3

I did not think about that it’s indeed much easier than the lock.

I keep you posted of my progress.

(Amirouche) #4

Does it make sens to have create a timeseres database on top of foundationdb?

(David Scherer) #5

Sure. In fact, Wavefront’s product is at bottom more or less a time series database.