Layer interoperability

Development FoundationDB Layers
1

So I had an idea (probably not original) for how to write layers that interoperate across languages or across process / machine boundaries. This would be useful if someone had a highly-developed layer with a lot of functionality that you would like to use in the context of your transaction, but without needing to fork the layer.

It goes roughly like this:

  1. Layer One starts a transaction, gets a read version, and issues an RPC to Layer Two (e.g. another machine) containing the read version it just got.
  2. Layer Two starts a transaction and sets the read version to the read version from Layer One.
  3. Layer Two performs all operations normally, but doesn’t commit. It takes all the mutations and conflict ranges that would’ve been committed and sends them to Layer One, along with the higher-level result of the query/operation that has semantic meaning to Layer One (the real result of the RPC).
  4. Layer One examines the result of the RPC and decides to commit or abort the transaction as if it had performed all that work itself.
  5. In this example Layer One commits, so it applies all the mutations and conflict ranges from Layer Two. If it had aborted, nothing would’ve happened because Layer Two also never committed the transaction.

This probably relies on a feature somewhere that doesn’t exist yet around being able to examine the contents of the transaction’s mutations and conflict ranges, but I don’t think it does anything impossible. :sweat_smile:

I also don’t think it invalidates the workload management of FDB too much because the Layer One transaction will still wait in line for the read version if needed. Layer Two is doing work on behalf of Layer One that, under a monolithic layer model, it would’ve performed anyway. It is just happening somewhere else. It does have the potential to cause more wasted work if you spend too much time in the RPC transaction between One and Two and go over the 5s limit. It does add additional network traffic proportional to the size of the mutations and conflict ranges which would’ve only been sent by the client once which are now being sent by the Layer Two back to Layer One, then through the transaction pipeline.

This does rely on the layers trusting each other to both behave correctly (ignoring security, still not a security boundary), but I don’t think that’s a terribly large hurdle if you hide this functionality behind a wrapper library, as the results of the RPC and/or the side effects of it are the only thing the calling layer will need. The mutations and conflict ranges are an implementation detail.

2

Doesn’t Layer One also need to pass the mutations it (or Layer Three) has already done, so that Layer Two operates in this context, rather than the original of the read version?

3

I hadn’t considered that aspect, but that seems true. My scenario didn’t envision a dependency between the operations Layer One already performed before calling Layer Two, but that’s a reasonable thing to expect to work.

4

I think there are two choices:

  1. Each layer has some functionality another layers does not have. So we want to stitch the functionalities from different layers. (This needs to solve Mike’s question).
  2. One main layer has all functionalities and other layers just call the main layer to do the work. Then we don’t need to worry about the question.

I’m not sure which one is better, though. Just some thought.

5

We would probably need some new API on transactions:

  • one that can export all the mutations + read/write ranges that a transaction has currently done into some binary blob
  • another one to import this blob into a transaction and make it as if this transaction did all of that by itself.

note: the blob would probably compress well due to all the common key prefixes

Layers could exchange this blob between them (up and down), and the “top” layer (the one who created the initial transaction) could then aggregate all of this, and do the commit ?

Though, there are some layers who have to do some things after a successful commit, so how would they reliably know that the top layer succeeded in committing the transaction?