Lionrock: A gRPC Facade for FoundationDB and more

Finally got a chance to sit down and get something I have been working on to become open-source, published, etc. There has been the discussion of having an RPC layer that does not require the use of native libraries for the longest time from what I can tell. I also felt that the same RPC layer itself need not be tied to the project as you should be able to front a transactional, key-value store with a common protocol language (think SQL) that allows one to swap implementations (an in-memory version for testing? something that’s backed by RDS on AWS if you don’t need the scale/cost-profile of FoundationDB?).

Lionrock is a facade for Transactional, Key-value Stores

It includes:

  • An implementation agnostic protocol definition that’s heavily inspired by what FoundationDB does (of course, because it’s probably the best way to code against a transactional, key-value store =p)
  • An actual implementation of said API against a FoundationDB cluster (or multiple FoundationDB clusters) that allows horizontal scaling, multi-version client support, and ultimately packaged as a container that one could run and scale independently to the underlying cluster.
    [lionrock-foundationdb-server]
  • An implementation of Database and Transaction classes in fdb-java that uses gRPC underneath the hood which should allow (mostly) drop-in replacement of the gRPC-based client in place of the native library one (still need to depend on fdb-java for the interfaces and things like Tuple, we’ll have to break off an fdb-java-api artifact if we want to ditch the native library for good on the FoundationDB side)
    [lionrock-foundationdb-client]

Running the Server (until I get the image deployed on dockerhub)

./gradlew bootRun --args='--logging.level.io.github.panghy.lionrock=DEBUG'

Accessing a Server (from Java)

public class FdbClient {
  public static void main(String[] args) {
    ManagedChannel channel = ManagedChannelBuilder.forAddress("localhost", 6565).
            usePlaintext().
            build();
    // "fdb" is the default database on the server.
    Database db = RemoteFoundationDBDatabaseFactory.open("fdb", "my client", channel);
    db.runAsync(tx -> {
      tx.set(HELLO_B, WORLD_B);
      return tx.get(HELLO_B);
    }).join();
    channel.shutdown();
  }
}

Cool stuff

  • Paves the way for any clients to be built for all gRPC supported languages (help wanted!)
  • Removes the need to have multi-version client binaries on anything that touches FoundationDB
  • Supports things like watches and getting versionstamps post-commit (gotcha there is that watches will fail if the server dies while waiting for an update, just need to watch the key again)
  • The server publishes metrics and collects tracing spans (you can turn that on with a throw-away account on https://wavefront.surf by following the instructions). Can also be configured to publish prometheus metrics and zipkin traces.

Future Plans

  • Build an authorization layer on-top of the lionrock protocol that prevents some users from doing writes? Restricts access for some users to a certain key range?
  • Build a caching layer that allows stale reads?
  • Build a batching layer that batches blind writes together?
  • An in-memory implementation of the gRPC server packaged as a container for unit tests?
  • A pure gRPC CLI tool?
  • An implementation of the protocol against a SQL-database (probably not optimized for speed but just for simplicity in some deployments)?

I am aware of the efforts in:

Maybe it solves that problem/issue, open to discussion here.

My next milestone is to be able to run, say, the record layer with the gRPC client instead of the native one. Should be fun!

Traces from the Server (including async operations within a single transaction)

Screen Shot 2021-07-12 at 12.32.00 PM

11 Likes

Nice work! I also started some experiments using gRPC last week-end, but Lionrock is far more advanced than my prototype :smiley:

I can see that we both used the same rpc-style:

  • Bi-directional service to handle transactions,
  • oneof requests ands responses.

Metrics and zipkin traces are a nice bonus :+1:

Yeah, if you had a working rust client, that might not be too hard to port over.

And now with a CLI client (written in Java):

shell:>connect
connected to: localhost:6565 as: lionrock-cli accessing named database: fdb
shell:>get hello
`hello' is `world2'
shell:>set hello world
ERROR: writemode must be enabled to set or clear keys in the database.
Details of the error have been omitted. You can use the stacktrace command to print the full stacktrace.
shell:>writemode on
shell:>set hello world
Committed (105220050616009)
shell:>get hello
`hello' is `world'
shell:>watch hello
Committed (-1)
shell:>set hello world3
Committed (105220062686213)
shell:>
watch fired for key: hello

Still need to package it up and maybe compile it with GraalVM so it’s easier to run.

2 Likes

Now with GraalVM native images for linux + mac (for the CLI) with pico and jline3.

2021-07-14 22.38.54

Also hooked up to sonatype and maven central:
https://repo1.maven.org/maven2/io/github/panghy/lionrock/

1 Like

PR to split fdb-java into fdb-java and fdb-java-api (which allows us to ship a gRPC FDB client that should be mostly a drop-in replacement to the native library powered one):

2 Likes

v1.0.0 release is made

Finally one can run FoundationDB locally with Docker in self-contained tests (blank servers coming up with randomized ports, running tests in parallel). :slight_smile:

Also filed: Need to separate fdb-java into api and impl · Issue #5190 · apple/foundationdb · GitHub
and the PR: Refactor FDBDatabase and FDBDatabaseFactory by panghy · Pull Request #1344 · FoundationDB/fdb-record-layer · GitHub

Should allow the record layer to use a different implementation of fdb-java.