Comments on Deno KV's use of FoundationDB requested

Using FoundationDB
1

Deno KV internals: building a database for the modern web | Deno details their use of FoundationDB at scale for a key-value store with a nice API to TypeScript. The striking feature is their introduction of an additional “transaction layer” on top of FDB, which I found surprising. The cited reasons are mostly around lack of additional pushdown predicates to limit the number of network roundtrips, if I understood correctly.

The HackerNews discussion of this Deno KV internals: building a database for the modern web | Hacker News has posts that share my surprise, while the Deno developer that did the FDB integration thinks that additional layer is pretty much necessary for their use case: losfair's comments | Hacker News

My question to the community: Is it really true that Deno had no alternative? Is there a simpler way that avoids a complex extra layer?

Any insights greatly appreciated!

1 Like
2

Early on in the blog post, they mention one of the reasons they built an extra layer on top of FDB:

Deno KV can store signed varints (bigint), and we want to support an atomic sum operation on these, but FoundationDB itself does not support atomic sums on varints.

Foundation DB provides only a limited set of server-side operations. You can perform server-side additions on 64-bit integers, but not on varints, as Deno KV desires. This leaves you with two options:

  1. You perform the operation on the client within a transaction.
  2. You write a proxy layer in front of FDB which perform the operation.

If your proxy layer is deployed on the same network as the Foundation DB cluster, you can minimize the latency between this layer and the DB and potentially get better performance on server-side operations. Also, the proxy layer could batch several operations from different client into a single transaction.

At my job, we have proxy layers for certain key-spaces for the above reasons. Instead of talking to FDB for those key-spaces directly, we expose a set of RPC endpoints which do the operations on behalf of the client.