Introducing the Redis protocol FDB Gateway

Development FoundationDB Layers
1

I’ve written a proof of concept for an idea that’s been banging around in my head for a while. I didn’t write it with an existing production use case in mind, but I think the code is reasonably easy to read regardless.

The idea is that FDB today doesn’t provide any access control at the KV layer, which means you need something between your end user / end developer to mediate access and prevent a rogue user from issuing tn.clear_range("\x00", "\xFF") and causing a very bad day for everyone else.

If you could write another layer that as transparently as possible spoke the FDB API (not necessarily network protocol), you could shim in this access control layer along with a few other features. My first version of as transparently as possible is to use the Redis protocol, as there is an implementation in every popular language and it provides enough features to model the FDB API on top off.

So I call this the Gateway, since Proxy is already taken as a component name in the FDB architecture. You connect using the Redis protocol, issue a chroot command to put your client into a directory using the directory layer, and then can transparently issue commands as if you own the entire instance. You can’t access the system keyspace (such as the metadata version key), but I could provide an escape hatch for that later using some kind of ACLs system.

To use it, you first set a tuple in the directory sys-tenant-authorization in the formation of ["chroot_name", "username", "password"] => password. This can be re-evaluated, but I just copied Redis’ design for the authorization system coming in Redis 6 where you have any number of users and passwords and can rotate passwords by having two passwords for the same user for a period of time.

The commands you can issue are named after the FDB API, but I didn’t implement everything yet. Just the basic ones like get, set, get_range, clear_range, clear. To do any operations at all, you must chroot into a directory and start a transaction. This is to prevent you from doing something wrong if you disconnect and reconnect to a different Gateway and attempt to start issuing commands again as if you’re in the same transaction as from before you disconnected. You’ll receive an error saying there isn’t any active transaction if you try. Same thing if you forget to chroot, you’re denied.

Other than obvious stuff like more features, does this interest anyone here?

It could fulfill 3 roles:

  1. Provide a basic security boundary in a multi-tenant environment without sacrificing expressive power on the part of the tenants using the FDB API. This is useful for environments where an entire FDB cluster for each tenant is cost prohibitive or undesirable from a management perspective.
  2. Provide an interface for an existing Redis-using application to replace Redis. This is more useful if additional Redis commands were added, but GET and SET are the same AFAIK.
  3. Provide a generic platform for community investment in features not in core FDB today, such as predicate pushdown. One could imagine the Gateway coordinating with other Gateways deployed on storage servers using the boundary key API to match predicates of keys or values in large scans before returning them to the client. Another idea I had is this is a safe place to implement scripting like Redis has today with Lua. Instead of deploying an entire application with client bindings and managing all that, you could write a Lua script that does the transactional bit of your interaction with FDB entirely on the Gateway and returns and entire fully formed result, like a stored procedure. An example stored procedure is the high-contention queue pattern could be implemented in Lua.

There are a few negatives to using this. You do not get the current behavior of zero-latency set because to you have to send the data to the Gateway for it to call the real set on the transaction. I think this is tolerable given you would be deploying the Gateway in the same datacenter as your application and you can pipeline multiple set commands as you can with Redis today. The second and more serious concern is maintaining correctness. My testing strategy (which I haven’t started yet, just thought about) would be to write a workload like the existing binding tester which does some operations and compares them to the operations of a regular API client. Past that, I think the regular guarantees of FDB would still apply to the Gateway.

Please poke holes, request features, and express your positive (or negative) feelings! :grin:

I’m not super invested in this conceptually, so I would prefer to hear why this wouldn’t work well, is not possible to make safe, or is not desired by anyone now before I write any more code.

4 Likes
2

I’m sure I’m not the only one that would like some (any) access control built into fdb itself. Your idea of a chroot like mechanism would be a significant improvement. I appreciate that access control gets controversial almost immediately and there’s a million options (and opinions!). That said, if clients could proactively drop their privileges with a chroot (the client is constrained to a key range or to a directory (and all subdirectories), say), fdb could provide a lot of (opt-in) safety without having to develop (and argue about) a richer access control system. Maybe?

1 Like
3

Would something like this help - https://github.com/apple/foundationdb/issues/1419 ?

1 Like
4

My use case is a little involved, and certainly there’s a role for the kind of restriction in 1419. This is for the CouchDB-on-FDB development but also in the IBM (Cloudant) context where there’s a strong multi-tenant component. I’m hijacking your thread so will wrap it up here, but ideally there’s a way to stop a client wandering into someone else’s keyspace that the server enforces unilaterally. A weaker but acceptable approach is like you have here, where each clients first action is to chroot itself (in fdb context, that’s constraining yourself to a subset of keys, the most basic form being a single key range, but 1419 also sees the problem with directory layer. It’s logical to grant access to a directory and all subdirectories, which a single key range could not deliver).

One thing we’ve considered at IBM is either a small fork of FDB that adds the access control we’d like to see for our specific needs or, slightly more appealing, is a tiny shim/proxy we put inbetween that enforces that. We’re only looking at preventing mistakes in client code rather than a strong form of access control that implies we have no trust in our clients at all (after all, we’ll be the authors of all that client code, our users/customers won’t have direct access to the fdb cluster).

5

The solution I posted only allows you to do anything once you chroot yourself, you get nothing by default. I think you can also run the directory layer in a mode in which subdirectories are prefixed with the root directory. My intention is actually that if you wanted the features of the directory layer (which you may or may not), you would re-implement it as a part of the application and the gateway wouldn’t have any knowledge of it. Your directory layer (as opposed to the global one used to enforce chrooting and where the authentication data is stored) is entirely yours.

You could imagine having language bindings that exposed the same API and almost the same semantics as the official bindings and porting code to use them would be trivial.

6

Ah, I missed that, and that is better obviously.