DR streaming and locking only a range of keys

If I have understood correctly the suggested use case would be to operate a single FDB cluster and have multiple layers provide service for multiple applications on top of them.

At some point one of the applications might grow big enough to warrant setting up a dedicated FDB cluster for it, some application might require a different level of data redundancy so moving to an another cluster would be needed, some application would be better served from a different DC, or some by a different infrastructure provide altogether.

For this kind of a migration to happen smoothly, it might be beneficial to be able to stream changes happening in one or several ranges of keys to an another FDB cluster (much running a DR job), and then be able to atomically issue a write lock on all of those ranges within the cluster, so all modification requests going to the “old” cluster would fail.

There are some strategies I feel one could employ on the layer level to get the locking (and possibly also transparent query forwarding) to work, and when combined with the current DR backup one could maybe make a migration happen within a trusted domain… But if security reasons or resource planning reasons justify exposing only a range of the data to the recipient cluster, a top level feature to be able to filter a DR job would be needed. Also having a top level feature to lock a range would allow doing these migrations much more easily for application consisting of multiple layers, without having to modify each layer separately for the migration purpose.

Does anyone else feel like these would be valuable additions to the core?

I think these are actually largely already implemented in the core.

I’d suggest reading over fdbdr if you haven’t already. I believe fdbdr switch provides the atomic write lock on one side and unlock on the other side that you were looking for. fdbdr start -k provides the key-restricted streaming. Part of the DR framework is having dr_agents running as well.

I believe the write lock that’s provided is entire-database wide, so that piece might still be a valid feature request if it’s something you’re interested in.

Oh I had completely missed that also the fdbdr start has the -k switch! On my first reading I gathered it was fdbbackup feature only, but I am happy to be mistaken :slight_smile:

Do you know if it is possible to have multiple fdbdr backups with different -k values running simultaneously? The documentation states something about the job not starting if one is already running, and upon reading the discussion here about a the watch feature for a ranges of keys, it seemed to me that the implementation details might provide some challenges on this too (unless the dr implementation is completely separate)?

The reason I am asking about all this is that I am looking to build something which would allow different tenants using their own subspaces on a shared fdb cluster, and having those subspaces be easily migrated from a shared fdb cluster to an another shared fdb cluster. My current plan is to provide the isolation on the layer level so tenants would not be able to directly interact with fdb, but I am wondering if there are any hidden security implications I should be aware of when using fdb in a multi-tenant environment?

I think isolating on the layer level would also mean that I could in theory build the locking feature as shared code on the on the layer level, but having the checking of a lock at the start of each layer operation would not be optimal, and having the mentioned switch feature also have the -k range filter would definitely help with this. Do you know if there are any fundamental design limitations that would prevent adding a feature like this?

Yeah, that’s possible. You would specify that by using the -k flag multiple times, e.g.:

fdbdr start -s source.cluster -d destination.cluster -k "range_1_start range_1_end" -k "range_2_start range_2_end"

One caveat here is that the cluster locking that the DR process does is done over the whole cluster rather than just the DR destination range, so if you wanted to DR a subspace somewhere else and continue doing other things in other ranges on that destination, you might run into problems. I think in your particular case that you will run into issues when you move the data from one FDB cluster to the other if both of these shared clusters are active.

I don’t know if I’d say that locking is done at the cluster level for any deep architectural reason (instead I expect it is because the most common case is expected to be using DR to backup an entire cluster to a secondary cluster), but adding locking on a key-range basis would probably require a fair amount of plumbing and performance wise would probably be about the same as checking a lock at the layer level.

If you’re interested in doing something multi-tenanted on top of FDB, then I would say that probably handling that in some kind of isolation layer on top of FDB is probably the correct call. I will point out that FDB itself doesn’t offer any kind of login mechanism, so if you have two tenants hitting the same cluster, nothing can stop one client from reading the data associated with another client unless you do something like put a proxy layer in-between that handles that. That’s the main security concern I would have. Additionally, there is no per-client load balancing done at the FDB level, so if you have two users sharing this cluster, if one client saturates the database, the other client will also see degraded performance (despite doing nothing else). If you wished, you could consider this to be a vector for a DOS attack (in that one client can deny the other client access to the database), but I would usually just think of this as a kind of performance pathology.

To move data around, there isn’t really a better option currently than handling that yourself, either by stopping all operations for a user while you shift data from one cluster to another and then starting them back up once the data have been moved or by doing something more sophisticated where you are careful about where and when you send updates and serve reads. (For example, when you move data, you could start logging all updates to the source cluster for the subspace you are moving using versionstamp keys to maintain transaction ordering while simultaneously copying over the subspace you’re copying. While you do these reads, you also read from the list of updates and apply any to ranges you’ve already copied over. You keep doing this until you’ve moved everything from the source to the destination. Then you stop taking writes, copy over any lingering updates, and then make the destination active. You serve reads from the source cluster until you’ve copied everything over, at which point you start reading from the destination cluster.) If you’re doing something like keeping a mapping (somewhere) of a subspace to shared cluster so you know where to read, you can keep meta-data about which cluster ranges are locked and which ones are being moved. You have to be somewhat careful here if you cache that result (which, like, not the most unreasonable thing), but I think it could be done.