Can I force key prefixes into different shards?

Using FoundationDB
1

We’ve got an oddity here that we’re trying to fix in our code but I’m also looking at things I could do within FDB itself.

We’re running FDB 7.1 currently. We’ve got a relatively small range of keys that are very heavily read from and quite heavily written to. Querying the FDB metrics keyspace seems to show that all of those keys are in the same shard.

Looking at our overall sharding breakdown, we’ve got several prefixes that we know are quite large (and also have quite large values) and have been split across shards, but those prefixes tend to be ‘write once, read almost never’. It’s basically audit/accounting data that isn’t used by the rest of the system day-to-day.

But it looks like all of our (surprisingly small) amount of data that is used currently by the primary application has ended up in the same shard. Or maybe 2 shards. We’ve basically got a shard from "" to <some_value_1>, a whole load of shards from <some_value_1> through to <some_value_2> that all have the same common prefix (as in they all start X Y, but the first might be X Y through to X Y C and the second is X Y D … etc) that encapsulate all that accounting data, and then a shard from <some_value_2> through to \xFF.

When we run application load tests we see 2 FDB storage nodes getting absolutely hammered and their durability lag spiking massively, which I think is because it’s the processes hosting those first and last shards (or maybe even just replicas of one of them), so they’re getting a whole load of writes and all the reads.

My understanding is that a shard can’t be split across processes (it has to be split into multiple shards first), and a process can host multiple shards. I’d really love to be able to force certain key prefixes to be in separate shards, even if for the time being they end up hosted on the same process.

Because of the structure and size of our data currently, I think that even if we generate random/interned values for the prefixes we run the risk of all of the ‘core’ data being co-located in the same shard. Is there any way I can prod FDB into where I think data should be sharded beyond just ensuring there is no common prefix? I’d be happy for FDB to add more shards if necessary, but I’d like to be able to make it not aggregate certain keys into the same shard.

2

Good news, FDB can do what you want but you’ll have to upgrade to FDB 7.3. In 7.3 you can use the rangeconfig command to set different options for different key ranges which effectively injects explicit shard boundaries chosen by you. You can also increase the replication factor for a range. The range config rules are used as hints by DataDistribution to make shard splitting and team assignment decisions. Any adjacent key ranges which have different configurations will be split to different shards. If a configured key range is (or grows to) larger than a shard should be then the cluster will still split it further but the range config rule remains the same.

Note that this does not guarantee that each range you configure lives on a different storage servers, but they likely will especially since traffic metrics used for shard movement decisions would be separate for each range. Also note that you can’t decrease the replication factor for a range lower than the cluster’s replication mode. If you don’t wish to increase the replication factor for any ranges, just set your different targeted ranges to different teamID values and that will ensure they exist as different shards.

The rangeconfig command in fdbcli has help text which you can see in code here.