We’ve got an oddity here that we’re trying to fix in our code but I’m also looking at things I could do within FDB itself.
We’re running FDB 7.1 currently. We’ve got a relatively small range of keys that are very heavily read from and quite heavily written to. Querying the FDB metrics keyspace seems to show that all of those keys are in the same shard.
Looking at our overall sharding breakdown, we’ve got several prefixes that we know are quite large (and also have quite large values) and have been split across shards, but those prefixes tend to be ‘write once, read almost never’. It’s basically audit/accounting data that isn’t used by the rest of the system day-to-day.
But it looks like all of our (surprisingly small) amount of data that is used currently by the primary application has ended up in the same shard. Or maybe 2 shards. We’ve basically got a shard from "" to <some_value_1>, a whole load of shards from <some_value_1> through to <some_value_2> that all have the same common prefix (as in they all start X Y, but the first might be X Y through to X Y C and the second is X Y D … etc) that encapsulate all that accounting data, and then a shard from <some_value_2> through to \xFF.
When we run application load tests we see 2 FDB storage nodes getting absolutely hammered and their durability lag spiking massively, which I think is because it’s the processes hosting those first and last shards (or maybe even just replicas of one of them), so they’re getting a whole load of writes and all the reads.
My understanding is that a shard can’t be split across processes (it has to be split into multiple shards first), and a process can host multiple shards. I’d really love to be able to force certain key prefixes to be in separate shards, even if for the time being they end up hosted on the same process.
Because of the structure and size of our data currently, I think that even if we generate random/interned values for the prefixes we run the risk of all of the ‘core’ data being co-located in the same shard. Is there any way I can prod FDB into where I think data should be sharded beyond just ensuring there is no common prefix? I’d be happy for FDB to add more shards if necessary, but I’d like to be able to make it not aggregate certain keys into the same shard.