Rebalancing not happening on read hot shards in 7.4.43?

Using FoundationDB
1

Over the course of the week-end one of our cluster is starting to see a lot more load and some of the storage servers are using ~75% of CPU on average when some others are mostly idle.

I already mentioned this a bit in this: How accurate is readOpsPerSec when readSample is enabled?

But this post is focused on the apparent lack of rebalancing

Using the hotrange command I can see that we have 2 ss with very different read profiles:

fdb> hotrange 10.0.49.70:4505 bytes "" "\xff"  1

[
    {
        "begin" : "",
        "bytes" : 647490485,
        "end" : "\u00FF\u0002/<snip>",
        "readBytesPerSec" : 7500,
        "readOpsPerSec" : 6124100
    }
]
fdb> hotrange 10.0.235.187:4503 bytes "" "\xff"  1

[
    {
        "begin" : "",
        "bytes" : 946779588,
        "end" : "\u00FF\u0002/<snip>",
        "readBytesPerSec" : 453333.33333333331,
        "readOpsPerSec" : 4803600
    }
]

I would have expected fdb to actually rebalance the read hot shards to the one that are mostly doing nothing but it’s not happening.

One thing I suspect is happening is because all datadistribution is enabled that that the reblance_disk is taking precedence on the rebalance_read.
But so far I don’t have data to back this hypothesis.

Any advices or clues ?

2

Replying to myself after reading a bit of the code.

  1. you need to check that --knob_read_sampling_enabled is set to true on both the storage and the stateless nodes especially on the node data as the datadistributor role
  2. If you don’t see "DDRebalancePaused" in the logs of the datadistributor when you disable rebalance_read (ie. datadistribution disable rebalance_disk) then it’s a sign that rebalance is not happening on reads.
3

Hey Matthieu :waving_hand:

Indeed read_sampling_enabled knob is required to have read aware distribution enabled: doc link
Furthermore, keep in mind that this feature will only redistribute shards to spread the load on storages. It doesn’t split shard ! It means that you can still have hot shard that consume all the cpu of a storage.

4

Hi Anleg,

So it would help only if you have multiple shards that are hot on the same storage server right ?

And is there any plan to split shards so that when you have a one hot read shard it is split in smaller ones ?