Is it possible to increase the replication factor in a three_data_hall configuration from 3x to e.g. 6x? I have a read-heavy workload with hotspots that I’d like to temporarily solve with this strategy if possible.
I think three_datacenter will probably accomplish what I want here…
I don’t recall writing an answer about modes and recruitment policies before, so:
The chunk of code you want to look at for “what modes exist and what do they do?” is in ManagementApi.actor.cpp. You’d be looking for something with a storagePolicy of Across(3, data_hall, Across(2, zone_id, One())) (read as: Within each of 3 data_halls, within each of 2 zone_ids, recruit one storage server.)
Which, as you’ve noted, is three_datacenter if you swap data_hall for dcid.
We’ve decided to live life on the edge and use fileconfigure to manually set the replication factor for our three_data_hall configuration from 3 to 6.
echo "{\"storage_replicas\":6}" > /tmp/config.json
fdbcli --exec "fileconfigure /tmp/config.json`
It seems to work fine? (famous last words)
@alexmiller Do you have an opinion on the method I used above? It seems stable on our staging clusters and we’d like to try it in prod if you don’t see any significant downsides. The obvious ones are increased CPU/network resource consumption for the tlogs and 2x the storage use, which are both fine for us.
I think you could have done a simpler fdbcli --exec 'configure storage_replicas := 6'.
As a replication policy, I have no objections to it. I think my only caveat of concern here is that replication policies outside of the standard ones provided via replication modes aren’t really tested. So it probably works?
If you wanted to build some confidence, try editing the three_data_hall mode in simulation to include storage_replicas:=6, and see what happens: SimulatedCluster.actor.cpp - apple/foundationdb - Sourcegraph
They are tested. With some (relatively low probability) the simulator will generate a custom policy. If I understand the code correctly, three data hall with 6 replicas is one of the possible choices. This doesn’t mean it is well enough tested, but it doesn’t seem to be completely untested.
1 Like