I have a cluster ingesting ~300 MB/s without any reads (yet). The problem I’m having is that data replication is causing large spikes in reads (sometimes in excess of 3 GB/s). That amount of reads ends up causing some of the storage servers to lag and my workload to get throttled. I’m wondering if there is a way to throttle reads from replication a little bit more strongly so that my primary workload doesn’t get disrupted so much.
It’s possible the issue is that I’ve set reply_byte_limit too high. I increased that to 1 MB from 80 kB because it seemed like data distribution was not keeping up. I will play with this number as well.