In my data model, there are a couple of “hot” storage servers that are getting a lot of read requests, causing their CPU usage to get pegged at 100% for long durations. This, in turn, causes the storage queues on these servers to grow (due to reads having higher priority than writes), leading to the cluster reporting
While I am working independently on changing the access patterns to resolve these hotspots, I wanted to understand a couple of things about this:
- What is the effect of this cluster state? Will it cause the rate-keeper to slow down the rate at which transactions can be created (in other words: the rate at which read_version can be obtained)?
- Are there any knobs to increase the priority of writes so that cluster can keep draining the queue at a faster rate (probably at the expense of slowing down reads/transactions uniformly over time)? Are there any other potential downsides or risks to playing around with read vs write request priority knobs?