Process class and machine sharing deployment questions

[ajbeamon]

One important question here is what the bottleneck was while you were running the tests. If the bottleneck was the storage servers, then a 30% reduction in storage processes may have been more costly than the benefit gained from separating the other roles (though it’s still possible that you pay a latency penalty for not separating the other roles if you are running below saturation). Similarly, increasing the number of logs and proxies on a cluster that was storage bound probably wouldn’t have much benefit, but would have an extra cost in particular when they are being shared with storage servers.

@arob

As one last test, I undid the role changes and tried running with just one process per machine. I was surprised to see that reads were mostly unaffected while writes got cut nearly in half. This is odd, since it would seem to suggest that the cluster was more CPU-bound than I/O-bound, but only on writes. That seems like it’s probably related to the fact that isolating transaction logs from storage servers hurt more than it helped, since you’d only expect that to help if disk I/O was mostly saturated. Perhaps larger value sizes would better exhibit the expected change in performance.

Are these read and write tests run separately or concurrently?

I think there are a few things that could be contributing to this. In addition to the possibility that you are CPU saturated, another is that with half the number of processes, you will end up with a lot more processes sharing roles with storage servers, reducing their effectiveness. In a read-only test, this probably wouldn’t matter as much because reads don’t drive a ton of load to these other processes, but writes would probably have more of an impact.

The other thing is that the SSD storage engine only has a write queue depth of 1 per process. That means that you may not be able to push your disk to maximum throughput with only one process. See SSD engine and IO queue depth for some additional discussion of this.

As a further experiment, you could try a configuration with a higher storage to other process ratio. For example, we’ve run some clusters with a 7:1 storage to transaction disk ratio, with 2 storage servers per disk. Stateless processes don’t use the disk and can safely run on the same machine as storage processes as long as you have sufficient CPU resources.

Also, be a little wary of using vCPU counts to budget for your processes. It’s probably best to have a physical core available for each process, but if you don’t, you’ll want to be careful that you don’t end up saturating the machine CPU, which could have a number of undesirable and potentially confusing effects.