SSD engine and IO queue depth

(Ryan Worl) #1

At this timestamp in his talk, @SteavedHams says the SSD engine has only one outstanding IOP at any given time, which I am taking to mean that the queue depth if you deploy one disk per storage process would only ever get to up to one.

This makes me believe we should be more explicit about telling people to deploy more storage processes per disk if their disks reach peak IOPS at high queue depths (16-32 for many SSDs, and… a lot more for NVMe).

The current Performance section of the documentation, for example, uses c3.8xlarge nodes, which have 2 disks each, which means it probably had at least 5 storage processes per disk. The bare metal deployment in the latency section of that doc probably had 9 or more storage processes per disk.

While Redwood will (hopefully) address this given it is an inherited limitation from SQLite, I think adding a description of this issue to the docs would be useful.

I’d be happy to write it up, but I just want to confirm my assessment is correct before I do.

(Steve Atherton) #2

Just to clarify, the SSD engine’s single writer has a queue depth of 1, while there are many concurrent readers (currently hardcoded to 64).

Assuming your assumption is “Some workloads will see higher write throughput by deploying more than 1 storage server processes per disk” then yes it is true. We don’t have a lot of experimental data to suggest how to decide what your StorageServer/disk count should be, however. It’s going to depend on the disk and the workload, and also the total processes per machine should probably not exceed the CPU core count. Probably the best advice is to try increasing it and see if you like the results.

Also, while I haven’t verified your quoted StorageServer/disk counts for our example deployments I can tell you we did not choose those with the intent to suggest a recommended SS/disk density.