Distributed transaction performance test

Using FoundationDB Running FoundationDB
1

We’re trying to test the performance of distributed transaction on FoundationDB.
We’re running the test with 6 cloud servers on gcp, with 8 cores each(c4-standard-8), and a disk with 3000IOPS and 200mb throughput.

Since we’re not really familiar with fdb, we’re not sure if the numbers we got a reasonable. The number seems pretty low so we were wondering if we have deployed the cluster in the right way.

We have deployed 6*8=48 processes, and configured 6 log server, 6 resolver and 4 grv proxy processes. Redundancy mode is set to triple. Other processes are auto configed by the cluster.

We have a client that sends transaction with 4 updates, and thebest tps we got is about 8000-10000. This seems pretty low. The cpu usage on servers were pretty low, about 10% of the cpus on servers. The IO of ssd is pretty much saturated, showing 3000IOPS.

Is this number reasonable? Or is there anything we can do to improve the performance? Thanks

2

There are some good suggestions in forum posts, e.g., Why many clientThreads will cause fdbserver(stateless & grv_proxy) CPU too high? - #4 by Semisol

Is your workload pure writes? FDB is designed for read-mostly workload. It appears IOPS are all used, which is likely the reason performance is limited. For better write performance, consider using local SSDs for log processes.

3

@liunyl Without needing any more details about your write pattern, I see a few major issues just based on the host configuration and process counts you shared.

  1. It appears that you are running somewhere between 3 and 6 storage servers on each host which all share its fairly slow 3k IOPS disk. Any of FDB’s storage engines will easily max out 3k IOPS for a mostly write workload so running multiple storage servers on those disks is more likely to cause saturation problems than performance benefits.
  2. You are running the same number of logs as you have disks which means every disk has a log server on it so your storage servers are sharing a disk with a log in addition to other storage servers as mentioned above. Log servers and storage servers should not share a disk as their write/fsync patterns and latency requirements are very different and will interfere with each other.
  3. Once you address the above two points, your ratio of storage servers to log servers is too low. A log server can provide mutation streams to many storage servers so the ratio of storage:log servers should typically be at least 10:1.
4

I agree with the recommendation of using local SSDs for logs. They offer much lower latency for fsync (which is important as this will happen hundreds to thousands of times per second) and better IO/$.

Another thing is colocating SS with TLog will cause a bunch of issues. TLogs do a lot of small flushes, while SSes do less but bigger flushes when making data durable.
SSDs are not going to perform great on mixed worklaods.

There also appears to be too many SS on one disk, I assume 6 from your numbers + 1 TLog + 1 stateless.
(3 SS/disk is what I found reasonable with bare metal NVMe disks pushing over 30K IOPS or 2GB/s)

I recommend you try the following configuration and tune it to your needs:

  • For each host, use the configuration with the least amount of cores while keeping at least 1 core/process.
  • Have 1 SS process per disk. If you do not see IOPS saturation even at maximum load, increase it.
  • Have 1 stateless process per host. These handle the CC, master, and GRV proxy tasks. If you have excess cores, put more stateless processes.
  • Each server should ideally do one task, logs or the rest of the transaction system. Put 1 commitproxy process or 1 TLog/disk on each host.
  • Have dedicated local disks for logs. Putting SS/TLog on the same host as long as they do not share a disk is mostly fine. (check bandwidth utilization)
  • Configure 3 log processes to start, increase/decrease as necessary. Always have 1-2 more processes available than configured to be able to handle failures.
  • Configure 1 resolver. Do not add more unless you are absolutely sure it is the resolver. Many issues can also be alleviated by the client, such as optimizing conflict ranges.
    This should be just fine placed on stateless processes on a small cluster.

An example could look like this with 4-core hosts:

  • 4x SS TLog Stateless Stateless (1 SS disk, 1 local TLog disk)
  • 4x SS SS Stateless CommitProxy (2 SS disks)

This also would come at a lower amount of vCPUs needed compared to your current setup.

Other suggestions:

  • The Redwood engine offers way better performance. Use it if you are not already.
  • Depending on your risk profile, consider using local SSDs for SS as well. The cost savings are worth it, even with increased replication factor.