Our binaries have distributed performance tests built into them.
First start additional fdbserver processes with the “–class test” role. These processes will act as clients the run the benchmarking workload.
Next write a test file that describes a workload. Generally you will want to run a “ReadWrite” test, the following file is a good starting point that you can change depending on what kind of test you want to run:
https://github.com/apple/foundationdb/blob/master/tests/RandomReadWrite.txt
Most of the parameters are self explanatory. The alpha parameter specifies the percentage chance of running a “B” transaction. For example alpha=0.0 would only run “A” transactions, and alpha=1.0 would only run “B” transactions. Generally you will always want at least one read in each transaction. The file I linked is setup to have 90% of transactions be read only, and 10% of transactions write 10 random key value pairs and also read one key.
Once you have a test file, run fdbserver with the following parameters: -r multitest -f [TESTFILE] --num_testers [NUMBER OF TEST CLASS FDBSERVERS]
Please note that this benchmark will load data into the cluster, and does not clean up after the test is over. Therefore, you should not run it on clusters that already have data in them.
After the test finishes, the multitest process will aggregate results from all for the test processes and display them from you. The first then to check is if the cluster achieved the targeted transaction per second.
If it was not able to achieve it, the interested metrics are how many transactions per second the test achieved, and possibly the read and commit latencies. In saturation we attempt to push all of the latency into getting read versions, so that number should be really high, along with the total latency.
If it was able to achieve it, then all the latency numbers will be interesting. The size of the data set will determine how many reads needed to read from disk, so be aware of that detail when looking at read latencies.
Getting good performance from FoundationDB does not come out of the box. There are a number of things that will dramatically reduce your performance if you just start up a ton of fdbserver processes with no additional work. See https://apple.github.io/foundationdb/configuration.html#guidelines-for-setting-process-class for details.
In addition to the advice there, make sure to only run one transaction class process per disk. It is okay to run multiple storage class processes per disk, but do not run more fdbserver processes than physical cores. All the machines for the same process class should be homogenous. If you ran half your disks with one process and the other half with two processes things will not go well. If you included one storage server running on a really old disk the cluster would perform much worse than if you did not add that process to the cluster.
Before running a large FoundationDB cluster I would strongly recommend running benchmarks while changing the number of proxies, resolvers, and transaction logs to get a feel for the approximate ratio you should run. More is not always better, especially for proxies and resolvers, so I would hesitate to run too many of them, but check for yourself to see what gets good performance. Always have spares for stateless and transaction processes or performance during/after a failure will suffer dramatically.