Is it possible to support 1.5M reads & 1.5M writes per second with FDB?

Using FoundationDB
1

I am evaluating FDB for a traffic scenario which requires 1.5 million reads & 1.5 million writes per second (3.0M ops per second in total). The typical value size is 1KB.

For read, I am considering to use reads with versions are set. And read versions are only refreshed may be every 10ms . The business accepts the consequence of this “dirty” read. With this precondition, I think it is possible to achieve 1.5M reads / second.

Writing is what I am not sure. As in the code, I saw the precondition that 1M transactions per second are every where. Like

 init( VERSIONS_PER_SECOND,                                   1e6 );

init (RATEKEEPER_DEFAULT_LIMIT,                              1e6 );

The writes are almost without reads, aka pure writes. So I can set the read version as well or FDB never issues getReadVersion without read. Then the burden would be on proxy/master/tLog/SS.

Any suggestion on my tough read/write requirement? What should I do to support this.

Thanks in advance.

1 Like
2

I think you would have to benchmark a bit and play around with the cluster configurations and evaluate the feasibility.

Fdb will implicitly do batching of transactions per transaction version so the 1million/sec transaction version limit will not be a problem.

There are a lot of factors that will determine whether or not fdb is able to scale for your workload patterns - distribution/concentration of read and write keys (hotspots), key/space collocation of the keys being written/read (ie can a single page satisfy multiple keys), disk io speed, cluster configuration (storage and log nodes) and client concurrency.

I would suggest to start with a medium size cluster with ~15 or so storage nodes and ~5 log nodes and load it up with your workload. Once you see what is becoming a bottleneck (disk io, data distribution, fdb server queues, cpu, or client concurrency), you could post back here the results for more discussions.

1 Like
3

We have benchmarked FDB cluster with below setup.

Version: 6.3.22
3 machine  36 storage 6 proxy 6 resolver 12 logs Triple mode

100 client random write 10 key: 209,377. CPU usage: 90%,92%,86%, 61%.  Average of (Storage, proxy, resolver, log)
100 client random read 10 key: 556,083. CPU usage: 50%,91%,1%,1%	 Average of (Storage, proxy, resolver,log)

For writing, I think we need to have more storage processes.

4

The 1M versions per second applies after transaction batching, and versions will progress faster than 1M/s if doing so is required by the workload.

The read side of your workload, I believe FDB should be able to sustain well, but the write side brings me concern that FDB might not be the best fit for your usecase. 1.5M writes/s of 1KB kv size is 1.5GB/s of user writes, so 4.5GB/s replicated before any write amp. That’s… very near the limit of what I’ve heard throughput saturation tests be able to attain, but it has been some time since I’ve checked in with users with heavier write workloads.

7.0 brings some simplifications of the GRV path, which would additionally help your read benchmarks in large clusters. I’d also suggest benchmarking with the new redwood storage engine on 7.0, (via configure ssd-redwood-1-experimental), as the write throughput benefits it brings to storage servers sound quite impactful for your usecase.