I am evaluating FDB for a traffic scenario which requires 1.5 million reads & 1.5 million writes per second (3.0M ops per second in total). The typical value size is 1KB.
For read, I am considering to use reads with versions are set. And read versions are only refreshed may be every 10ms . The business accepts the consequence of this “dirty” read. With this precondition, I think it is possible to achieve 1.5M reads / second.
Writing is what I am not sure. As in the code, I saw the precondition that 1M transactions per second are every where. Like
The writes are almost without reads, aka pure writes. So I can set the read version as well or FDB never issues getReadVersion without read. Then the burden would be on proxy/master/tLog/SS.
Any suggestion on my tough read/write requirement? What should I do to support this.
I think you would have to benchmark a bit and play around with the cluster configurations and evaluate the feasibility.
Fdb will implicitly do batching of transactions per transaction version so the 1million/sec transaction version limit will not be a problem.
There are a lot of factors that will determine whether or not fdb is able to scale for your workload patterns - distribution/concentration of read and write keys (hotspots), key/space collocation of the keys being written/read (ie can a single page satisfy multiple keys), disk io speed, cluster configuration (storage and log nodes) and client concurrency.
I would suggest to start with a medium size cluster with ~15 or so storage nodes and ~5 log nodes and load it up with your workload. Once you see what is becoming a bottleneck (disk io, data distribution, fdb server queues, cpu, or client concurrency), you could post back here the results for more discussions.
100 client random write 10 key: 209,377. CPU usage: 90%,92%,86%, 61%. Average of (Storage, proxy, resolver, log)
100 client random read 10 key: 556,083. CPU usage: 50%,91%,1%,1% Average of (Storage, proxy, resolver,log)
For writing, I think we need to have more storage processes.
The 1M versions per second applies after transaction batching, and versions will progress faster than 1M/s if doing so is required by the workload.
The read side of your workload, I believe FDB should be able to sustain well, but the write side brings me concern that FDB might not be the best fit for your usecase. 1.5M writes/s of 1KB kv size is 1.5GB/s of user writes, so 4.5GB/s replicated before any write amp. That’s… very near the limit of what I’ve heard throughput saturation tests be able to attain, but it has been some time since I’ve checked in with users with heavier write workloads.
7.0 brings some simplifications of the GRV path, which would additionally help your read benchmarks in large clusters. I’d also suggest benchmarking with the new redwood storage engine on 7.0, (via configure ssd-redwood-1-experimental), as the write throughput benefits it brings to storage servers sound quite impactful for your usecase.
client commit msg with data {a=1, b=2} to commit proxy;
commit proxy ask for a tuple of previous commit version and commit version;
commit proxy submit key of data to corresponding resolver to check conflict;
commit proxy group data into two sub msg {a=1} and {b=2} based on tags;
commit proxy commit sub msg with same [previous commit version, commit version] tuple to corresponding tlogs, and commit empty msg to the rest tlogs with above version tuple;
storage server will consume log from tlogs;
For improving the write throughput limit, we can scale out the tlogs/commit proxy/resovler?
