I’ve been are seeing a huge impact on Ratekeeper’s limits when the amount of write operations increases (~2x the amount of write transactions), thus affecting our write throughput and increasing latency in the system. We are keeping transaction sizes below 1MB (around 250 records per transaction) and key/value sizes below the Known Limitations suggestions.
As can be seen in the graph below, the number of available transactions/s goes down from ~50M ops/s to around 3M ops/s. We are querying
fdb_cluster_qos_transactions_per_second_limit for available tx/s and
fdb_cluster_qos_released_transactions_per_second for the release tx/s.
The storage queue (
fdb_cluster_processes_roles_storage_query_queue_max) also sees an impact with the increased amount of writes going from ~15 to 500+ (high spikes not a constant change) and we can see the amount of data growing as well going from around 12Mbps to 25Mbps (which is expected as we are nearly doubling the amount of data written)(
And tho the disk operations increase (
fdb_cluster_processes_disk_reads_hz), the disks don’t seem to be anywhere near close to stressed as
fdb_cluster_processes_disk_busy is always bellow 12%.
The RPS on the client app decrease and latency increases as soon as we start writing more data and the ratekeeper limits kick in.
Any pointers on how to configure/increase ratekeeper limits, and/or increase write throughput is greatly appreciated!
Please feel free to ask for whatever information I might have missed and could be useful to addressing this issue.