# Ratekeeper limits dropping substantially with bulk writes

**URL:** <https://forums.foundationdb.org/t/ratekeeper-limits-dropping-substantially-with-bulk-writes/4336>\
**Category:** Record Layer\
**Tags:** performance\
**Created:** [January 25, 2024, 7:29pm UTC](https://forums.foundationdb.org/t/ratekeeper-limits-dropping-substantially-with-bulk-writes/4336 "2024-01-25T19:29:12Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![patob](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/patob/32/1728_2.png) [@patob](https://forums.foundationdb.org/u/patob)\
**Post date:** [January 25, 2024, 7:29pm UTC](https://forums.foundationdb.org/t/ratekeeper-limits-dropping-substantially-with-bulk-writes/4336/1 "2024-01-25T19:29:12Z")

</div>

Hi all!

I’ve been are seeing a huge impact on Ratekeeper’s limits when the amount of write operations increases (~2x the amount of write transactions), thus affecting our write throughput and increasing latency in the system. We are keeping transaction sizes below 1MB (around 250 records per transaction) and key/value sizes below the Known Limitations suggestions.

As can be seen in the graph below, the number of available transactions/s goes down from ~50M ops/s to around 3M ops/s. We are querying `fdb_cluster_qos_transactions_per_second_limit` for available tx/s and `fdb_cluster_qos_released_transactions_per_second` for the release tx/s.

 ![Ratekeeper limits](https://global.discourse-cdn.com/foundationdb/original/2X/e/e9b1b49bd6e94be9e1088a68fad5e65d72c11d56.png)  
`fdb_cluster_qos_transactions_per_second_limit` and `fdb_cluster_qos_released_transactions_per_second`

The storage queue (`fdb_cluster_processes_roles_storage_query_queue_max`) also sees an impact with the increased amount of writes going from ~15 to 500+ (high spikes not a constant change) and we can see the amount of data growing as well going from around 12Mbps to 25Mbps (which is expected as we are nearly doubling the amount of data written)(`fdb_cluster_processes_network_megabits_received_hz`).

 ![Storage Query Queue](https://global.discourse-cdn.com/foundationdb/original/2X/e/eb63a9c2f8ac04efe64563ceee20274416b27675.jpeg)  
`fdb_cluster_processes_roles_storage_query_queue_max`

 ![Megabits Received](https://global.discourse-cdn.com/foundationdb/original/2X/c/c76869308a26aa7d7c71963b54a1614df551df98.jpeg)  
`fdb_cluster_processes_network_megabits_received_hz`

And tho the disk operations increase (`fdb_cluster_processes_disk_writes_hz` and `fdb_cluster_processes_disk_reads_hz`), the disks don’t seem to be anywhere near close to stressed as `fdb_cluster_processes_disk_busy` is always bellow 12%.

 ![Disk Busy](https://global.discourse-cdn.com/foundationdb/original/2X/3/3307ea6254d8a8dbe9d66bb1fe95ab1649615a11.jpeg)  
`fdb_cluster_processes_disk_busy`

 ![Disk Writes](https://global.discourse-cdn.com/foundationdb/original/2X/2/2c632548b4f1c99ef543a0d0b6b10767028be41c.jpeg)  
`fdb_cluster_processes_disk_writes_hz`

 ![DIsk Reads](https://global.discourse-cdn.com/foundationdb/original/2X/4/47b2ffd45ea46bc8bb30d05dfb304c916d09dbce.jpeg)  
`fdb_cluster_processes_disk_reads_hz`

The RPS on the client app decrease and latency increases as soon as we start writing more data and the ratekeeper limits kick in.

 ![RPS](https://global.discourse-cdn.com/foundationdb/original/2X/4/4a911a512a036b7f89cf25ab582d6b070592c275.jpeg)

 ![Latency](https://global.discourse-cdn.com/foundationdb/original/2X/f/f3ae49bc34063373854f5bbaed6223b14c765e28.png)

Any pointers on how to configure/increase ratekeeper limits, and/or increase write throughput is greatly appreciated!

Please feel free to ask for whatever information I might have missed and could be useful to addressing this issue.

Best regards!

---

<div class="post-metadata">

**Author:** ![jzhou](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/jzhou/32/445_2.png) [@jzhou](https://forums.foundationdb.org/u/jzhou)\
**Post date:** [January 27, 2024, 6:30am UTC](https://forums.foundationdb.org/t/ratekeeper-limits-dropping-substantially-with-bulk-writes/4336/2 "2024-01-27T06:30:17Z")

</div>

Seems ratekeeper was throttling, so you should start looking at `RkUpdate` trace events in its log, where each entry has something like `ID="c78b41de2e8e417d" TPSLimit="45698.1" Reason="2"`. The reason is the one we are looking for, corresponds to values defined [here](https://github.com/apple/foundationdb/blob/393fef71fa26e7b4b1db7b72bdf6128e18ae4b2c/fdbserver/Ratekeeper.actor.cpp#L29).

---

<div class="post-metadata">

**Author:** ![SteavedHams](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/steavedhams/32/18_2.png) [@SteavedHams](https://forums.foundationdb.org/u/SteavedHams)\
**Post date:** [January 29, 2024, 8:57pm UTC](https://forums.foundationdb.org/t/ratekeeper-limits-dropping-substantially-with-bulk-writes/4336/3 "2024-01-29T20:57:42Z")

</div>

Questions

- Which FDB version are you using?
- Which storage engine?
- What is the key locality of your writes? Are they random/scattered or are they mostly sequential/adjacent?
- What kind of disks are you using?
- Are you placing \>1 storage server on a disk volume?
- Does each storage server have 1 physical CPU core to use?

---

<div class="post-metadata">

**Author:** ![patob](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/patob/32/1728_2.png) [@patob](https://forums.foundationdb.org/u/patob)\
**Post date:** [February 2, 2024, 6:55pm UTC](https://forums.foundationdb.org/t/ratekeeper-limits-dropping-substantially-with-bulk-writes/4336/4 "2024-02-02T18:55:30Z")

</div>

Hi Steve, thank you for getting back on this, here’s the data you requested.

- Which FDB version are you using?

```auto
FoundationDB 6.2 (v6.2.15)
source version 20566f2ff06a7e822b30e8cfd91090fbd863a393
protocol fdb00b062010001

```

- Which storage engine?

```auto
onfiguration:
  Redundancy mode - double
  Storage engine - ssd-2
  Coordinators - 6
  Exclusions - 31 (type `exclude' for details)
  Desired Proxies - 12
  Desired Resolvers - 8
  Desired Logs - 12

Cluster:
  FoundationDB processes - 308 (less 2 excluded; 0 with errors)
  Zones - 24
  Machines - 24
  Memory availability - 5.5 GB per process on machine with least available
  Retransmissions rate - 1 Hz
  Fault Tolerance - 1 machine
  Server time - 02/01/24 17:33:58

Data:
  Replication health - Healthy
  Moving data - 0.000 GB
  Sum of key-value sizes - 4.770 TB
  Disk space used - 12.933 TB

Operating space:
  Storage server - 1338.7 GB free on most full server
  Log server - 826.1 GB free on most full server

Workload:
  Read rate - 127406 Hz
  Write rate - 6424 Hz
  Transactions started - 100350 Hz
  Transactions committed - 233 Hz
  Conflict rate - 9 Hz

```

- What is the key locality of your writes? Are they random/scattered or are they mostly sequential/adjacent?  
Mostly random/scattered as they are UUIDs + app id combination

- What kind of disks are you using?  
NVME (instance store)

- Are you placing \>1 storage server on a disk volume?  
Yes

- Does each storage server have 1 physical CPU core to use?  
Yes, we don’t see anywhere near full CPU/disk usage as shown in the original post.

---

<div class="post-metadata">

**Author:** ![SteavedHams](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/steavedhams/32/18_2.png) [@SteavedHams](https://forums.foundationdb.org/u/SteavedHams)\
**Post date:** [March 12, 2024, 8:04am UTC](https://forums.foundationdb.org/t/ratekeeper-limits-dropping-substantially-with-bulk-writes/4336/5 "2024-03-12T08:04:23Z")

</div>

Random writes are a worst-case IO pattern and the `ssd-2` engine does not handle it well because it will incur many serial disk latencies on the write path, so you could be IO bound on the Storage Servers which causes their Storage Queue to build up and then Ratekeeper to limit. As @jzhou said you should look at the `RkUpdate` events to see what the limiting reason is.