# Scaling issues with FDB for write throughput **URL:** \ **Category:** Running FoundationDB\ **Created:** [September 3, 2020, 4:48am UTC](https://forums.foundationdb.org/t/scaling-issues-with-fdb-for-write-throughput/2327 "2020-09-03T04:48:25Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![dsubramaniam](https://avatars.discourse-cdn.com/v4/letter/d/f17d59/32.png) [@dsubramaniam](https://forums.foundationdb.org/u/dsubramaniam)\ **Post date:** [September 3, 2020, 4:48am UTC](https://forums.foundationdb.org/t/scaling-issues-with-fdb-for-write-throughput/2327/1 "2020-09-03T04:48:25Z")
Hi, We are seeing some concerns when we scale FDB but not able to scale on throughput. We are running a 14 node cluster. 3 i3xl and 11 i34xl. the i3xl are just used to run transaction processes. Currently our workload is write heavy. When we hit 100K RPS in the cluster, the storage lag increases and cluster goes to a unhealthy state. ---
**Author:** ![ajames](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@ajames](https://forums.foundationdb.org/u/ajames)\ **Post date:** [September 7, 2020, 6:36pm UTC](https://forums.foundationdb.org/t/scaling-issues-with-fdb-for-write-throughput/2327/2 "2020-09-07T18:36:20Z")
Adding some more info. Added 2 more storage nodes, now we have a 16 node FDB cluster. 3 i3.xl (Tx nodes) + 13 i3.4xl (SP nodes) FoundationDB 6.2 (v6.2.15) Configuration: Redundancy mode - double Storage engine - ssd-2 Coordinators - 6 Desired Resolvers - 6 Desired Logs - 6 Cluster: FoundationDB processes - 220 Zones - 16 Machines - 16 Memory availability - 7.3 GB per process on machine with least available Retransmissions rate - 3 Hz Fault Tolerance - 1 machine Server time - 09/03/20 04:43:55 Data: Replication health - Healthy (Repartitioning.) Moving data - 0.549 GB Sum of key-value sizes - 574.755 GB Disk space used - 7.380 TB Operating space: Storage server - 1182.3 GB free on most full server Log server - 824.9 GB free on most full server Workload: Read rate - 43579 Hz Write rate - 95600 Hz Transactions started - 277 Hz Transactions committed - 260 Hz Conflict rate - 3 Hz Performance limited by process: Storage server performance (storage queue). Each i3.4xl has 12 storage processes running (6/disk), rest 4 are stateless process (3 nodes have 1 coordinators). All 3 i3.xl nodes run 2 Tx process each (They also have 1 coordinator + 1 stateless process running). Even with 40-50K RPS + 50-80K WPS, we see cluster going into unhealthy state. As per the FDB performance bottleneck reason, it shows - ‘storage\_server\_write\_queue\_size’ Disk busy is under 60% (fdb\_cluster\_processes\_disk\_busy). Disk reads: 3K-6K, Disk writes : 10K-30K. ![Screen Shot 2020-09-07 at 11.22.55 AM](https://global.discourse-cdn.com/foundationdb/original/1X/0664961913242bc5c2e0ab0958e6d9a91dddfd44.jpeg) We do see storage process queue going up to 1GB randomly on different SPs. Does not look it is always hitting same SP or same host. (fdb\_cluster\_processes\_roles\_storage\_input\_bytes\_counter - fdb\_cluster\_processes\_roles\_storage\_durable\_bytes\_counter) ![Screen Shot 2020-09-07 at 11.25.27 AM](https://global.discourse-cdn.com/foundationdb/original/1X/7a5af7fec1ad815da8f4fa1dab5ea59dd250d8c5.jpeg) ![Screen Shot 2020-09-07 at 11.32.56 AM](https://global.discourse-cdn.com/foundationdb/original/1X/0855b677977bc64a2fbccb916d324cfb2333722f.jpeg) See SP CPU hitting 100%. ![Screen Shot 2020-09-07 at 11.28.54 AM](https://global.discourse-cdn.com/foundationdb/original/1X/c212335824da71a9fde16bfc0cc1578f96be71e2.jpeg) Each write transaction has ~200 records, total Tx size is limited below 1MB. We also see fair bit of Tx conflicts. ![Screen Shot 2020-09-07 at 11.15.22 AM](https://global.discourse-cdn.com/foundationdb/original/1X/822617cd2a7ba2434da56b6aa95726204a18f475.jpeg) Initially cluster size was 6 node (3Tx+3SP), we don’t see linear performance as we added 13 more SP nodes. Trying to see where the current bottleneck is. Any help is much appreciated. ---
**Author:** ![ryanworl](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ryanworl/32/440_2.png) [@ryanworl](https://forums.foundationdb.org/u/ryanworl)\ **Post date:** [September 7, 2020, 7:33pm UTC](https://forums.foundationdb.org/t/scaling-issues-with-fdb-for-write-throughput/2327/3 "2020-09-07T19:33:40Z")
Can you describe the workload a bit more? The information given here says each record is around 5KB, which would put your total write bandwidth at 80000\*5000b, or 400MB/s (800MB/s with double replication). It looks like you’ve got 156 storage processes in this cluster. If the workload was perfectly evenly distributed, you’re writing over 5MB/s per storage process, which is outside the range I would consider sustainable on the existing SSD storage engine. If your workload is not evenly distributed, which the most non-overlapping spikes in storage queue size suggest (in my opinion), you may be hot-spotting some storage servers far above that 5MB/s. ---
**Author:** ![ajames](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@ajames](https://forums.foundationdb.org/u/ajames)\ **Post date:** [September 8, 2020, 11:01pm UTC](https://forums.foundationdb.org/t/scaling-issues-with-fdb-for-write-throughput/2327/4 "2020-09-08T23:01:36Z")
Thank You @ryanworl ! Yes, we have 156 storage processes running. Looking at the actual payload (tcpdump), each record varies between 100-400 bytes. The key comprises of many fields and starts with ip of machine sending event. We also have secondary index which is based on current system timestamp. Currently we are trying to see if this is causing hotrange issues. Also, could you please explain more about > you’re writing over 5MB/s per storage process, which is outside the range I would consider sustainable on the existing SSD storage engine"? What is recommended? ---
**Author:** ![ryanworl](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ryanworl/32/440_2.png) [@ryanworl](https://forums.foundationdb.org/u/ryanworl)\ **Post date:** [September 8, 2020, 11:20pm UTC](https://forums.foundationdb.org/t/scaling-issues-with-fdb-for-write-throughput/2327/5 "2020-09-08T23:20:24Z")
Thanks for the update. > (100-400B \* 80,000WPS \* 2x replication) / 156 processes = 100KB/s-400KB/s / process I would say that is sustainable all other things being equal. I would target 1MB/s, so if you’re doing a delete to expire that data after a period of time, your sustainable write rate is approximately 500KB/s because deleting data requires re-writing those pages. Thanks for giving more details about the workload. > We also have secondary index which is based on current system timestamp. Currently we are trying to see if this is causing hotrange issues. I would guess this is the problem. If your index key something like this looks like this: ```auto [TIMESTAMP, IP:PORT, ..., EventID] => '' ``` You’ll be targeting approximately one shard at any given time, and your cluster will spend tons of IO on splitting the shard and copying over and over again. RocksDB is coming as a storage engine option in the future, which may make this pattern sustainable at a higher write rate, but it will never be ideal because shard splits will still take place. I wouldn’t base your decision to use FDB or not for this workload based on the potential that RocksDB is available in the future though. Instead, you can add a prefix to that key, perhaps by hashing the IP address modding it by some constant to make a prefix you can scan if you know the IP address you need to query. The constant would be large enough to create enough shards to sustain your write rate. You’ll still have splits occurring, just at a slower rate. ---
**Author:** ![gaurav](https://avatars.discourse-cdn.com/v4/letter/g/b487fb/32.png) [@gaurav](https://forums.foundationdb.org/u/gaurav)\ **Post date:** [September 14, 2020, 12:59pm UTC](https://forums.foundationdb.org/t/scaling-issues-with-fdb-for-write-throughput/2327/6 "2020-09-14T12:59:06Z")
Please see [this](https://forums.foundationdb.org/t/storage-server-cpu-bottleneck-growing-data-lag/2234/2) thread for some discussion on possible disk saturation and how to reason about it. ---
**Author:** ![ajames](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@ajames](https://forums.foundationdb.org/u/ajames)\ **Post date:** [September 14, 2020, 11:02pm UTC](https://forums.foundationdb.org/t/scaling-issues-with-fdb-for-write-throughput/2327/7 "2020-09-14T23:02:57Z")
Thank You @gaurav for the useful info. However in our case, we found that the secondary index was causing hot ranges. Thank you @ryanworl for pinpointing the issue.