# FoundationdDB Cluster Performance Issues **URL:** \ **Category:** Using FoundationDB\ **Tags:** performance, operator\ **Created:** [May 6, 2020, 4:01pm UTC](https://forums.foundationdb.org/t/foundationddb-cluster-performance-issues/2105 "2020-05-06T16:01:40Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![v-pap](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/v-pap/32/936_2.png) [@v-pap](https://forums.foundationdb.org/u/v-pap)\ **Post date:** [May 6, 2020, 4:01pm UTC](https://forums.foundationdb.org/t/foundationddb-cluster-performance-issues/2105/1 "2020-05-06T16:01:40Z")
Currently, I have deployed a FoundationDB cluster with the help of the kubernetes operator. I was testing the cluster to see how far it could go, by writing timeseries data. After increasing the number of writes, I hit a plateau in which the number of writes won’t increase. ```auto fdb> status details Using cluster file `/var/dynamic-conf/fdb.cluster'. Unable to start batch priority transaction after 5 seconds. Configuration: Redundancy mode - double Storage engine - ssd-2 Coordinators - 3 Desired Proxies - 3 Desired Resolvers - 1 Desired Logs - 3 Cluster: FoundationDB processes - 12 Zones - 12 Machines - 12 Memory availability - 10.6 GB per process on machine with least available Retransmissions rate - 8 Hz Fault Tolerance - 1 machine Server time - 05/06/20 15:40:39 Data: Replication health - Healthy (Repartitioning) Moving data - 0.100 GB Sum of key-value sizes - 23.638 GB Disk space used - 59.957 GB Operating space: Storage server - 345.3 GB free on most full server Log server - 345.9 GB free on most full server Workload: Read rate - 180 Hz Write rate - 22760 Hz Transactions started - 57 Hz Transactions committed - 26 Hz Conflict rate - 1 Hz Performance limited by process: Storage server performance (storage queue). Most limiting process: 10.112.15.50:4501 Backup and DR: Running backups - 0 Running DRs - 0 Process performance details: 10.112.14.23:4501 ( 2% cpu; 6% machine; 0.009 Gbps; 15% disk IO; 1.8 GB / 11.9 GB RAM ) 10.112.15.50:4501 ( 25% cpu; 37% machine; 0.005 Gbps; 5% disk IO; 3.8 GB / 11.1 GB RAM ) 10.112.17.16:4501 ( 18% cpu; 26% machine; 0.004 Gbps; 11% disk IO; 3.7 GB / 11.4 GB RAM ) 10.112.27.11:4501 ( 25% cpu; 25% machine; 0.005 Gbps; 7% disk IO; 3.0 GB / 11.0 GB RAM ) 10.112.29.14:4501 ( 2% cpu; 13% machine; 0.009 Gbps; 15% disk IO; 1.9 GB / 11.7 GB RAM ) 10.112.30.11:4501 ( 2% cpu; 5% machine; 0.010 Gbps; 16% disk IO; 1.9 GB / 11.4 GB RAM ) 10.112.31.18:4501 ( 25% cpu; 34% machine; 0.005 Gbps; 12% disk IO; 3.3 GB / 11.2 GB RAM ) 10.112.32.8:4501 ( 8% cpu; 12% machine; 0.046 Gbps; 0% disk IO; 0.3 GB / 11.3 GB RAM ) 10.112.32.35:4501 ( 5% cpu; 12% machine; 0.020 Gbps; 0% disk IO; 0.3 GB / 11.4 GB RAM ) 10.112.33.13:4501 ( 25% cpu; 19% machine; 0.009 Gbps; 9% disk IO; 1.2 GB / 11.5 GB RAM ) 10.112.33.54:4501 ( 2% cpu; 19% machine; 0.000 Gbps; 0% disk IO; 0.3 GB / 10.6 GB RAM ) 10.112.34.12:4501 ( 25% cpu; 18% machine; 0.007 Gbps; 13% disk IO; 3.1 GB / 11.4 GB RAM ) Coordination servers: 10.112.27.11:4501 (reachable) 10.112.31.18:4501 (reachable) 10.112.33.13:4501 (reachable) Client time: 05/06/20 15:40:33 fdb> setclass There are currently 12 processes in the database: 10.112.14.23:4501: log (command_line) 10.112.15.50:4501: storage (command_line) 10.112.17.16:4501: storage (command_line) 10.112.27.11:4501: storage (command_line) 10.112.29.14:4501: log (command_line) 10.112.30.11:4501: log (command_line) 10.112.31.18:4501: storage (command_line) 10.112.32.8:4501: proxy (set_class) 10.112.32.35:4501: stateless (command_line) 10.112.33.13:4501: storage (command_line) 10.112.33.54:4501: stateless (command_line) 10.112.34.12:4501: storage (set_class) fdb> ``` From another thread in the forums ([Storage queue limiting performance when initially loading data](https://forums.foundationdb.org/t/storage-queue-limiting-performance-when-initially-loading-data/1135)), there was a suggestion to increase the number of storage processes per disk. At the moment I have tried to do that through the kubernetes operator by changing the spec (increasing the storage process count), but instead it tried to create a new pod instead (with a PVC). Is there is an easy way to achieve that? Or I should try something else instead? ---
**Author:** ![john\_brownlee](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/john_brownlee/32/22_2.png) [@john\_brownlee](https://forums.foundationdb.org/u/john_brownlee)\ **Post date:** [May 7, 2020, 4:37pm UTC](https://forums.foundationdb.org/t/foundationddb-cluster-performance-issues/2105/2 "2020-05-07T16:37:37Z")
No, we don’t currently have a way to run multiple storage processes per disk through the operator. ---
**Author:** ![john\_brownlee](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/john_brownlee/32/22_2.png) [@john\_brownlee](https://forums.foundationdb.org/u/john_brownlee)\ **Post date:** [May 7, 2020, 5:42pm UTC](https://forums.foundationdb.org/t/foundationddb-cluster-performance-issues/2105/3 "2020-05-07T17:42:51Z")
For what it’s worth, I’d be surprised if you could max out writes with a cluster this small. What write rate are you getting? It might be worth adding more logs and proxies, and maybe resolvers as well. ---
**Author:** ![v-pap](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/v-pap/32/936_2.png) [@v-pap](https://forums.foundationdb.org/u/v-pap)\ **Post date:** [May 7, 2020, 7:23pm UTC](https://forums.foundationdb.org/t/foundationddb-cluster-performance-issues/2105/4 "2020-05-07T19:23:34Z")
Thank you for your response, according to the stats the write rate was around 22k - 28k Hz. One thing that I observed was, that this 25% CPU load on the storage processes was partly to blame on the pod blueprint in which the limit was set to 250m. I managed to increase that to 500m and the CPU usage increased to 50% as well (for the storage processes). At the moment, I haven’t tried to increase it any further, but if my understanding is correct the usage would not increase beyond the one CPU core, is that correct? ---
**Author:** ![alexmiller](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/alexmiller/32/326_2.png) [@alexmiller](https://forums.foundationdb.org/u/alexmiller)\ **Post date:** [May 7, 2020, 7:53pm UTC](https://forums.foundationdb.org/t/foundationddb-cluster-performance-issues/2105/5 "2020-05-07T19:53:52Z")
> [@v-pap](#): > > 10.112.32.8:4501: proxy (set\_class) I believe that with the current recruitment logic, having one process with `proxy` class will mean that we won’t consider stateless class processes for proxies, so you’ll likely get 1 proxy instead of 3. > [@v-pap](#): > > the usage would not increase beyond the one CPU core, is that correct? Correct. Storage servers, like the rest of FDB, are a single threaded process, and thus won’t be able to take advantage of having \>1 cores available to them. ---
**Author:** ![john\_brownlee](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/john_brownlee/32/22_2.png) [@john\_brownlee](https://forums.foundationdb.org/u/john_brownlee)\ **Post date:** [May 7, 2020, 10:18pm UTC](https://forums.foundationdb.org/t/foundationddb-cluster-performance-issues/2105/6 "2020-05-07T22:18:33Z")
The usage wouldn’t increase beyond one physical CPU core, but that would correspond to `2000m` of CPU in the Kubernetes resource requirements, so I think it’s worth testing it at that level as well. ---
**Author:** ![v-pap](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/v-pap/32/936_2.png) [@v-pap](https://forums.foundationdb.org/u/v-pap)\ **Post date:** [May 11, 2020, 6:14pm UTC](https://forums.foundationdb.org/t/foundationddb-cluster-performance-issues/2105/7 "2020-05-11T18:14:16Z")
After following your advice about increasing the CPU requests/limit close to `2000m`, the write throughput increased considerably even though the load still seems to be CPU bound rather than disk related. As an experiment, I kept increasing the writes to the cluster to find out after how many writes I would need to add more storage processes. After reaching `40k Hz` writes I added more storage processes (I started a new cluster with 3 storage processes and 4 proxies and then I added 3 more storage processes). I left it for 2 days to repartition the data, but the original 3 storage processes had a storage queue of around 0.9 GB to 1.4 GB, and the new storage processes had at most 100 MB of data in their storage queue. What do you think is the problem here? ---
**Author:** ![rbtcollins](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/rbtcollins/32/941_2.png) [@rbtcollins](https://forums.foundationdb.org/u/rbtcollins)\ **Post date:** [May 22, 2020, 2:39am UTC](https://forums.foundationdb.org/t/foundationddb-cluster-performance-issues/2105/8 "2020-05-22T02:39:53Z")
Is there any in-principal objection to that feature (multiple fdbservers per disk/pod) being added? If not, whats the way forward? Start a design discussion on GitHub? (I have a different cluster POC I’m working with, but this was a question I had :)) ---
**Author:** ![john\_brownlee](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/john_brownlee/32/22_2.png) [@john\_brownlee](https://forums.foundationdb.org/u/john_brownlee)\ **Post date:** [May 22, 2020, 3:50pm UTC](https://forums.foundationdb.org/t/foundationddb-cluster-performance-issues/2105/9 "2020-05-22T15:50:16Z")
I don’t object to it in principle, and I haven’t heard from anyone who does. I suspect that it’s going to be fairly complex to implement, because of the secondary effects on port management, exclusions, bounces, etc… If you’d like to champion this feature, you can file an issue on GitHub and we can start discussing the design there. ---
**Author:** ![rbtcollins](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/rbtcollins/32/941_2.png) [@rbtcollins](https://forums.foundationdb.org/u/rbtcollins)\ **Post date:** [May 26, 2020, 3:49am UTC](https://forums.foundationdb.org/t/foundationddb-cluster-performance-issues/2105/10 "2020-05-26T03:49:31Z")
Thank you. I have a few things to investigate immediately but I will likely come back to that on GitHub in the near term. ---
**Author:** ![rbtcollins](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/rbtcollins/32/941_2.png) [@rbtcollins](https://forums.foundationdb.org/u/rbtcollins)\ **Post date:** [October 21, 2020, 11:55am UTC](https://forums.foundationdb.org/t/foundationddb-cluster-performance-issues/2105/11 "2020-10-21T11:55:00Z")
I was sure I saw a spec /proposal from you on this, but I can’t find it now :(. ---
**Author:** ![john\_brownlee](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/john_brownlee/32/22_2.png) [@john\_brownlee](https://forums.foundationdb.org/u/john_brownlee)\ **Post date:** [October 21, 2020, 3:20pm UTC](https://forums.foundationdb.org/t/foundationddb-cluster-performance-issues/2105/12 "2020-10-21T15:20:11Z")
Is this what you were thinking of? [https://github.com/FoundationDB/fdb-kubernetes-operator/wiki/Multiple-Storage-Servers-per-Disk-Design](https://github.com/FoundationDB/fdb-kubernetes-operator/wiki/Multiple-Storage-Servers-per-Disk-Design)