# RAM used does not release after high load test **URL:** \ **Category:** Running FoundationDB\ **Created:** [July 21, 2022, 9:13am UTC](https://forums.foundationdb.org/t/ram-used-does-not-release-after-high-load-test/3462 "2022-07-21T09:13:14Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![CodingSuen](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/codingsuen/32/1367_2.png) [@CodingSuen](https://forums.foundationdb.org/u/CodingSuen)\ **Post date:** [July 21, 2022, 9:13am UTC](https://forums.foundationdb.org/t/ram-used-does-not-release-after-high-load-test/3462/1 "2022-07-21T09:13:14Z")
Hi All, env : ec2 ubuntu 22.04 fdb: 6.3.24 5 storage nodes: 2 cores, 16 memory, ssd 3 log nodes: 2cores, 8 memory, ssd 3 stateless nodes: 2 cores, 8 memory, ssd For workload, writes reach to 104k ops/s, then speed deincreased. ![图片](https://global.discourse-cdn.com/foundationdb/original/2X/a/a758df3e98eaa8893f2c0a2a905dbb54dc4c66c9.png) As I know, Transactions per Second Limit decreased means the cluster performance is limited in some way by ratekeeper. ![图片](https://global.discourse-cdn.com/foundationdb/original/2X/e/e0214b02bd4bc1c96cb135162931cc6acd6138fa.png) With the default knobs, the target queue size is 1.0GB, and ratekeeper will start trying to reduce the transaction rate when a storage server’s queue size reaches 900MB. ![图片](https://global.discourse-cdn.com/foundationdb/original/2X/3/32d5b3e46128110b6bd960d431548ace61321a1a.png) Command status details shows RAM used almost 100% on storage node… ```auto 10.3.55.185:4500 ( 0% cpu; 1% machine; 0.000 Gbps; 1% disk IO;11.7 GB / 12.0 GB RAM ) 10.3.55.187:4500 ( 1% cpu; 1% machine; 0.000 Gbps; 1% disk IO;11.8 GB / 12.0 GB RAM ) 10.3.55.188:4500 ( 1% cpu; 1% machine; 0.000 Gbps; 1% disk IO;11.9 GB / 12.0 GB RAM ) 10.3.55.189:4500 ( 1% cpu; 1% machine; 0.000 Gbps; 1% disk IO;12.0 GB / 12.0 GB RAM ) 10.3.55.192:4500 ( 1% cpu; 1% machine; 0.000 Gbps; 1% disk IO;11.9 GB / 12.0 GB RAM ) ``` A few hours passed, RAM still used almost 100% on storage node, seems like RAM does not release… Does anyone has this problem? ---
**Author:** ![markus.pilman](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/markus.pilman/32/379_2.png) [@markus.pilman](https://forums.foundationdb.org/u/markus.pilman)\ **Post date:** [July 21, 2022, 4:47pm UTC](https://forums.foundationdb.org/t/ram-used-does-not-release-after-high-load-test/3462/2 "2022-07-21T16:47:35Z")
FDB doesn’t give memory back to the OS. So in your case each fdbserver process is probably holding on to some unused memory. Our custom allocator was implemented that way. I wanted to change this a while ago but failed. If an allocator wants to give back memory to the OS it needs to do some form of book-keeping and the cost of this is high enough to introduce a performance regression. ---
**Author:** ![CodingSuen](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/codingsuen/32/1367_2.png) [@CodingSuen](https://forums.foundationdb.org/u/CodingSuen)\ **Post date:** [July 22, 2022, 6:04am UTC](https://forums.foundationdb.org/t/ram-used-does-not-release-after-high-load-test/3462/3 "2022-07-22T06:04:58Z")
Thanks 👍 This issue has been bothering me for two days 😂.