RAM used does not release after high load test

Using FoundationDB Running FoundationDB
1

Hi All,
env : ec2 ubuntu 22.04
fdb: 6.3.24
5 storage nodes: 2 cores, 16 memory, ssd
3 log nodes: 2cores, 8 memory, ssd
3 stateless nodes: 2 cores, 8 memory, ssd

For workload, writes reach to 104k ops/s, then speed deincreased.
图片
As I know, Transactions per Second Limit decreased means the cluster performance is limited in some way by ratekeeper.
图片
With the default knobs, the target queue size is 1.0GB, and ratekeeper will start trying to reduce the transaction rate when a storage server’s queue size reaches 900MB.
图片
Command status details shows RAM used almost 100% on storage node…

10.3.55.185:4500       (  0% cpu;  1% machine; 0.000 Gbps;  1% disk IO;11.7 GB / 12.0 GB RAM  )
10.3.55.187:4500       (  1% cpu;  1% machine; 0.000 Gbps;  1% disk IO;11.8 GB / 12.0 GB RAM  )
10.3.55.188:4500       (  1% cpu;  1% machine; 0.000 Gbps;  1% disk IO;11.9 GB / 12.0 GB RAM  )
10.3.55.189:4500       (  1% cpu;  1% machine; 0.000 Gbps;  1% disk IO;12.0 GB / 12.0 GB RAM  )
10.3.55.192:4500       (  1% cpu;  1% machine; 0.000 Gbps;  1% disk IO;11.9 GB / 12.0 GB RAM  )

A few hours passed, RAM still used almost 100% on storage node, seems like RAM does not release…
Does anyone has this problem?

2

FDB doesn’t give memory back to the OS. So in your case each fdbserver process is probably holding on to some unused memory.

Our custom allocator was implemented that way. I wanted to change this a while ago but failed. If an allocator wants to give back memory to the OS it needs to do some form of book-keeping and the cost of this is high enough to introduce a performance regression.

1 Like
3

Thanks :+1:
This issue has been bothering me for two days :joy:.