While developing Swift library for FDB I noticed slight memory leakage. At first I though it was my fault (you know, that cheeky Data objects, you always have memory issues with those things).
I continued debugging and ended up with just a C program with looped fdb_transaction_clear function execution (within the same transaction, it doesn’t make any difference). The leak is still here — see line 67.
Probably it’s just me, I rarely use C/C++ and I’m not very competent in it. But I’m quite sure I did everything by the book.
Can someone help?
HMMM. When I made a few more transactional cycles, it didn’t allocate more memory. Instead, it seems like it used previously allocated memory. So I suppose, it might eat up to 4GB RAM (or whatever the memory config directive is).
But I thought this is supposed to be server behaviour, not client. Isn’t it?
Furthermore, after complete connection shutdown (but before exit), it released ~60 megabytes of total ~700 allocated megabytes. This is definitely not OK.
Most of the memory used by fdb (client or server) is allocated via a “fast allocator” (flow/FastAlloc.h) which allocates memory from the OS in large blocks and then keeps it for reuse within the process. This means that from the OS perspective an fdb process’s memory footprint will generally not go down.
After committing your transaction, if you were to repeat the same transaction again you would not see an increase in memory usage as the memory which was freed (internally) after the first attempt would be reused.
Transactions have a per-instance memory pool called an “arena” (flow/Arena.h) in which all memory needed for the transaction’s operations are allocated. This includes temporary copies of user keys, which is what is happening in fdb_transaction_clear(). This arena is freed all at once when the transaction is committed (whether successful or not), reset, or destroyed. The arena’s memory comes from the fast allocator, so once freed it remains in the fast allocator to be reused within the process instead of being given back to the OS.
While it would be possible to optimize fdb_transaction_clear() to require less temporary memory, write transactions in FDB are limited to 10MB of work and 5 seconds of lifetime so it is not expected that a sane write transaction workload would cause a significant amount of memory to be allocated.
2 Likes
Thank you so much for such a detailed answer!
I’m aiming to use my Swift FoundationDB wrapper in long-living server application (on top of Swift-NIO), so stable memory usage and consumption is a matter of great importance for me.
I hope, everything is gonna be OK 