Most of the memory used by fdb (client or server) is allocated via a “fast allocator” (flow/FastAlloc.h) which allocates memory from the OS in large blocks and then keeps it for reuse within the process. This means that from the OS perspective an fdb process’s memory footprint will generally not go down.
After committing your transaction, if you were to repeat the same transaction again you would not see an increase in memory usage as the memory which was freed (internally) after the first attempt would be reused.
Transactions have a per-instance memory pool called an “arena” (flow/Arena.h) in which all memory needed for the transaction’s operations are allocated. This includes temporary copies of user keys, which is what is happening in fdb_transaction_clear(). This arena is freed all at once when the transaction is committed (whether successful or not), reset, or destroyed. The arena’s memory comes from the fast allocator, so once freed it remains in the fast allocator to be reused within the process instead of being given back to the OS.
While it would be possible to optimize fdb_transaction_clear() to require less temporary memory, write transactions in FDB are limited to 10MB of work and 5 seconds of lifetime so it is not expected that a sane write transaction workload would cause a significant amount of memory to be allocated.