I am running a long-duration testing for my Java application that uses the FDB Java Binding Library to access the FDB backend. My version of the FDB backend is 6.0.15 and the Java Library is with 6.0 correspondingly. I just found that over the 9-hour window, the VM container running my Java application process (only 1 process in one VM container) has the memory residential memory (RSS) increased over 8 GB and thus reaches the container’s memory limit and then gets killed by the VM container manager. Though I am still investigating where the memory leaking comes from, I found that there is a FDB release note, https://apple.github.io/foundationdb/old-release-notes/release-notes-200.html, and there is a section on “Java” in it, which states “Fix: calling getRange on a Transaction could leak memory”. In fact, all of my application’s access to the FDB backend is via “getRange” queries. So I would like to see whether this is really the problem that I run into.
(1) Is this the bug that has been there in FDB 6.0.15 already?
(1) could you describe under what circumstance the memory leak can happen for getRange?
(2) could you point me to the PR that is related to this particular fix, in FDB 6.1*, and in Both Java and in the Client Library (likely), so that I can check whether my code does run into this memory leak problem?
That fix is from a very old release, so I don’t think it would apply to you here. There are two things that immediately come to mind to look for. The first is that there was a fix in 6.0.18 related to a memory leak when a client closes threads (see the release notes and PR). You could try updating your client to 6.0.18 and see if that helps.
The other thing is that the Java bindings require that you explicitly close various objects (most notably, Transaction) in order to free resources. Is it possible that you aren’t closing something? I believe by default, if you fail to close something that should have been closed it will log a warning to stderr.
I just checked the PR and it seems that the changes are all in the C++ side, nothing changed at the Java client. And for the FDB Java Library, I checked the Java version 6.0.15 and 6.0.18 from FDB Git Repo., foundationdb/bindings/java. I diff-ed the two snapshot versions and I found no difference at all.
Regarding Java object closing, I will double check whether transaction.close() gets invoked always.
(1) we are using AsyncIterator to retrieve the getRange results. If some code path does not invoke cancel () of AsyncIterator at the end, will that introduce memory leaking?
(2)also, is there some instrumentation that we can turn on, so that we can see the report on the memory use, for the C/C++ Client Library that gets invoked by the FDB Java Client Library?
Since the Java library loads the native client when it runs, a memory leak there would affect a user of the Java bindings. I would recommend trying to use the 6.0.18 version of the C client (libfdb_c) just to be sure.
AsyncIterator isn’t a closable type (one that descends from AutoCloseable), so you don’t have to do anything special. Some API functions return a CloseableAsyncIterator, and if you use one of them then you would need to close the iterator at the end.
There’s not a lot here, but if you turn on trace logging, then one of the events emitted in the logs is called MemoryMetrics. It has some data on the memory usage of the native client, and if we see the numbers in here growing it would point to the memory leak being native. If you think the numbers look interesting or suspicious, feel free to share some of your events for us to look at.
Also would be useful to get a Java heapdump and run it through a heap analyzer like Eclipse’s Mat. You can gauge whether the memory is from the Java or the native client.