Issues Using Java API on Long Running JVM

Using FoundationDB
1

Hi Guys -

I’m currently working on a POC project that involves writing aggregations from an Apache Flink (https://flink.apache.org/) cluster to FoundationDB. Everything so far as been through local development, and has worked flawlessly.

That being said, now that I’m attempting to launch my job on a long-lived Flink cluster, and I’m running into some issues around the Java API, or from the looks of it, the JNI subsystem specifically.

My job has a FoundationDBStorage object (Scala static class) that calls FDB.selectAPIVersion(520) once.

object FoundationDBStorage {
private final val fdb: FDB = FDB.selectAPIVersion(520)
private final val db: Database = fdb.open("/tmp/fdb.cluster")
…
…

This all works fine the first time after I submit my job to the Flink cluster. However, the second time I go to run the job, I am met with this FDBException.

Caused by: com.apple.foundationdb.FDBException: API version may be set only once
at com.apple.foundationdb.FDB.Select_API_version(Native Method)
at com.apple.foundationdb.FDB.selectAPIVersion(FDB.java:187)
at com.epicgames.streaming.metricsroyale.aggregation.storage.foundationdb.FoundationDBStorage$.(FoundationDBStorage.scala:30)
at com.epicgames.streaming.metricsroyale.aggregation.storage.foundationdb.FoundationDBStorage$.(FoundationDBStorage.scala)

What appears to be happening is, the static class is being cleaned up, but the underlying native code is not. That is, the second run verifies that the singleton is null and
attempts to continue on with calling the native Select_API_version method, which throws our exception.

Is there anything that I can call as a “shutdown” stage to cleanup the underlying C context so that this exception is not thrown on the second attempt?

Thanks,
Ricky
Epic Games, Inc.

2

There isn’t anything in FDB currently that lets you un-set an API version. Flink unloading code and then reloading the same code in the same JVM instance is… an unusual thing to do.

It looks like pretty much everyone trying to use JNI with Flink has these sorts of problems. The solutions that I’ve been able to find elsewhere read like this blog post or this StackOverflow question.

Would it be possible for you to do something similar, where you make a jar that only loads FDB and sets the API version, and then place that in lib/ to be loaded once per Flink instance?

1 Like
3

It’s worth a shot, I’ll give it a go and report back.

Thanks for the help
Ricky

4

Hey @alexmiller -

Your recommendation helped me get things working! I appreciate your help.

Essentially I did the following

  1. Created a sub-module in Gradle
  2. Added FDB Java library as part of the dependencies
  3. Created a static Java class that constructed the FDB API and exposed a simple getFDB() method.
  4. Added this sub-module as part of the parent’s dependency, and removed the FDB dependency from the parent.
  5. Excluded both the FDB Java API, as well as the sub-module from the shadowJar output.
  6. Added the sub-modules shadowJar to the Flink installation’s lib/ directory

Thanks again!
Ricky
Epic Games, Inc

1 Like