How to setup multi-version clients with Java?

Using FoundationDB
1

So I was reading https://apple.github.io/foundationdb/api-general.html#multi-version-client-api and have a few questions about how to do this.

Currently we are using the java bindings and are doing a System.load() on the fdb shared object library before opening a database instance with a cluster file.

We are planning to upgrade to 6.1.8 from 6.0.15 and are planning to make the clients be able to handle both versions so that the database can be upgraded similar to what I read here:
Upgrading FoundationDB

I tried a naive approach of loading both library versions with the following:

import com.apple.foundationdb.Database;
import com.apple.foundationdb.FDB;
import com.apple.foundationdb.tuple.Tuple;

import java.io.File;


public class Main {
    public static void main(String[] args) {

        File fdbLib6015 = new File("resources/libfdb_c_6.0.15.so");
        File fdbLib618 = new File("resources/libfdb_c_6.1.8.so");

        File clusterFile6015 = new File("resources/fdb_6015.cluster");
        File clusterFile618 = new File("resources/fdb_618.cluster");

        System.load(fdbLib6015.getAbsolutePath());
        System.load(fdbLib618.getAbsolutePath());

        FDB fdb = FDB.selectAPIVersion(600);
    
        File[] fdb_cluster_files = {clusterFile6015, clusterFile618};
        for (int i = 0; i < fdb_cluster_files.length; i++) {
            System.out.println("Trying to connect with cluster file: " + fdb_cluster_files[i].toString());
            try(Database db = fdb.open(fdb_cluster_files[i].getAbsolutePath())) {
                // Run an operation on the database
                db.run(tr -> {
                    tr.set(Tuple.from("hello").pack(), Tuple.from("world").pack());
                    return null;
                });

                // Get the value of 'hello' from the database
                String hello = db.run(tr -> {
                    byte[] result = tr.get(Tuple.from("hello").pack()).join();
                    return Tuple.fromBytes(result).getString(0);
                });
                System.out.println("Hello " + hello);
            }
            System.out.println("Success!\n");
        }
    }

    private Main() {}
}

But this hangs when trying to connect to the 6.1.8 cluster due to loading the 6.0.15 library first. I can reverse the order of the System.load() and the 6.1.8 will connect whereas the 6.0.15 will hang forever.

So I know that approach will not work. I then looked further through the javadoc https://apple.github.io/foundationdb/javadoc/index.html but did not see anything relevant to the EXTERNAL_CLIENT_DIRECTORY / EXTERNAL_CLIENT_LIBRARY network options that the docs say need to be set.

Looking here I can see that it definitely is a network option but I guess my question is how would I set this with the Java API bindings properly to be able to connect to both a 6.1.8 and a 6.0.15 cluster? And then with this set up properly, would that client be able to handle an upgrade of the 6.0.15 cluster going to 6.1.8 with minimal downtime?

Or am I approaching this all wrong?

1 Like
2

You will need to set one of those two EXTERNAL_CLIENT_... options in order to use the multi-version client. It can be done by calling the network option in Java:

FDB fdb = FDB.selectAPIVersion(600);
fdb.options().setExternalClientDirectory(...);

Or you can control it via environment variable:

FDB_NETWORK_OPTION_EXTERNAL_CLIENT_DIRECTORY=...

To set your client up to connect with both versions, you would need to use one of the libfdb_c.so files as your “primary” library that gets loaded normally. Typically, this would be done with the new library so that you have access to all its capabilities when the cluster is upgraded, though you could equally do it with the old one as the primary. After that, you would place your other version somewhere with a unique name and use one of the two external client options to tell the primary client where to find it.

Yes, there should be very little downtime during an upgrade if this is configured correctly. You’d basically just have to wait how long it takes the cluster to recover (usually in the neighborhood of a few seconds for the ssd storage engine) and then restart all of your in-flight transactions (which should happen automatically if you are using the built-in retry loops).

3

Can’t believe I missed that! Thanks for pointing it out and it works like a charm.

4

Would there be any concern/issue if I did:

fdb.options().setExternalClientDirectory("resources");

Which also contains the “resources/libfdb_c_6.0.15.so” that I am having the system load as the “primary” for fdb? Essentially duplicating this library as an external one?

5

The client will always favor the primary if there are two choices that have the same protocol version. Other than that, the costs of it should be pretty low.

You don’t want to have two different compatible external clients, though, because the choice between them will be arbitrary.