I have a Java application running on Ubuntu which is configured to connect to a primary FDB cluster (cluster A) and a secondary FDB cluster (cluster B). It attempts to connect to A on start-up and read the status JSON key to determine if the database is locked. If it is locked, it attempts to connect to B and read the status key JSON to determine if it is locked. This dance is to facilitate DR failover, and, any time a “database locked” error is encountered, this process starts again. An external process manages the failover, but that’s not really relevant to this issue.
I am attempting to upgrade the client version as given in the title. We are first upgrading the clients and setting API version 620 and we will deal with upgrading the servers later.
We receive this error after upgrading the clients.
Caused by: java.lang.RuntimeException: Failed to read cluster.database_locked or cluster.database_lock_state.locked from status={client={cluster_file={path=/tmp/foundation-db-cluster-files15548909453389961686/CLUSTER_NAME, up_to_date=true}, coordinators={coordinators=[{address=X.Y.Z.64:4500, reachable=false}, {address=X.Y.Z.84:4500, reachable=false}, {address=X.Y.Z.125:4500, reachable=false}, {address=X.Y.Z.126:4500, reachable=false}, {address=X.Y.Z.186:4500, reachable=false}], quorum_reachable=false}, database_status={available=false, healthy=false}, messages=[{description=Unable to reach a quorum of coordinators., name=quorum_not_reachable}], timestamp=1632757513}, cluster={layers={_valid=false}}}
The Failed to read cluster.database_locked or cluster.database_lock_state.locked is our error message, not something we’ve received directly from the database. The remainder is the original Exception from the database.
I’ve verified all the coordinators are available now and were available at that time. The cluster file has the same contents between both client versions.
CLUSTER_NAME:RANDOM_STRING_CONSTANT@X.Y.Z.64:4500,X.Y.Z.84:4500,X.Y.Z.125:4500,X.Y.Z.126:4500,X.Y.Z.186:4500
As you can see, the IP addresses do match as X.Y.Z is a constant. The coordinators are definitely working, as the 6.2.20 clients still operate correctly.
Am I missing anything obvious here?