The cluster file stored at /etc/foundationdb/fdb.cluster contains cluster information and is automatically updated on the server when coordinators are added or removed.
Does the cluster file also automatically get updated on clients? If so, when?
Yes. The clients will try to update the fdb.cluster file as soon as they register a change in coordinators. This will usually happen pretty much instantly.
This might be a bit problematic if the client process runs as a different use than fdbserver processes on the same machine. So in the future the clients (or servers) will only update the file if it can keep the permissions the same.
The way this works is rather simple: clients connect to the cluster coordinators at startup and keeps this connection open. As soon as the coordinators change, the coordinators will send a forward to all clients with the new set of coordinators. The clients will then try to update the cluster file.
On the servers, the mechanism is exactly the same.
So for the when, the answer is: usually within seconds. This happens asynchronously though and might depend on your network. But it does not get piggy-packed with other requests (so it doesn’t depend on clients executing transactions).
Thanks! And as a side note, what are the downsides to making all nodes in a cluster coordinators?