Since I’ve started using FoundationDB one of the most idiosyncratic operational characteristics is how it wants to watch the configuration files and respond to changes. This has surprised me a few times, but today I really got thrown for a loop.
A cluster I was using started giving this warning every time I connected:
$ fdbcli -C fdb.cluster Using cluster file `fdb.cluster'. The database is available, but has issues (type 'status' for more information). Welcome to the fdbcli. For help, type `help'. fdb> status Using cluster file `fdb.cluster'. 1 client(s) reported: Cluster file contents do not match current cluster connection string. Verify the cluster file and its parent directory are writable and that the cluster file has not been overwritten externally. Configuration: Redundancy mode - triple Storage engine - ssd-2 Coordinators - 5 ...
I checked all the fdbcli processes and configuration everywhere (or so I thought) and I was really pulling my hair out after about 45 minutes. Eventually I found that
status details gave me an IP address and then I figured out that another laptop I’d run fdbcli on still had that session open (but I’d changed to another git branch in the meantime, removing the fdb.cluster out from underneath fdbcli).
This behavior is really surprising for two reasons:
- If I run some software with a configuration file that tells it what servers to connect to, I don’t expect that it will monitor that file after startup and care if it changes.
- I don’t expect that the particular state of some fdbcli process on some laptop somewhere causes a warning to appear for everyone who connects to the cluster (as though there’s some problem with the cluster itself).
So, two questions (or maybe feature requests):
Is it possible to run fdbcli in a way that loads the config once and then doesn’t monitor the file? I tried tricking fdbcli this way but it didn’t work:
$ fdbcli -C <(cat fdb.cluster) Unexpected error loading cluster file `/proc/self/fd/11': 1516 File too large to be read
Is there any way to suppress warnings caused by fdbclis that are connected to the cluster? If we end up deploying FDB at my company I imagine that the “Cluster file contents do not match current …” warnings will be semi-permanent due to weird fdbcli connections.