I’m not 100% sure if this is a bug or not but figured I’d leave a note. I’ve got a test cluster that was accidentally reconfigured to have a single fdbserver per node due to some chef mishaps. While debugging what had happened I discovered one of the nodes had the entire set of proceses in the pre-single-fdbserver config still in the cluster.
Out of curiosity I
/etc/foundationdb/foundationdb.conf to see if maybe Chef had done something odd that prevented fdbmonitor from noticing the new config file. However, after touching foundationdb.conf, fdbmonitor logged that it was deconfiguring each of the expected processes to match the new config, yet none of the excess fdbserver processes have been stopped and are still part of the cluster.
Also to be clear, we’re being particularly brutal when we apply new test configurations. These are generally accompanied by
rm -rf $data_dir type operations as we reset a test cluster for benchmarking. It just surprised me enough to make a note.