We are slowly migrating our fleet from 3.x to 5.x and I have run into an issue when trying to assign new coordinators. The error message is as such:
ERROR: One of the specified coordinators is unreachable (which isn’t incredibly helpful as it does not specify which one it is)
However the coordinators I am assigning are totally reachable, in fact the cluster recognizes them as healthy and a member of the current acting cluster.
Has anyone run into this before? or know which trace files i should begin looking at to ascertain what is causing the coordination state to not change?
I have checked the master, cluster_controller, and the coordinators during the requested change and nothing is outstanding that hints as to why it will not update the coordinators