I have two-region, 3-dc setup.
Today I tried to do a migration of DC3 (moving nodes from one datacenter to another).
I already configured usable_regions to 1 (to be DC1) and then drop all the nodes (kubenetes pods) in old DC3. Unfortunately, that’s when I realized that I lost the majority of the coordination servers. Doing a
fdbcli status on a node in DC1 gives me:
Using cluster file `/var/lib/foundationdb/fdb.cluster'. The database is unavailable; type `status' for more information. Welcome to the fdbcli. For help, type `help'. fdb> status Using cluster file `/var/lib/foundationdb/fdb.cluster'. Could not communicate with a quorum of coordination servers: 10.104.192.30:4000:tls (unreachable) 10.104.196.250:4000:tls (unreachable) 10.104.198.234:4000:tls (reachable) 10.175.156.235:4500:tls (unreachable) 10.175.165.152:4500:tls (unreachable) 10.175.176.20:4500:tls (unreachable) 10.199.100.36:4500:tls (reachable) 10.199.133.42:4500:tls (reachable) 10.199.157.94:4500:tls (reachable)
Regarding the fact that the data is still there (in DC1), is there a way to fix this?
I tried to add more reachable processes to the fdb cluster file but it does not work.