Could not communicate with a quorum of coordination servers

Using FoundationDB
1

Hi,

I have two-region, 3-dc setup.

Today I tried to do a migration of DC3 (moving nodes from one datacenter to another).
I already configured usable_regions to 1 (to be DC1) and then drop all the nodes (kubenetes pods) in old DC3. Unfortunately, that’s when I realized that I lost the majority of the coordination servers. Doing a fdbcli status on a node in DC1 gives me:

Using cluster file `/var/lib/foundationdb/fdb.cluster'.

The database is unavailable; type `status' for more information.

Welcome to the fdbcli. For help, type `help'.
fdb> status

Using cluster file `/var/lib/foundationdb/fdb.cluster'.

Could not communicate with a quorum of coordination servers:
  10.104.192.30:4000:tls  (unreachable)
  10.104.196.250:4000:tls  (unreachable)
  10.104.198.234:4000:tls  (reachable)
  10.175.156.235:4500:tls  (unreachable)
  10.175.165.152:4500:tls  (unreachable)
  10.175.176.20:4500:tls  (unreachable)
  10.199.100.36:4500:tls  (reachable)
  10.199.133.42:4500:tls  (reachable)
  10.199.157.94:4500:tls  (reachable)

Regarding the fact that the data is still there (in DC1), is there a way to fix this?
I tried to add more reachable processes to the fdb cluster file but it does not work.

2

We do not currently have any tooling to be able to recover a cluster from a minority of coordinators. If you’ve permanently lost their data, then although that data does still exist in storage server files, there’s not an existing tool that you can use to optimistically recover as much of it as possible.

Though clearly too late, we generally advise having three regions with coordinators in them for multi-region, so that no region has a majority of the coordinators in it.

3

Thanks for confirm, @alexmiller.
Our cluster already configured so that no region can have majority of the coordinators.
Unfortunately, some coordinators of the remaining two regions were also down too, that caused the majority of coordinators to be lost.
I already dropped, redeploy a new cluster and start loading the data again. It is also a good lesson to learn: Never lose a majority of the coordinators, or the cluster will not be functional (please correct me if it is wrong).