Two datacenters with double redundancy in each?

I’m trying to build a configuration that can provide double redundancy across two data centers. My goal is to support the failure of an entire data center, plus one machine in the remaining data center, without affecting availability.

If I configure a single region cluster (using double redundancy) with two datacenters in that region, will FoundationDB automatically keep two copies of data in each data center?

We are also trying to find a good solution to run on two data centers and survive the failure of one. Sadly I am not aware of a good way of doing that. You can use fdbdr to get a hot standby, but then you still risk some data loss if you lose a data center.

The fundamental problem with two data centers is that there really is no good way to distribute your coordinators across two data centers. The coordinators run a majority vote so you always need to have a majority of coordinators up in order to survive a data center failure.

That’s a good point, Markus!

My most important requirement is having two copies of the data in each of both data centers. To always maintain write/update capability in the event of a DC failure in a 2-datacenter scenario, I could potentially maintain a small satellite in AWS so that a majority of coordinators can still be available to the survivor.

Another question comes to mind though; if a datacenter (which happens to have a minority of coordinators) loses it’s outbound network link, but is otherwise functioning internally, can I still read the data in that data center, even if it cannot be changed?

If you can do that (for some users this might be not possible due to security compliance reasons) that would probably work well as the amount of traffic that goes to the coordinators is tiny.

No. In such a scenario the cluster would not be able to recover and clients wouldn’t be able to open a connection.

Then it sounds like you’ll be fine. I’d suggest running 9 coordinators (3 in each DC and 3 in AWS) so that you can lose 1 DC and 1 machine and still have quorum. If you make a region config where you treat each of your datacenters as a “region”, you’ll be able to achieve your goals with multi-region. You should set one of the datacenters as your preferred primary, and then set up the other datacenter to be both a satellite and the preferred secondary.

1 Like

It’s possibly worth underscoring that in such a configuration, you also need to configure the logs to have presence in both regions, or FDB will never fail over from the primary to the secondary (as it can’t guarantee that there isn’t data only in the one region or the other), which is what setting the secondary data center as a satellite does.