Multiple DR running simultaneously

Using FoundationDB
1

Hi ,

I have a question regarding DR option for cross region replication. Can we have 2 DRs with the same primary i.e. the source is same but the destination are different. In this event , does one of the DR perform really bad as compared to the other one? I have given it a try with nyc-ldn and nyc-hkg setup, but one of the DR is performing considerably badly as compared to the other one. Also, in status details of primary cluster : 1 DR as primary is showing but in both the secondary regions, it’s showing Running DR: 1 as secondary. I have configured the DR with different tags as well.

Thanks.

2

Further details: I am using FDB version 6.2.19 , from what I have tested so far, it looks like the primary is always showing the tag for the last DR that was set.

3

I don’t believe it’s possible to have more than two usable_regions in a multi-region cluster. I could have sworn there was an issue somewhere about addressing that, but I wasn’t able to find it on GitHub.

However, I thought that was possible with the multi-cluster DR solution that predates the multi-region configuration, detailed more here (where you run 1 cluster per region and then use the fdbdr tool and dr_agents to asynchronously push changes from one cluster to the others): Backup, Restore, and Replication for Disaster Recovery — FoundationDB 7.1 Then both configured DRs should be able to copy information from the primary to the secondaries, though switching the primary might be complicated. You could also do something like run two regions as one cluster in a multi-region configuration, and then have the third configured using fdbdr. That being said, I’m not sure if this has been tried, so you might run into problems.

In what way is one worse than the other? I’m not really an expert, but it wouldn’t surprise me if, for example, London-NYC had a smaller lag than NYC-Hong Kong because of differences in networking delays, say

4

I have tried single DR setups also I mean with just nyc-ldn at a time and nyc-hkg at a time, but the lag seems way off now .

Also, I am confused about how many DRs status detail reports in the primary:

Primary(NYC) status details:

Configuration:
Redundancy mode - triple
Storage engine - ssd-2
Coordinators - 5

Cluster:
FoundationDB processes - 25
Zones - 5
Machines - 5
Memory availability - 150.2 GB per process on machine with least available
Fault Tolerance - 2 machines
Server time - 07/13/20 17:19:22

Data:
Replication health - Healthy
Moving data - 0.000 GB
Sum of key-value sizes - 0 MB
Disk space used - 937 MB

Operating space:
Storage server - 967.1 GB free on most full server
Log server - 967.4 GB free on most full server

Workload:
Read rate - 35 Hz
Write rate - 3 Hz
Transactions started - 23 Hz
Transactions committed - 3 Hz
Conflict rate - 1 Hz

Backup and DR:
Running backups - 0
Running DRs - 1 as primary

Running DR tags (as primary):
hkg - fde2ba65c66d356324bebb827c27fa25

Process performance details:
10.134.188.57:4490 ( 0% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.134.188.57:4491 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.134.188.57:4492 ( 0% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.134.188.57:4493 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.134.188.57:4494 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.134.188.101:4490 ( 0% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.134.188.101:4491 ( 0% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.134.188.101:4492 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.134.188.101:4493 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.134.188.101:4494 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.134.188.111:4490 ( 0% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.134.188.111:4491 ( 0% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.134.188.111:4492 ( 0% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.134.188.111:4493 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.134.188.111:4494 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.134.188.112:4490 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.134.188.112:4491 ( 0% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.134.188.112:4492 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.134.188.112:4493 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.134.188.112:4494 ( 0% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.134.188.118:4490 ( 0% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.134.188.118:4491 ( 0% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.134.188.118:4492 ( 1% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.134.188.118:4493 ( 1% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.134.188.118:4494 ( 0% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )

Coordination servers:
10.134.188.57:4490 (reachable)
10.134.188.101:4490 (reachable)
10.134.188.111:4490 (reachable)
10.134.188.112:4490 (reachable)
10.134.188.118:4490 (reachable)

Client time: 07/13/20 17:19:22

Secondary(HKG) status details:

Configuration:
Redundancy mode - triple
Storage engine - ssd-2
Coordinators - 5

Cluster:
FoundationDB processes - 25
Zones - 5
Machines - 5
Memory availability - 150.2 GB per process on machine with least available
Fault Tolerance - 2 machines
Server time - 07/14/20 05:20:35

Data:
Replication health - Healthy
Moving data - 0.000 GB
Sum of key-value sizes - 23 MB
Disk space used - 935 MB

Operating space:
Storage server - 966.9 GB free on most full server
Log server - 966.9 GB free on most full server

Workload:
Read rate - 28 Hz
Write rate - 0 Hz
Transactions started - 7 Hz
Transactions committed - 0 Hz
Conflict rate - 0 Hz

Backup and DR:
Running backups - 0
Running DRs - 1 as secondary

Running DR tags (as secondary):
hkg - fde2ba65c66d356324bebb827c27fa25

Process performance details:
10.174.204.6:4490 ( 0% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.174.204.6:4491 ( 0% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.174.204.6:4492 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.174.204.6:4493 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.174.204.6:4494 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.174.204.7:4490 ( 0% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.174.204.7:4491 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.174.204.7:4492 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.174.204.7:4493 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.174.204.7:4494 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.174.204.9:4490 ( 0% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.174.204.9:4491 ( 0% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.174.204.9:4492 ( 0% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.174.204.9:4493 ( 1% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.174.204.9:4494 ( 0% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.174.204.23:4490 ( 0% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.174.204.23:4491 ( 0% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.174.204.23:4492 ( 0% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.174.204.23:4493 ( 1% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.4 GB / 150.2 GB RAM )
10.174.204.23:4494 ( 0% cpu; 0% machine; 0.000 Gbps; 0% disk IO; 0.2 GB / 150.2 GB RAM )
10.174.204.29:4490 ( 0% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.174.204.29:4491 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.1 GB / 150.2 GB RAM )
10.174.204.29:4492 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.3 GB / 150.2 GB RAM )
10.174.204.29:4493 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.4 GB / 150.2 GB RAM )
10.174.204.29:4494 ( 1% cpu; 0% machine; 0.001 Gbps; 0% disk IO; 0.1 GB / 150.2 GB RAM )

Coordination servers:
10.174.204.6:4490 (reachable)
10.174.204.7:4490 (reachable)
10.174.204.9:4490 (reachable)
10.174.204.23:4490 (reachable)
10.174.204.29:4490 (reachable)

Client time: 07/14/20 05:20:35

Secondary(LDN) status details:

Configuration:
Redundancy mode - triple
Storage engine - ssd-2
Coordinators - 5

Cluster:
FoundationDB processes - 25
Zones - 5
Machines - 5
Memory availability - 148.0 GB per process on machine with least available
Fault Tolerance - 2 machines
Server time - 07/13/20 22:21:37

Data:
Replication health - Healthy
Moving data - 0.000 GB
Sum of key-value sizes - 0 MB
Disk space used - 915 MB

Operating space:
Storage server - 1633.8 GB free on most full server
Log server - 1633.8 GB free on most full server

Workload:
Read rate - 387 Hz
Write rate - 690 Hz
Transactions started - 120 Hz
Transactions committed - 37 Hz
Conflict rate - 8 Hz

Backup and DR:
Running backups - 0
Running DRs - 1 as secondary

Running DR tags (as secondary):
emea - b196d822c46854067e4249bf396cd066

Process performance details:
10.164.170.18:4490 ( 0% cpu; 0% machine; 0.002 Gbps; 0% disk IO; 0.3 GB / 149.9 GB RAM )
10.164.170.18:4491 ( 1% cpu; 0% machine; 0.002 Gbps; 0% disk IO; 0.2 GB / 149.9 GB RAM )
10.164.170.18:4492 ( 1% cpu; 0% machine; 0.002 Gbps; 0% disk IO; 0.3 GB / 149.9 GB RAM )
10.164.170.18:4493 ( 1% cpu; 0% machine; 0.002 Gbps; 0% disk IO; 0.3 GB / 149.9 GB RAM )
10.164.170.18:4494 ( 0% cpu; 0% machine; 0.002 Gbps; 0% disk IO; 0.2 GB / 149.9 GB RAM )
10.164.183.10:4490 ( 0% cpu; 0% machine; 0.004 Gbps; 0% disk IO; 0.3 GB / 148.0 GB RAM )
10.164.183.10:4491 ( 1% cpu; 0% machine; 0.004 Gbps; 0% disk IO; 0.2 GB / 148.0 GB RAM )
10.164.183.10:4492 ( 0% cpu; 0% machine; 0.004 Gbps; 0% disk IO; 0.2 GB / 148.0 GB RAM )
10.164.183.10:4493 ( 2% cpu; 0% machine; 0.004 Gbps; 0% disk IO; 0.3 GB / 148.0 GB RAM )
10.164.183.10:4494 ( 1% cpu; 0% machine; 0.004 Gbps; 0% disk IO; 0.2 GB / 148.0 GB RAM )
10.164.208.253:4490 ( 0% cpu; 0% machine; 0.003 Gbps; 0% disk IO; 0.3 GB / 148.3 GB RAM )
10.164.208.253:4491 ( 0% cpu; 0% machine; 0.003 Gbps; 0% disk IO; 0.2 GB / 148.3 GB RAM )
10.164.208.253:4492 ( 0% cpu; 0% machine; 0.003 Gbps; 0% disk IO; 0.2 GB / 148.3 GB RAM )
10.164.208.253:4493 ( 2% cpu; 0% machine; 0.003 Gbps; 0% disk IO; 0.3 GB / 148.3 GB RAM )
10.164.208.253:4494 ( 0% cpu; 0% machine; 0.003 Gbps; 0% disk IO; 0.2 GB / 148.3 GB RAM )
10.164.210.85:4490 ( 0% cpu; 0% machine; 0.004 Gbps; 0% disk IO; 0.3 GB / 149.8 GB RAM )
10.164.210.85:4491 ( 0% cpu; 0% machine; 0.004 Gbps; 0% disk IO; 0.2 GB / 149.8 GB RAM )
10.164.210.85:4492 ( 1% cpu; 0% machine; 0.004 Gbps; 0% disk IO; 0.3 GB / 149.8 GB RAM )
10.164.210.85:4493 ( 2% cpu; 0% machine; 0.004 Gbps; 0% disk IO; 0.3 GB / 149.8 GB RAM )
10.164.210.85:4494 ( 1% cpu; 0% machine; 0.004 Gbps; 0% disk IO; 0.2 GB / 149.8 GB RAM )
10.164.214.77:4490 ( 1% cpu; 0% machine; 0.005 Gbps; 0% disk IO; 0.3 GB / 149.9 GB RAM )
10.164.214.77:4491 ( 0% cpu; 0% machine; 0.005 Gbps; 0% disk IO; 0.2 GB / 149.9 GB RAM )
10.164.214.77:4492 ( 1% cpu; 0% machine; 0.005 Gbps; 0% disk IO; 0.3 GB / 149.9 GB RAM )
10.164.214.77:4493 ( 1% cpu; 0% machine; 0.005 Gbps; 0% disk IO; 0.3 GB / 149.9 GB RAM )
10.164.214.77:4494 ( 1% cpu; 0% machine; 0.005 Gbps; 0% disk IO; 0.2 GB / 149.9 GB RAM )

Coordination servers:
10.164.170.18:4490 (reachable)
10.164.183.10:4490 (reachable)
10.164.208.253:4490 (reachable)
10.164.210.85:4490 (reachable)
10.164.214.77:4490 (reachable)

Client time: 07/13/20 22:21:37

5

Also, for DR with tag emea:
foundationdb-bin-6.2.19/rhel6/client/fdbdr status -s fdb_americas.cluster -d fdb_emea.cluster -t emea
The DR on tag `emea’ is a complete copy of the primary database.

The DR is 0.393925 seconds behind.

and for tag hkg:
foundationdb-bin-6.2.19/rhel6/client/fdbdr status -t hkg -d fdb_hkg.cluster -s fdb_americas.cluster
The DR on tag `hkg’ is NOT a complete copy of the primary database.

The DR is 2171.646958 seconds behind.