Full vs. incremental resync when usable_regions config is changed

Using FoundationDB Running FoundationDB
1

We have a fdb cluster with 3-DC/2-region configuration built upon a secured Kubernetes env. K8s host OS is required to have timely (monthly) patching/upgrading to keep it secure. The host OS patching needs to reboot the K8s nodes of the fdb pods, thus affecting the fdb cluster.

We’d like to make a DC offline, so that K8s Os patching can be done more freely without the concern of affecting fdb cluster on the DC. Let’s say we make DC1 offline by switching primary DC to DC3 and setting usable_regions=1.

When I bring up DC1 online (by setting usable_regions=2), fdb will do a full resync between DC3 and DC1, even when DC1 is offline for a few mins. The full resync takes 10+ hours in our case.

We’d desire an incremental resync, meaning keeping the original data at DC1 and only resync what’s changed since DC1 was offline, assuming the changes are still in Tx logs. That would be much faster for us.

Can you confirm if an incremental resync is possible? Thanks.

2

Yes, just make the DC that will remain up the primary, and don’t reduce usable_regions before you take the kubernetes pods offline. It will look to FDB like the secondary region went offline, and it will start buffering all changes on the transaction logs, and replay them on the secondary when it comes back online.

3

That’s interesting. We’ll try it. We build the cluster upon Kubernetes. One way to simulate a pod as down is to replace the pod’s FDB Docker image by a dummy image, so that fdb cluster cannot communicate to it. Then revert back to fdb image to bring it up. We will test this approach. Thanks, Alex.