About incremental backup and restore for fdb dr

Development
1

Hi, I’ve got a requirement to perform incremental backup of a fdb cluster, and later to incrementally perform the restore to another fdb cluster.

Is it possible in current implementation of fdbbackup, fdbrestore, etc?

I checked the code of fdbbackup.

From backup.actor.cpp, runRestore’s Main workflow

  • Step1: decide targetVersion (by Parameter or targetTimeStamp)
  • Step2: openBackupContainer as bc
  • Step3: maximum restorable version (bc->describeBackup) if targetVersion is invalid
  • Step4: perform restore with backupAgent.restore or quit if not restorable

I chose FileBackupAgent.actor.cpp as the path to get familiar with the logic.
This way leads to StartFullRestoreTaskFunc, but before that, in submitRestore, there is a check to ensure there is no existing rows in the restore range.

for (index = 0; index < restoreRanges.size(); index++) {
			KeyRange restoreIntoRange = KeyRangeRef(restoreRanges[index].begin, restoreRanges[index].end)
			                                .removePrefix(removePrefix)
			                                .withPrefix(addPrefix);
			RangeResult existingRows = wait(tr->getRange(restoreIntoRange, 1));
			if (existingRows.size() > 0 && !onlyApplyMutationLogs) {
				throw restore_destination_not_empty();
			}
		}

Is the mainly because the restore comes both from snapshot and mutationlogs?

It’s to avoid such kind of scenarios
“Someone deleted a key “AAA” at the src site. The delete op has been applied to storage server, so it’s no longer part of mutationlogs. During restore, we can only get the snapshot of storage server and mutationlogs. Neither the snapshot nor the log can reflect the delete of key AAA. if we restore without the emptiness of destination side, this delete op will not be executed on the destination side. And AAA will be left on the destination site”

And if this description is correct, any plan or design to support incremental restore without the need to clear dest site.
New to fdb, thanks a lot for your help.

2

Not sure about this understanding: backup data contains two parts, one is from tlog, another is from storage server. I got this impression somewhere. But when I check the code today, I can only find the logic to pull data from TLog system in BackupWorker.actor.cpp

ACTOR Future<Void> pullAsyncData(BackupData* self) {