Restore multiple ranges in a kvranges file are failed

walter · May 22, 2023, 8:20am

I am working on a recovery service, attempting to recover multiple key ranges from backup as needed, so I conducted a test and found this issue.

Now here is a backup that contains keys aa, aaa, b, bb and bbb. Now I need to use restore to recover data and filter out a certain range, eg: b. I try to restore it with cmd:

fdbrestore start --dest-cluster-file /home/xxxx/workspace/fdb_test_env/fdb_cluster/fdb.cluster -r file:///home/xxxx/workspace/fdb_test_env/backups/backup-2023-05-22-11-56-40.011576/ -k 'bb bbbb' -k 'aa aaaa'

After the restore is finished, I try to use getrange to verify whether the restore was successfully executed, but I found that there was no range bb bbbb in the results:

Welcome to the fdbcli. For help, type `help'.
fdb> getrange a z

Range limited to 25 keys
`aa' is `2a'
`aaa' is `3a'

The excepted results should be:

Welcome to the fdbcli. For help, type `help'.
fdb> getrange a z

Range limited to 25 keys
`aa' is `2a'
`aaa' is `3a'
`bb' is `2b'
`bbb' is `3b'

I tracked the code and found that the problem may be caused by adjusting the fileRange during the RestoreRangeTaskFunc::_execute:

github.com

apple/foundationdb/blob/main/fdbclient/FileBackupAgent.actor.cpp#L3803-L3844


      
          		state KeyRange fileRange = KeyRangeRef(blockData.front().key, blockData.back().key);
          		state std::vector<KeyRange> originalFileRanges;
          		// If fileRange doesn't intersect restore range then we're done.
          		state int index;
          		for (index = 0; index < restoreRanges.get().size(); index++) {
          			auto& restoreRange = restoreRanges.get()[index];
          			if (!fileRange.intersects(restoreRange))
          				continue;
          
          
			// We know the file range intersects the restore range but there could still be keys outside the restore
          			// range. Find the subvector of kv pairs that intersect the restore range.  Note that the first and last
          			// keys are just the range endpoints for this file
          			int rangeStart = 1;
          			int rangeEnd = blockData.size() - 1;
          			// Slide start forward, stop if something in range is found
          			while (rangeStart < rangeEnd && !restoreRange.contains(blockData[rangeStart].key))
          				++rangeStart;
          			// Side end backward, stop if something in range is found
          			while (rangeEnd > rangeStart && !restoreRange.contains(blockData[rangeEnd - 1].key))
          				--rangeEnd;

This file has been truncated. show original

Line 3809, determine if there is an intersection between fileRange and restoreRange.
Line 3834-3840, update fileRange to the dest restoreRange.

When the program executes to 3809 again, the data in fileRange is no longer correct, so it does not intersect with restoreRange.

BTW, I am interested in fixing this issue, but I do not have permission to provide PR for Apple/Foundationdb’s repo. Can someone help invite me? My github ID is w41ter, thank you!

jzhou · May 23, 2023, 4:13am

I’m actually surprised the test didn’t find this issue. Looks like line 3803 should be moved inside the for loop. Thanks for finding this bug and @ammolitor can invite you to create PRs.

walter · May 24, 2023, 2:34am

Thanks

Hi, @ammolitor, could I get permission to open pull requests to the foundationdb repository? My github id is w41ter.

Topic		Replies	Views
Restoring key range from backup Using FoundationDB	0	241	August 7, 2023
`fdbrestore` with --remove-prefix and --keys fails on error: "begin or end of key range does not start with removePrefix" Using FoundationDB	2	489	April 27, 2024
Unable to restore a 6.0.15 backup to 6.3 Using FoundationDB	6	620	November 14, 2022
Why FoudationDB can't restore data to a non-empty database Using FoundationDB	2	957	March 9, 2020
Is the database fully offline during a key-range-based restore? Using FoundationDB	7	1146	July 10, 2024

Restore multiple ranges in a kvranges file are failed

Related topics