# Can not restore fdb 6.2.98 backup into 6.3.24

**URL:** <https://forums.foundationdb.org/t/can-not-restore-fdb-6-2-98-backup-into-6-3-24/3530>\
**Category:** Using FoundationDB\
**Tags:** performance, bindings, operator\
**Created:** [September 1, 2022, 10:49am UTC](https://forums.foundationdb.org/t/can-not-restore-fdb-6-2-98-backup-into-6-3-24/3530 "2022-09-01T10:49:42Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![liyyue](https://avatars.discourse-cdn.com/v4/letter/l/54ee81/32.png) [@liyyue](https://forums.foundationdb.org/u/liyyue)\
**Post date:** [September 1, 2022, 10:49am UTC](https://forums.foundationdb.org/t/can-not-restore-fdb-6-2-98-backup-into-6-3-24/3530/1 "2022-09-01T10:49:42Z")

</div>

Hi,Experts

Now I am trying to restore a fdb backup from 6.2.98 into 6.3.24, I got following error message:

```auto
...
<Event Severity="10" Time="1662014110.218822" DateTime="2022-09-01T06:35:10Z" Type="BackupContainerGetRestoreSet" ID="0000000000000000" RangeFilter=" - \xff" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="10" Time="1662014110.220801" DateTime="2022-09-01T06:35:10Z" Type="AsyncFileOpened" ID="0000000000000000" SuppressedEventCount="0" Filename="/var/backup_volume/backup-2022-09-01-06-02-23.603897/snapshots/snapshot,693800162,693800162,23" Fd="13" Flags="2228225" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="20" Time="1662014110.221161" DateTime="2022-09-01T06:35:10Z" Type="TB_ExecuteFailure" ID="0000000000000000" Error="backup_not_filterable_with_key_ranges" ErrorDescription="Backup before 6.3 cannot be filtered with key ranges" ErrorCode="2320" TaskUID="256497c1e122e1f96d09178ba8f77ae7" TaskType="restore_start" Priority="0" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="20" Time="1662014110.221161" DateTime="2022-09-01T06:35:10Z" Type="FileRestoreError" ID="0000000000000000" Error="backup_not_filterable_with_key_ranges" ErrorDescription="Backup before 6.3 cannot be filtered with key ranges" ErrorCode="2320" RestoreUID="81bf2873a4fe5d15" Description="'Backup before 6.3 cannot be filtered with key ranges' on 'restore_start'" TaskInstance="0" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="10" Time="1662014110.221161" DateTime="2022-09-01T06:35:10Z" Type="AsyncFileClosed" ID="0000000000000000" SuppressedEventCount="0" Fd="13" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="10" Time="1662014110.227570" DateTime="2022-09-01T06:35:10Z" Type="CodeCoverage" ID="0000000000000000" File="/home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/TaskBucket.actor.cpp" Line="222" Condition="true" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="10" Time="1662014110.239629" DateTime="2022-09-01T06:35:10Z" Type="FileRestoreVersionUpgrade" ID="0000000000000000" RestoreVersion="753761706" Dest="753796882" InconsistentSnapshotOnly="0" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="10" Time="1662014110.239629" DateTime="2022-09-01T06:35:10Z" Type="BackupContainerGetRestoreSet" ID="0000000000000000" RangeFilter=" - \xff" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="20" Time="1662014110.241516" DateTime="2022-09-01T06:35:10Z" Type="TB_ExecuteFailure" ID="0000000000000000" Error="backup_not_filterable_with_key_ranges" ErrorDescription="Backup before 6.3 cannot be filtered with key ranges" ErrorCode="2320" TaskUID="256497c1e122e1f96d09178ba8f77ae7" TaskType="restore_start" Priority="0" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
...

```

Is there any suggestion to fix it? Thanks!

---

<div class="post-metadata">

**Author:** ![jzhou](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/jzhou/32/445_2.png) [@jzhou](https://forums.foundationdb.org/u/jzhou)\
**Post date:** [September 6, 2022, 4:10pm UTC](https://forums.foundationdb.org/t/can-not-restore-fdb-6-2-98-backup-into-6-3-24/3530/2 "2022-09-06T16:10:53Z")

</div>

This error

> [@liyyue](#):
>
> `backup_not_filterable_with_key_ranges`

suggests you used `-k` or `--keys` options for restore. Can you try without this flag?

---

<div class="post-metadata">

**Author:** ![liyyue](https://avatars.discourse-cdn.com/v4/letter/l/54ee81/32.png) [@liyyue](https://forums.foundationdb.org/u/liyyue)\
**Post date:** [September 7, 2022, 2:32pm UTC](https://forums.foundationdb.org/t/can-not-restore-fdb-6-2-98-backup-into-6-3-24/3530/3 "2022-09-07T14:32:47Z")

</div>

@jzhou thanks for your help on this issue! I have tried the -t option, but the problem still exists:  
fdbrestore start --dest\_cluster\_file /etc/foundationdb/fdb.cluster -w -r file:///data/fdbbak/backup-2022-09-07-07-26-49.129822 -k a (here a is one test key)

```auto
...
Tag: default UID: 564836a4ed6519615050bc91c194164f State: starting Blocks: 0/0 BlocksInProgress: 0 Files: 0 BytesWritten: 0 CurrentVersion: -1 FirstConsistentVersion: 0 ApplyVersionLag: 0 LastError: ''Backup before 6.3 cannot be filtered with key ranges' on 'restore_start'' 4s ago.
...

```

And the -k operation can only support a specific key or key range? If customer has many key ranges, such as 300G data, how can we restore it to a higher version db?

---

<div class="post-metadata">

**Author:** ![jzhou](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/jzhou/32/445_2.png) [@jzhou](https://forums.foundationdb.org/u/jzhou)\
**Post date:** [September 7, 2022, 4:27pm UTC](https://forums.foundationdb.org/t/can-not-restore-fdb-6-2-98-backup-into-6-3-24/3530/4 "2022-09-07T16:27:57Z")

</div>

I think restore old versions of backup to newer versions of FDB cluster is something not tested. In your case, it’s probably better to restore the backup to 6.2 cluster, then upgrade the cluster from 6.2 to 6.3.

Regarding `-k` option, the option value is a list of key ranges to be restored, each range is separated by a space. See how the option is parsed here: [foundationdb/backup.actor.cpp at d3d1ff602320061d5e50f8aa648442ec1e779033 · apple/foundationdb · GitHub](https://github.com/apple/foundationdb/blob/d3d1ff602320061d5e50f8aa648442ec1e779033/fdbbackup/backup.actor.cpp#L2940). The documentation [Backup, Restore, and Replication for Disaster Recovery — FoundationDB 7.1](https://apple.github.io/foundationdb/backups.html#fdbrestore-command-line-tool) doesn’t seem to give examples of the exact format for key ranges.

---

<div class="post-metadata">

**Author:** ![liyyue](https://avatars.discourse-cdn.com/v4/letter/l/54ee81/32.png) [@liyyue](https://forums.foundationdb.org/u/liyyue)\
**Post date:** [September 8, 2022, 1:27am UTC](https://forums.foundationdb.org/t/can-not-restore-fdb-6-2-98-backup-into-6-3-24/3530/5 "2022-09-08T01:27:43Z")

</div>

@jzhou thanks for your suggestion! Now we still encounter some problem when upgrade fdb cluster from 6.2 to 6.3. After upgrade, I encounter the following error message:

```auto
...
Time="1662460973.975867" Severity="40" LogGroup="mdm-foundationdb-ibm" Process="fdbserver.1": Warning: FDBD has not joined the cluster after 5 seconds.
...

```

---

<div class="post-metadata">

**Author:** ![jzhou](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/jzhou/32/445_2.png) [@jzhou](https://forums.foundationdb.org/u/jzhou)\
**Post date:** [September 8, 2022, 3:41am UTC](https://forums.foundationdb.org/t/can-not-restore-fdb-6-2-98-backup-into-6-3-24/3530/6 "2022-09-08T03:41:27Z")

</div>

Can you check all `fdbserver` binaries are upgraded to 6.3? This looks like they are using different versions.

---

<div class="post-metadata">

**Author:** ![liyyue](https://avatars.discourse-cdn.com/v4/letter/l/54ee81/32.png) [@liyyue](https://forums.foundationdb.org/u/liyyue)\
**Post date:** [September 9, 2022, 1:51am UTC](https://forums.foundationdb.org/t/can-not-restore-fdb-6-2-98-backup-into-6-3-24/3530/7 "2022-09-09T01:51:19Z")

</div>

@jzhou I have not found any guide to introduce how to upgrade fdb cluster on k8s/ocp environment, so I just tried some way myself 🙂 I just replaced the docker image for the fdb pod from 6.2 to 6.3. Before upgrade I set the skip to true for foundationdbcluster cr and delete all fdb pods, then I changed the cluster yaml file by replacing the image from 6.2 to 6.3, then set skip to false in cluster yaml file and reapply the yaml file, then the cluster rebuilt with 6.3 image. I have checked all fdb pods are running with 6.3 images now. Is there any additional steps need to do? Or you have any formal guide to introduce upgrade fdb on k8s environment? Thanks!

---

<div class="post-metadata">

**Author:** ![liyyue](https://avatars.discourse-cdn.com/v4/letter/l/54ee81/32.png) [@liyyue](https://forums.foundationdb.org/u/liyyue)\
**Post date:** [October 12, 2022, 6:57am UTC](https://forums.foundationdb.org/t/can-not-restore-fdb-6-2-98-backup-into-6-3-24/3530/8 "2022-10-12T06:57:16Z")

</div>

@jzhou @johscheuer I have checked the foundationdb code and found the following code:

```auto
...
// Old backup does not have metadata about key ranges and can not be filtered with key ranges.
if (keyRangesFilter.size() && results.second.empty() && !results.first.empty()) {
	throw backup_not_filterable_with_key_ranges();
}
...

```

It seems fdb 6.2 backup does not have metadata about key ranges and can not be filtered with key ranges, right?  
Is there any workaround to restore 6.2 backup into 6.3 database. Any more suggestion? Thanks!

---

<div class="post-metadata">

**Author:** ![SteavedHams](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/steavedhams/32/18_2.png) [@SteavedHams](https://forums.foundationdb.org/u/SteavedHams)\
**Post date:** [February 24, 2023, 8:58am UTC](https://forums.foundationdb.org/t/can-not-restore-fdb-6-2-98-backup-into-6-3-24/3530/9 "2023-02-24T08:58:44Z")

</div>

@liyyue This is the fix for this issue, it will be in 7.3 but it’s a simple fix in case you want to patch an older version.

> <https://github.com/apple/foundationdb/pull/9467>
>
> Bug fix: Partial restore still works if the backup does not have a key range ma…p, the restore is just less efficient because no range files can be filtered out.
> 
> \# Code-Reviewer Section
> 
> The general pull request guidelines can be found \[here\](https://github.com/apple/foundationdb/wiki/FoundationDB-Commit-Process).
> 
> Please check each of the following things and check \*all\* boxes before accepting a PR.
> 
> \- \[\] The PR has a description, explaining both the problem and the solution.
> \- \[\] The description mentions which forms of testing were done and the testing seems reasonable.
> \- \[\] Every function/class/actor that was touched is reasonably well documented.
> 
> \## For Release-Branches
> 
> If this PR is made against a release-branch, please also check the following:
> 
> \- \[\] This change/bugfix is a cherry-pick from the next younger branch (younger \`release-branch\` or \`main\` if this is the youngest branch)
> \- \[\] There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)