Can not restore fdb 6.2.98 backup into 6.3.24

Hi,Experts

Now I am trying to restore a fdb backup from 6.2.98 into 6.3.24, I got following error message:

...
<Event Severity="10" Time="1662014110.218822" DateTime="2022-09-01T06:35:10Z" Type="BackupContainerGetRestoreSet" ID="0000000000000000" RangeFilter=" - \xff" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="10" Time="1662014110.220801" DateTime="2022-09-01T06:35:10Z" Type="AsyncFileOpened" ID="0000000000000000" SuppressedEventCount="0" Filename="/var/backup_volume/backup-2022-09-01-06-02-23.603897/snapshots/snapshot,693800162,693800162,23" Fd="13" Flags="2228225" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="20" Time="1662014110.221161" DateTime="2022-09-01T06:35:10Z" Type="TB_ExecuteFailure" ID="0000000000000000" Error="backup_not_filterable_with_key_ranges" ErrorDescription="Backup before 6.3 cannot be filtered with key ranges" ErrorCode="2320" TaskUID="256497c1e122e1f96d09178ba8f77ae7" TaskType="restore_start" Priority="0" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="20" Time="1662014110.221161" DateTime="2022-09-01T06:35:10Z" Type="FileRestoreError" ID="0000000000000000" Error="backup_not_filterable_with_key_ranges" ErrorDescription="Backup before 6.3 cannot be filtered with key ranges" ErrorCode="2320" RestoreUID="81bf2873a4fe5d15" Description="'Backup before 6.3 cannot be filtered with key ranges' on 'restore_start'" TaskInstance="0" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="10" Time="1662014110.221161" DateTime="2022-09-01T06:35:10Z" Type="AsyncFileClosed" ID="0000000000000000" SuppressedEventCount="0" Fd="13" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="10" Time="1662014110.227570" DateTime="2022-09-01T06:35:10Z" Type="CodeCoverage" ID="0000000000000000" File="/home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/TaskBucket.actor.cpp" Line="222" Condition="true" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="10" Time="1662014110.239629" DateTime="2022-09-01T06:35:10Z" Type="FileRestoreVersionUpgrade" ID="0000000000000000" RestoreVersion="753761706" Dest="753796882" InconsistentSnapshotOnly="0" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="10" Time="1662014110.239629" DateTime="2022-09-01T06:35:10Z" Type="BackupContainerGetRestoreSet" ID="0000000000000000" RangeFilter=" - \xff" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
<Event Severity="20" Time="1662014110.241516" DateTime="2022-09-01T06:35:10Z" Type="TB_ExecuteFailure" ID="0000000000000000" Error="backup_not_filterable_with_key_ranges" ErrorDescription="Backup before 6.3 cannot be filtered with key ranges" ErrorCode="2320" TaskUID="256497c1e122e1f96d09178ba8f77ae7" TaskType="restore_start" Priority="0" ThreadID="533899467947030388" Machine="10.254.14.14:7" LogGroup="default" />
...

Is there any suggestion to fix it? Thanks!

This error

suggests you used -k or --keys options for restore. Can you try without this flag?

@jzhou thanks for your help on this issue! I have tried the -t option, but the problem still exists:
fdbrestore start --dest_cluster_file /etc/foundationdb/fdb.cluster -w -r file:///data/fdbbak/backup-2022-09-07-07-26-49.129822 -k a (here a is one test key)

...
Tag: default  UID: 564836a4ed6519615050bc91c194164f  State: starting  Blocks: 0/0  BlocksInProgress: 0  Files: 0  BytesWritten: 0  CurrentVersion: -1  FirstConsistentVersion: 0  ApplyVersionLag: 0  LastError: ''Backup before 6.3 cannot be filtered with key ranges' on 'restore_start'' 4s ago.
...

And the -k operation can only support a specific key or key range? If customer has many key ranges, such as 300G data, how can we restore it to a higher version db?

I think restore old versions of backup to newer versions of FDB cluster is something not tested. In your case, it’s probably better to restore the backup to 6.2 cluster, then upgrade the cluster from 6.2 to 6.3.

Regarding -k option, the option value is a list of key ranges to be restored, each range is separated by a space. See how the option is parsed here: foundationdb/backup.actor.cpp at d3d1ff602320061d5e50f8aa648442ec1e779033 · apple/foundationdb · GitHub. The documentation Backup, Restore, and Replication for Disaster Recovery — FoundationDB 7.1 doesn’t seem to give examples of the exact format for key ranges.

@jzhou thanks for your suggestion! Now we still encounter some problem when upgrade fdb cluster from 6.2 to 6.3. After upgrade, I encounter the following error message:

...
Time="1662460973.975867" Severity="40" LogGroup="mdm-foundationdb-ibm" Process="fdbserver.1": Warning: FDBD has not joined the cluster after 5 seconds.
...

Can you check all fdbserver binaries are upgraded to 6.3? This looks like they are using different versions.

@jzhou I have not found any guide to introduce how to upgrade fdb cluster on k8s/ocp environment, so I just tried some way myself :slight_smile: I just replaced the docker image for the fdb pod from 6.2 to 6.3. Before upgrade I set the skip to true for foundationdbcluster cr and delete all fdb pods, then I changed the cluster yaml file by replacing the image from 6.2 to 6.3, then set skip to false in cluster yaml file and reapply the yaml file, then the cluster rebuilt with 6.3 image. I have checked all fdb pods are running with 6.3 images now. Is there any additional steps need to do? Or you have any formal guide to introduce upgrade fdb on k8s environment? Thanks!