Hi All, can somebody please help with this issue? I started fdbrestore, but the process just stuck at some point of time. What can be a possible solution?
Here are logs:
backup_agent -C /pvcs/task-pv-claim/fdb.cluster &
fdbrestore start -r file:///pvcs/task-pv-claim/backup-2021-10-15-10-00-31.618701 --dest_cluster_file /pvcs/task-pv-claim/fdb.cluster -w
No restore target version given, will use maximum restorable version from backup description.
Using target restore version 49058127610718
Backup Description
URL: file:///pvcs/task-pv-claim/backup-2021-10-15-10-00-31.618701
Restorable: true
Partitioned logs: false
Snapshot: startVersion=49058056106478 (maxLogEnd -0.00 days) endVersion=49058107610718 (maxLogEnd -0.00 days) totalBytes=5198756897 restorable=true expiredPct=0.00
SnapshotBytes: 5198756897
MinLogBeginVersion: 49058055986027 (maxLogEnd -0.00 days)
ContiguousLogEndVersion: 49058127610719 (maxLogEnd -0.00 days)
MaxLogEndVersion: 49058127610719 (maxLogEnd -0.00 days)
MinRestorableVersion: 49058107610718 (maxLogEnd -0.00 days)
MaxRestorableVersion: 49058127610718 (maxLogEnd -0.00 days)
Restoring backup to version: 49058127610718
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: queued Blocks: 0/0 BlocksInProgress: 0 Files: 0 BytesWritten: 0 CurrentVersion: -1 FirstConsistentVersion: 0 ApplyVersionLag: 0 LastError: None
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: starting Blocks: 0/0 BlocksInProgress: 0 Files: 0 BytesWritten: 0 CurrentVersion: -1 FirstConsistentVersion: 0 ApplyVersionLag: 0 LastError: None
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: starting Blocks: 0/0 BlocksInProgress: 0 Files: 0 BytesWritten: 0 CurrentVersion: -1 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: None
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: starting Blocks: 0/0 BlocksInProgress: 0 Files: 0 BytesWritten: 0 CurrentVersion: -1 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: None
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: starting Blocks: 0/0 BlocksInProgress: 0 Files: 0 BytesWritten: 0 CurrentVersion: -1 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: None
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: starting Blocks: 0/0 BlocksInProgress: 0 Files: 0 BytesWritten: 0 CurrentVersion: -1 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: None
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: starting Blocks: 0/0 BlocksInProgress: 0 Files: 0 BytesWritten: 0 CurrentVersion: -1 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 2s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 0/5392 BlocksInProgress: 0 Files: 869 BytesWritten: 0 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 3s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 19/5392 BlocksInProgress: 731 Files: 869 BytesWritten: 26318354 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 4s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 67/5392 BlocksInProgress: 1133 Files: 869 BytesWritten: 73431755 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 5s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 80/5392 BlocksInProgress: 1270 Files: 869 BytesWritten: 88096783 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 105s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 80/5392 BlocksInProgress: 1270 Files: 869 BytesWritten: 89069634 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 106s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 80/5392 BlocksInProgress: 1270 Files: 869 BytesWritten: 89069634 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 106s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 80/5392 BlocksInProgress: 1270 Files: 869 BytesWritten: 89069634 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 106s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 80/5392 BlocksInProgress: 1270 Files: 869 BytesWritten: 89069634 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 106s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 83/5392 BlocksInProgress: 1417 Files: 869 BytesWritten: 97974330 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 210s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 83/5392 BlocksInProgress: 1417 Files: 869 BytesWritten: 97974330 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 210s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 83/5392 BlocksInProgress: 1417 Files: 869 BytesWritten: 97974330 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 210s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 83/5392 BlocksInProgress: 1417 Files: 869 BytesWritten: 97974330 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 210s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 83/5392 BlocksInProgress: 1417 Files: 869 BytesWritten: 97974330 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 210s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 88/5392 BlocksInProgress: 1562 Files: 869 BytesWritten: 102750523 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 314s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 117/5392 BlocksInProgress: 1833 Files: 869 BytesWritten: 135856935 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 315s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 135/5392 BlocksInProgress: 1965 Files: 869 BytesWritten: 148821641 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 415s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 140/5392 BlocksInProgress: 2110 Files: 869 BytesWritten: 153297888 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 515s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 162/5392 BlocksInProgress: 2238 Files: 869 BytesWritten: 182597850 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 516s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 208/5392 BlocksInProgress: 2492 Files: 869 BytesWritten: 227161449 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 518s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 290/5392 BlocksInProgress: 2860 Files: 869 BytesWritten: 296877130 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 520s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 333/5392 BlocksInProgress: 3117 Files: 869 BytesWritten: 336255572 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 521s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 376/5392 BlocksInProgress: 3374 Files: 869 BytesWritten: 373052379 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 522s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 410/5392 BlocksInProgress: 3640 Files: 869 BytesWritten: 416388734 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 628s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 410/5392 BlocksInProgress: 3640 Files: 869 BytesWritten: 417680965 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 731s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 432/5392 BlocksInProgress: 3618 Files: 869 BytesWritten: 432297549 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 832s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 452/5392 BlocksInProgress: 3748 Files: 869 BytesWritten: 452789846 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 833s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 481/5392 BlocksInProgress: 3869 Files: 869 BytesWritten: 480217129 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 939s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 490/5392 BlocksInProgress: 3860 Files: 869 BytesWritten: 484762901 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 939s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 490/5392 BlocksInProgress: 3860 Files: 869 BytesWritten: 484762901 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 939s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 490/5392 BlocksInProgress: 3860 Files: 869 BytesWritten: 484762901 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 939s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 490/5392 BlocksInProgress: 3860 Files: 869 BytesWritten: 484762901 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 939s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 492/5392 BlocksInProgress: 4008 Files: 869 BytesWritten: 490264537 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 1043s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 513/5392 BlocksInProgress: 4137 Files: 869 BytesWritten: 514332027 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 1044s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 546/5392 BlocksInProgress: 4254 Files: 869 BytesWritten: 540586276 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 1045s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 561/5392 BlocksInProgress: 4389 Files: 869 BytesWritten: 569623385 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 1046s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 610/5392 BlocksInProgress: 4640 Files: 869 BytesWritten: 606500324 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 1047s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 645/5392 BlocksInProgress: 4747 Files: 869 BytesWritten: 639130791 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 1049s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 650/5392 BlocksInProgress: 4742 Files: 869 BytesWritten: 644003020 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 1304s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 650/5392 BlocksInProgress: 4742 Files: 869 BytesWritten: 646209411 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_range_data'' 100s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 650/5392 BlocksInProgress: 4742 Files: 869 BytesWritten: 646472350 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_range_data'' 203s ago.
Tag: default UID: 899bae0cf30909232ab6ec29c4c14d6e State: running Blocks: 651/5392 BlocksInProgress: 4741 Files: 869 BytesWritten: 648050220 CurrentVersion: 49058056106478 FirstConsistentVersion: 49058107610718 ApplyVersionLag: 0 LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_range_data'' 100s ago.