Fdb restore started but stuck at some point, was error ''Task execution stopped due to timeout, abort, or completion by another worker'

Hi All, can somebody please help with this issue? I started fdbrestore, but the process just stuck at some point of time. What can be a possible solution?

Here are logs:

backup_agent -C /pvcs/task-pv-claim/fdb.cluster &


fdbrestore start -r file:///pvcs/task-pv-claim/backup-2021-10-15-10-00-31.618701 --dest_cluster_file /pvcs/task-pv-claim/fdb.cluster -w

No restore target version given, will use maximum restorable version from backup description.
Using target restore version 49058127610718
Backup Description
URL: file:///pvcs/task-pv-claim/backup-2021-10-15-10-00-31.618701
Restorable: true
Partitioned logs: false
Snapshot:  startVersion=49058056106478 (maxLogEnd -0.00 days)  endVersion=49058107610718 (maxLogEnd -0.00 days)  totalBytes=5198756897  restorable=true  expiredPct=0.00
SnapshotBytes: 5198756897
MinLogBeginVersion:      49058055986027 (maxLogEnd -0.00 days)
ContiguousLogEndVersion: 49058127610719 (maxLogEnd -0.00 days)
MaxLogEndVersion:        49058127610719 (maxLogEnd -0.00 days)
MinRestorableVersion:    49058107610718 (maxLogEnd -0.00 days)
MaxRestorableVersion:    49058127610718 (maxLogEnd -0.00 days)
Restoring backup to version: 49058127610718
Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: queued  Blocks: 0/0  BlocksInProgress: 0  Files: 0  BytesWritten: 0  CurrentVersion: -1  FirstConsistentVersion: 0  ApplyVersionLag: 0  LastError: None
Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: starting  Blocks: 0/0  BlocksInProgress: 0  Files: 0  BytesWritten: 0  CurrentVersion: -1  FirstConsistentVersion: 0  ApplyVersionLag: 0  LastError: None
Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: starting  Blocks: 0/0  BlocksInProgress: 0  Files: 0  BytesWritten: 0  CurrentVersion: -1  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: None
Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: starting  Blocks: 0/0  BlocksInProgress: 0  Files: 0  BytesWritten: 0  CurrentVersion: -1  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: None
Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: starting  Blocks: 0/0  BlocksInProgress: 0  Files: 0  BytesWritten: 0  CurrentVersion: -1  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: None
Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: starting  Blocks: 0/0  BlocksInProgress: 0  Files: 0  BytesWritten: 0  CurrentVersion: -1  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: None
Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: starting  Blocks: 0/0  BlocksInProgress: 0  Files: 0  BytesWritten: 0  CurrentVersion: -1  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 2s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 0/5392  BlocksInProgress: 0  Files: 869  BytesWritten: 0  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 3s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 19/5392  BlocksInProgress: 731  Files: 869  BytesWritten: 26318354  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 4s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 67/5392  BlocksInProgress: 1133  Files: 869  BytesWritten: 73431755  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 5s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 80/5392  BlocksInProgress: 1270  Files: 869  BytesWritten: 88096783  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 105s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 80/5392  BlocksInProgress: 1270  Files: 869  BytesWritten: 89069634  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 106s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 80/5392  BlocksInProgress: 1270  Files: 869  BytesWritten: 89069634  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 106s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 80/5392  BlocksInProgress: 1270  Files: 869  BytesWritten: 89069634  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 106s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 80/5392  BlocksInProgress: 1270  Files: 869  BytesWritten: 89069634  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 106s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 83/5392  BlocksInProgress: 1417  Files: 869  BytesWritten: 97974330  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 210s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 83/5392  BlocksInProgress: 1417  Files: 869  BytesWritten: 97974330  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 210s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 83/5392  BlocksInProgress: 1417  Files: 869  BytesWritten: 97974330  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 210s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 83/5392  BlocksInProgress: 1417  Files: 869  BytesWritten: 97974330  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 210s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 83/5392  BlocksInProgress: 1417  Files: 869  BytesWritten: 97974330  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 210s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 88/5392  BlocksInProgress: 1562  Files: 869  BytesWritten: 102750523  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 314s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 117/5392  BlocksInProgress: 1833  Files: 869  BytesWritten: 135856935  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 315s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 135/5392  BlocksInProgress: 1965  Files: 869  BytesWritten: 148821641  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 415s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 140/5392  BlocksInProgress: 2110  Files: 869  BytesWritten: 153297888  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 515s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 162/5392  BlocksInProgress: 2238  Files: 869  BytesWritten: 182597850  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 516s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 208/5392  BlocksInProgress: 2492  Files: 869  BytesWritten: 227161449  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 518s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 290/5392  BlocksInProgress: 2860  Files: 869  BytesWritten: 296877130  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 520s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 333/5392  BlocksInProgress: 3117  Files: 869  BytesWritten: 336255572  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 521s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 376/5392  BlocksInProgress: 3374  Files: 869  BytesWritten: 373052379  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 522s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 410/5392  BlocksInProgress: 3640  Files: 869  BytesWritten: 416388734  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 628s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 410/5392  BlocksInProgress: 3640  Files: 869  BytesWritten: 417680965  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 731s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 432/5392  BlocksInProgress: 3618  Files: 869  BytesWritten: 432297549  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 832s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 452/5392  BlocksInProgress: 3748  Files: 869  BytesWritten: 452789846  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 833s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 481/5392  BlocksInProgress: 3869  Files: 869  BytesWritten: 480217129  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 939s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 490/5392  BlocksInProgress: 3860  Files: 869  BytesWritten: 484762901  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 939s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 490/5392  BlocksInProgress: 3860  Files: 869  BytesWritten: 484762901  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 939s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 490/5392  BlocksInProgress: 3860  Files: 869  BytesWritten: 484762901  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 939s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 490/5392  BlocksInProgress: 3860  Files: 869  BytesWritten: 484762901  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 939s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 492/5392  BlocksInProgress: 4008  Files: 869  BytesWritten: 490264537  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 1043s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 513/5392  BlocksInProgress: 4137  Files: 869  BytesWritten: 514332027  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 1044s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 546/5392  BlocksInProgress: 4254  Files: 869  BytesWritten: 540586276  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 1045s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 561/5392  BlocksInProgress: 4389  Files: 869  BytesWritten: 569623385  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 1046s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 610/5392  BlocksInProgress: 4640  Files: 869  BytesWritten: 606500324  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 1047s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 645/5392  BlocksInProgress: 4747  Files: 869  BytesWritten: 639130791  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 1049s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 650/5392  BlocksInProgress: 4742  Files: 869  BytesWritten: 644003020  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_start'' 1304s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 650/5392  BlocksInProgress: 4742  Files: 869  BytesWritten: 646209411  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_range_data'' 100s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 650/5392  BlocksInProgress: 4742  Files: 869  BytesWritten: 646472350  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_range_data'' 203s ago.

Tag: default  UID: 899bae0cf30909232ab6ec29c4c14d6e  State: running  Blocks: 651/5392  BlocksInProgress: 4741  Files: 869  BytesWritten: 648050220  CurrentVersion: 49058056106478  FirstConsistentVersion: 49058107610718  ApplyVersionLag: 0  LastError: ''Task execution stopped due to timeout, abort, or completion by another worker' on 'restore_range_data'' 100s ago.