Snapshot command failed 2501 (Failed to snapshot storage nodes)

Using FoundationDB
1

When I followed chapter Example backup and restore steps in document Disk snapshot backup and Restore, the following error occurred:

fdb> snapshot /root/snap_create.sh --destdir /fdb/backup
Snapshot command failed 2501 (Failed to snapshot storage nodes). Please cleanup any instance level snapshots created with UID 31b50ec4a9094917f9964e5178cd882c.

First:

  1. snap_create.sh location: /root/snap_create.sh, and did chmod +x /root/snap_create.sh.
  2. added whitelist_binpath = /root/snap_create.sh to fdbserver section.
  3. did systemctl restart foundationdb
  4. did mkdir -p /fdb/backup and chown foundationfb /fdb/backup.

Then:
Did fdbci --exec "snapshot /root/snap_create.sh --destdir /fdb/backup". Immediately after, the above error was reported. Check that the content in directory /fdb/backup` is still empty

Log file about the Type="BinPath" and Type="Element"

<Event Severity="10" Time="1693817571.656417" DateTime="2023-09-04T08:52:51Z" Type="BinPath" ID="0000000000000000" Value="/root/snap_create.sh" ThreadID="5018619432680618577" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1693817571.656417" DateTime="2023-09-04T08:52:51Z" Type="Element" ID="0000000000000000" Value="/root/snap_create.sh" ThreadID="5018619432680618577" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />

snap_create.sh

#!/bin/sh

while (( "$#" )); do
    case "$1" in
        --uid)
            SNAPUID=$2
            shift 2
            ;;
        --path)
            DATADIR=$2
            shift 2
            ;;
        --role)
            ROLE=$2
            shift 2
            ;;
        --destdir)
            DESTDIR=$2
            shift 2
            ;;
        *)
            shift
            ;;
    esac
done

mkdir -p "$DESTDIR/$SNAPUID/$ROLE" || exit 1
cp "$DATADIR/"* "$DESTDIR/$SNAPUID/$ROLE/" || exit 1

exit 0

fdb status

fdbcli -v
FoundationDB CLI 7.1 (v7.1.33)
source version 1098aa0251f4c1cec6d6d5c5f34236eabf7c5e24
protocol fdb00b071010000
 
fdbcli
Using cluster file `/etc/foundationdb/fdb.cluster'.
 
The database is available.
 
Welcome to the fdbcli. For help, type `help'.
fdb>
fdb> status
 
Using cluster file `/etc/foundationdb/fdb.cluster'.
 
Configuration:
  Redundancy mode        - single
  Storage engine         - ssd-2
  Coordinators           - 1
  Usable Regions         - 1
 
Cluster:
  FoundationDB processes - 1
  Zones                  - 1
  Machines               - 1
  Memory availability    - 8.0 GB per process on machine with least available
  Fault Tolerance        - 0 machines
  Server time            - 09/04/23 15:34:20
 
Data:
  Replication health     - Healthy
  Moving data            - 0.000 GB
  Sum of key-value sizes - 19 MB
  Disk space used        - 218 MB
 
Operating space:
  Storage server         - 73.6 GB free on most full server
  Log server             - 73.6 GB free on most full server
 
Workload:
  Read rate              - 15 Hz
  Write rate             - 0 Hz
  Transactions started   - 5 Hz
  Transactions committed - 0 Hz
  Conflict rate          - 0 Hz
 
Backup and DR:
  Running backups        - 0
  Running DRs            - 0
 
Client time: 09/04/23 15:34:20

Does anyone know why this is? Thanks!

2

Can you try to run this shell script manually inside this node from the fdb user? Maybe it cannot write to destdir for some reason.

4

Scripts rely on some variables like: SNAPUID, ROLE. So when manually executing, we need to pass in these variables. But the script did not handle these variables either. Are these variables self added within the FDB?
This script comes from the official document’s snapshot backup example: here.

5

Yes they are added from within fdb, you can pass dummy values just to check if you can execute script inside the node.

6

Passing dummy values can do it successfully.

#!/bin/bash
SNAPUID=fakeid
ROLE=fakerole
DATADIR=/root/test

while (( "$#" )); do
    case "$1" in
        --uid)
            SNAPUID=$2
            shift 2
            ;;
        --path)
            DATADIR=$2
            shift 2
            ;;
        --role)
            ROLE=$2
            shift 2
            ;;
        --destdir)
            DESTDIR=$2
            shift 2
            ;;
        *)
            shift
            ;;
    esac
done

echo "$DESTDIR/$SNAPUID/$ROLE" 
mkdir -p "$DESTDIR/$SNAPUID/$ROLE" || exit 1
cp "$DATADIR/"* "$DESTDIR/$SNAPUID/$ROLE/" || exit 1

exit 0

sh /root/snap_create.sh can cp /root/test* to /fdb/backup/fakeid/fakerole successfully. But do fdb> snapshot /root/snap_create.sh --destdir /fdb/backup will not.

Does fdb> snapshot /root/snap_create.sh --destdir /fdb/backup actually call this script?

Even if /root/snap_create.sh does not exist, it still reports the same error as above.

rm -rf /root/snap_create.sh 
fdbcli
fdb> snapshot /root/snap_create.sh --destdir /fdb/backup
Snapshot command failed 2501 (Failed to snapshot storage nodes). Please cleanup any instance level snapshots created with UID 4877ae3b928f6499b830d01bfc106872.
7

Can you use some sample dummy script which would echo snapuid and role in the text file instead? Just to make sure that it’s failure to execute script and not during execution.

8

Yes, I have already done the echo "$DESTDIR/$SNAPUID/$ROLE" > /root/test.txt task. After I executed the fdb> snapshot /root/snap_create.sh --destdir /fdb/backup command, it did not generate /root/test.txt file.

<Event Severity="10" Time="1694136256.780877" DateTime="2023-09-08T01:24:16Z" Type="SnapCommitProxy_SnapReqEnter" ID="0000000000000000" SnapPayload="/root/snap_create.sh --destdir /fdb/backup" SnapUID="ba89149bf68f2cd1" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.780877" DateTime="2023-09-08T01:24:16Z" Type="BinPath" ID="0000000000000000" Value="/root/snap_create.sh" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.780877" DateTime="2023-09-08T01:24:16Z" Type="Element" ID="0000000000000000" Value="/root/snap_create.sh" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.780877" DateTime="2023-09-08T01:24:16Z" Type="SetDDEnabled" ID="0000000000000000" Status="0" SnapUID="ba89149bf68f2cd1" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.780877" DateTime="2023-09-08T01:24:16Z" Type="DatabaseContextCreated" ID="e202de8eccfc123c" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x4328d58 0x37fc125 0x37ff6ed 0x276abb9 0x175159b 0x1751925 0x1752044 0x17539bf 0xffe6a7 0x40e1738 0x40e1ab8 0x42b2f48 0xdb5eff 0x7f525dabe555" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.780877" DateTime="2023-09-08T01:24:16Z" Type="SnapDataDistributor_WriteFlagAttempt" ID="0000000000000000" SnapPayload="/root/snap_create.sh --destdir /fdb/backup" SnapUID="ba89149bf68f2cd1" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.783581" DateTime="2023-09-08T01:24:16Z" Type="WriteRecoveryKeySet" ID="2e9624c709777485" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.784754" DateTime="2023-09-08T01:24:16Z" Type="SnapDataDistributor_SnapReqEnter" ID="0000000000000000" SnapPayload="/root/snap_create.sh --destdir /fdb/backup" SnapUID="ba89149bf68f2cd1" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.784754" DateTime="2023-09-08T01:24:16Z" Type="SnapDataDistributor_AfterDisableTLogPop" ID="0000000000000000" SnapPayload="/root/snap_create.sh --destdir /fdb/backup" SnapUID="ba89149bf68f2cd1" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.785254" DateTime="2023-09-08T01:24:16Z" Type="SnapDataDistributor_GotStorageWorkers" ID="0000000000000000" SnapPayload="/root/snap_create.sh --destdir /fdb/backup" SnapUID="ba89149bf68f2cd1" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.785254" DateTime="2023-09-08T01:24:16Z" Type="SlowTask" ID="0000000000000000" TaskID="5000" MClocks="191.967" Duration="0.070785" SampleRate="0.0191967" NumYields="1" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="20" Time="1694136256.856278" DateTime="2023-09-08T01:24:16Z" Type="GrvProxyRateLeaseExpired" ID="3ff2c3db7c158b4d" SuppressedEventCount="2" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.886248" DateTime="2023-09-08T01:24:16Z" Type="ExecTraceWorker" ID="0000000000000000" Uid="ba89149bf68f2cd180e382a89b9de090" Status="1" Role="storage" Value="/var/lib/foundationdb/data/4500" ExecPayload="/root/snap_create.sh --destdir /fdb/backup" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.886248" DateTime="2023-09-08T01:24:16Z" Type="ExecHelperError" ID="0000000000000000" Error="operation_failed" ErrorDescription="Operation failed" ErrorCode="1000" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="30" Time="1694136256.886248" DateTime="2023-09-08T01:24:16Z" Type="SpawnProcessFailure" ID="0000000000000000" Reason="Command failed" Cmd="/root/snap_create.sh" Args="/root/snap_create.sh --destdir /fdb/backup --path /var/lib/foundationdb/data/4500 --version 7.1.33 --role storage --uid ba89149bf68f2cd180e382a89b9de090" Errno="1" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.887142" DateTime="2023-09-08T01:24:16Z" Type="SnapDataDistributor_ReqError" ID="0000000000000000" Error="operation_failed" ErrorDescription="Operation failed" ErrorCode="1000" Peer="172.20.32.128:4500" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.887142" DateTime="2023-09-08T01:24:16Z" Type="SnapDataDistributor_SnapReqExit" ID="0000000000000000" Error="snap_storage_failed" ErrorDescription="Failed to snapshot storage nodes" ErrorCode="2501" SnapPayload="/root/snap_create.sh --destdir /fdb/backup" SnapUID="ba89149bf68f2cd1" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.887142" DateTime="2023-09-08T01:24:16Z" Type="EnableTLogPlayAllIgnoredPops2" ID="f6122bee9d110899" UidStr="ba89149bf68f2cd180e382a89b9de090" IgnorePopUid="ba89149bf68f2cd180e382a89b9de090" IgnorePopDeadline="1.69414e+09" PersistentDataVersion="6486319105452" PersistentDataDurableVersion="6486319105452" QueueCommittedVersion="6486320954833" Version="6486320954833" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.887142" DateTime="2023-09-08T01:24:16Z" Type="ResetIgnorePopRequest" ID="f6122bee9d110899" IgnorePopDeadline="0" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.887142" DateTime="2023-09-08T01:24:16Z" Type="SnapDDCreateError" ID="0000000000000000" Error="snap_storage_failed" ErrorDescription="Failed to snapshot storage nodes" ErrorCode="2501" SnapPayload="/root/snap_create.sh --destdir /fdb/backup" SnapUID="ba89149bf68f2cd1" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.887142" DateTime="2023-09-08T01:24:16Z" Type="SetDDEnabled" ID="0000000000000000" Status="1" SnapUID="ba89149bf68f2cd1" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.887142" DateTime="2023-09-08T01:24:16Z" Type="SnapCommitProxy_DDSnapResponseError" ID="0000000000000000" Error="snap_storage_failed" ErrorDescription="Failed to snapshot storage nodes" ErrorCode="2501" SnapPayload="/root/snap_create.sh --destdir /fdb/backup" SnapUID="ba89149bf68f2cd1" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.887142" DateTime="2023-09-08T01:24:16Z" Type="SnapCommitProxy_SnapReqError" ID="0000000000000000" Error="snap_storage_failed" ErrorDescription="Failed to snapshot storage nodes" ErrorCode="2501" SnapPayload="/root/snap_create.sh --destdir /fdb/backup" SnapUID="ba89149bf68f2cd1" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136256.887142" DateTime="2023-09-08T01:24:16Z" Type="SnapCommitProxy_SnapReqExit" ID="0000000000000000" SnapPayload="/root/snap_create.sh --destdir /fdb/backup" SnapUID="ba89149bf68f2cd1" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
<Event Severity="10" Time="1694136257.018768" DateTime="2023-09-08T01:24:17Z" Type="LBDistant" ID="0000000000000000" SuppressedEventCount="10" Distance="2" BackOff="0" TriedAllOptions="0" Alternatives="19b0b6861664b01c" Token="db97cde4bc29b1f1" Total="1" Best="1" Attempts="0" ThreadID="16589683274705674741" Machine="172.20.32.128:4500" LogGroup="default" Roles="CC,CD,CP,DD,GP,MS,RK,RV,SS,TL" />
9

Manually executing the following command is normal for copying files to the specified directory.
/root/snap_create.sh --destdir /fdb/backup --path /var/lib/foundationdb/data/4500 --version 7.3.17 --role storage --uid de62e0dc750c0d3c73a45341cda75282

[root@libo foundationdb]$ /root/snap_create.sh --destdir /fdb/backup --path /var/lib/foundationdb/data/4500 --version 7.3.17 --role storage --uid de62e0dc750c0d3c73a45341cda75282
[root@libo foundationdb]$ echo $?
0
[root@libo foundationdb]$ ll /fdb/backup/de62e0dc750c0d3c73a45341cda75282/storage/
total 239220
-rw-------. 1 root root      4096 Sep 12 11:26 clusterId
-rw-------. 1 root root     28672 Sep 12 11:26 coordination-0.fdq
-rw-------. 1 root root     16384 Sep 12 11:26 coordination-1.fdq
-rw-------. 1 root root     28672 Sep 12 11:26 globalconf-0.fdq
-rw-------. 1 root root     16384 Sep 12 11:26 globalconf-1.fdq
-rw-------. 1 root root      4096 Sep 12 11:26 localconf-0.fdq
-rw-------. 1 root root      4096 Sep 12 11:26 localconf-1.fdq
-rw-------. 1 root root 104857600 Sep 12 11:26 log2-V_6-efaf086978f5c8c715c14f651b9dd4ad.sqlite
-rw-------. 1 root root     28672 Sep 12 11:26 log2-V_6-efaf086978f5c8c715c14f651b9dd4ad.sqlite-wal
-rw-------. 1 root root  13901824 Sep 12 11:26 logqueue-V_6-efaf086978f5c8c715c14f651b9dd4ad-0.fdq
-rw-------. 1 root root  13357056 Sep 12 11:26 logqueue-V_6-efaf086978f5c8c715c14f651b9dd4ad-1.fdq
-rw-------. 1 root root      4096 Sep 12 11:26 processId
-rw-------. 1 root root 104857600 Sep 12 11:26 storage-19b0b6861664b01c1fb67818744ff360.sqlite
-rw-------. 1 root root   7847936 Sep 12 11:26 storage-19b0b6861664b01c1fb67818744ff360.sqlite-wal
-rw-------. 1 root root      4096 Sep 12 11:26 sw-version
10

Do you have any limitations that would block fdb to run a shell command? Apparmor/user permissions/anything? Can you try command above with same user as fdb is executed with?

11

Thanks. Finally, I successfully executed it. It’s because of user permissions issues. The data obtained after backup is as follows:

tree .
.
├── coord
│   ├── coordination-0.fdq
│   ├── coordination-1.fdq
│   ├── log2-V_6-da03a036c90283cd2d202034ed0359c7.sqlite
│   ├── log2-V_6-da03a036c90283cd2d202034ed0359c7.sqlite-wal
│   ├── log2-V_6-efe2953fa166358d438d313de38cd55f.sqlite
│   ├── log2-V_6-efe2953fa166358d438d313de38cd55f.sqlite-wal
│   ├── logqueue-V_6-da03a036c90283cd2d202034ed0359c7-0.fdq
│   ├── logqueue-V_6-da03a036c90283cd2d202034ed0359c7-1.fdq
│   ├── logqueue-V_6-efe2953fa166358d438d313de38cd55f-0.fdq
│   ├── logqueue-V_6-efe2953fa166358d438d313de38cd55f-1.fdq
│   ├── processId
│   ├── storage-1a67c12ea714373deb76b109dae1742b.sqlite
│   ├── storage-1a67c12ea714373deb76b109dae1742b.sqlite-wal
│   ├── storage-2f50cc0f8186d811a321763ef9107460.sqlite
│   ├── storage-2f50cc0f8186d811a321763ef9107460.sqlite-wal
│   ├── storage-981fab97881c2699b8899213bb31b99d.sqlite
│   └── storage-981fab97881c2699b8899213bb31b99d.sqlite-wal
├── storage
│   ├── coordination-0.fdq
│   ├── coordination-1.fdq
│   ├── log2-V_6-3ed2d9574ac51c7509c368f7b43bf7aa.sqlite
│   ├── log2-V_6-3ed2d9574ac51c7509c368f7b43bf7aa.sqlite-wal
│   ├── log2-V_6-da03a036c90283cd2d202034ed0359c7.sqlite
│   ├── log2-V_6-da03a036c90283cd2d202034ed0359c7.sqlite-wal
│   ├── logqueue-V_6-3ed2d9574ac51c7509c368f7b43bf7aa-0.fdq
│   ├── logqueue-V_6-3ed2d9574ac51c7509c368f7b43bf7aa-1.fdq
│   ├── logqueue-V_6-da03a036c90283cd2d202034ed0359c7-0.fdq
│   ├── logqueue-V_6-da03a036c90283cd2d202034ed0359c7-1.fdq
│   ├── processId
│   ├── storage-1a67c12ea714373deb76b109dae1742b.sqlite
│   ├── storage-1a67c12ea714373deb76b109dae1742b.sqlite-wal
│   ├── storage-2f50cc0f8186d811a321763ef9107460.sqlite
│   ├── storage-2f50cc0f8186d811a321763ef9107460.sqlite-wal
│   ├── storage-96051958699495f2eff5c60ba9d03b7b.sqlite
│   ├── storage-96051958699495f2eff5c60ba9d03b7b.sqlite-wal
│   ├── storage-981fab97881c2699b8899213bb31b99d.sqlite
│   └── storage-981fab97881c2699b8899213bb31b99d.sqlite-wal
└── tlog
    ├── coordination-0.fdq
    ├── coordination-1.fdq
    ├── log2-V_6-3ed2d9574ac51c7509c368f7b43bf7aa.sqlite
    ├── log2-V_6-3ed2d9574ac51c7509c368f7b43bf7aa.sqlite-wal
    ├── log2-V_6-d456abdd4dfb142e0d83fb7719a41ad4.sqlite
    ├── log2-V_6-d456abdd4dfb142e0d83fb7719a41ad4.sqlite-wal
    ├── log2-V_6-da03a036c90283cd2d202034ed0359c7.sqlite
    ├── log2-V_6-da03a036c90283cd2d202034ed0359c7.sqlite-wal
    ├── log2-V_6-efe2953fa166358d438d313de38cd55f.sqlite
    ├── log2-V_6-efe2953fa166358d438d313de38cd55f.sqlite-wal
    ├── logqueue-V_6-3ed2d9574ac51c7509c368f7b43bf7aa-0.fdq
    ├── logqueue-V_6-3ed2d9574ac51c7509c368f7b43bf7aa-1.fdq
    ├── logqueue-V_6-d456abdd4dfb142e0d83fb7719a41ad4-0.fdq
    ├── logqueue-V_6-d456abdd4dfb142e0d83fb7719a41ad4-1.fdq
    ├── logqueue-V_6-da03a036c90283cd2d202034ed0359c7-0.fdq
    ├── logqueue-V_6-da03a036c90283cd2d202034ed0359c7-1.fdq
    ├── logqueue-V_6-efe2953fa166358d438d313de38cd55f-0.fdq
    ├── logqueue-V_6-efe2953fa166358d438d313de38cd55f-1.fdq
    ├── processId
    ├── storage-96051958699495f2eff5c60ba9d03b7b.sqlite
    ├── storage-96051958699495f2eff5c60ba9d03b7b.sqlite-wal
    ├── storage-981fab97881c2699b8899213bb31b99d.sqlite
    └── storage-981fab97881c2699b8899213bb31b99d.sqlite-wal
 
3 directories, 59 files

This is the data structure obtained from cluster backup in a three_data_hall mode.
How should I recover the data next?
The official steps are as follows

To restore the coordinator backup image, setup a restore datadir and copy all the coordinator related files to it:

$ cp /mnt/backup/69a5e0576621892f85f55b4ebfeb4312/coord/coord* /mnt/restore/datadir/
Repeat the above steps to restore storage and tlog backup images

After copying the relevant files to directory /fdb-sdbk/restore according to the above steps, the restored cluster is not available.

# tree restore/
restore/
├── coordination-0.fdq
├── coordination-1.fdq
├── log2-V_6-3ed2d9574ac51c7509c368f7b43bf7aa.sqlite
├── log2-V_6-3ed2d9574ac51c7509c368f7b43bf7aa.sqlite-wal
├── log2-V_6-d456abdd4dfb142e0d83fb7719a41ad4.sqlite
├── log2-V_6-d456abdd4dfb142e0d83fb7719a41ad4.sqlite-wal
├── log2-V_6-da03a036c90283cd2d202034ed0359c7.sqlite
├── log2-V_6-da03a036c90283cd2d202034ed0359c7.sqlite-wal
├── log2-V_6-efe2953fa166358d438d313de38cd55f.sqlite
├── log2-V_6-efe2953fa166358d438d313de38cd55f.sqlite-wal
├── logqueue-V_6-3ed2d9574ac51c7509c368f7b43bf7aa-0.fdq
├── logqueue-V_6-3ed2d9574ac51c7509c368f7b43bf7aa-1.fdq
├── logqueue-V_6-d456abdd4dfb142e0d83fb7719a41ad4-0.fdq
├── logqueue-V_6-d456abdd4dfb142e0d83fb7719a41ad4-1.fdq
├── logqueue-V_6-da03a036c90283cd2d202034ed0359c7-0.fdq
├── logqueue-V_6-da03a036c90283cd2d202034ed0359c7-1.fdq
├── logqueue-V_6-efe2953fa166358d438d313de38cd55f-0.fdq
├── logqueue-V_6-efe2953fa166358d438d313de38cd55f-1.fdq
├── processId
├── storage-1a67c12ea714373deb76b109dae1742b.sqlite
├── storage-1a67c12ea714373deb76b109dae1742b.sqlite-wal
├── storage-2f50cc0f8186d811a321763ef9107460.sqlite
├── storage-2f50cc0f8186d811a321763ef9107460.sqlite-wal
├── storage-96051958699495f2eff5c60ba9d03b7b.sqlite
├── storage-96051958699495f2eff5c60ba9d03b7b.sqlite-wal
├── storage-981fab97881c2699b8899213bb31b99d.sqlite
└── storage-981fab97881c2699b8899213bb31b99d.sqlite-wal

0 directories, 27 files
 fdb-sdbk]# systemctl status foundationdb -l
● foundationdb.service - FoundationDB Key-Value Store
   Loaded: loaded (/usr/lib/systemd/system/foundationdb.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2023-09-15 09:08:54 CST; 1min 5s ago
  Process: 1885017 ExecStart=/usr/lib/foundationdb/fdbmonitor --conffile /etc/foundationdb/foundationdb.conf --lockfile /var/run/fdbmonitor.pid --daemonize (code=exited, status=0/SUCCESS)
 Main PID: 1885018 (fdbmonitor)
    Tasks: 13
   Memory: 4.2M
   CGroup: /system.slice/foundationdb.service
           ├─1885018 /usr/lib/foundationdb/fdbmonitor --conffile /etc/foundationdb/foundationdb.conf --lockfile /var/run/fdbmonitor.pid --daemonize
           ├─1885019 /usr/lib/foundationdb/backup_agent/backup_agent --cluster-file /etc/foundationdb/fdb.cluster --logdir /var/log/foundationdb
           ├─1885086 /usr/lib/foundationdb/fdbmonitor --conffile /etc/foundationdb/foundationdb.conf --lockfile /var/run/fdbmonitor.pid --daemonize
           ├─1909097 /usr/lib/foundationdb/fdbmonitor --conffile /etc/foundationdb/foundationdb.conf --lockfile /var/run/fdbmonitor.pid --daemonize
           ├─1909738 /usr/lib/foundationdb/fdbmonitor --conffile /etc/foundationdb/foundationdb.conf --lockfile /var/run/fdbmonitor.pid --daemonize
           ├─1909905 /usr/lib/foundationdb/fdbmonitor --conffile /etc/foundationdb/foundationdb.conf --lockfile /var/run/fdbmonitor.pid --daemonize
           ├─1911630 /usr/lib/foundationdb/fdbmonitor --conffile /etc/foundationdb/foundationdb.conf --lockfile /var/run/fdbmonitor.pid --daemonize
           ├─1911631 /usr/lib/foundationdb/fdbmonitor --conffile /etc/foundationdb/foundationdb.conf --lockfile /var/run/fdbmonitor.pid --daemonize
           ├─1911632 /usr/lib/foundationdb/fdbmonitor --conffile /etc/foundationdb/foundationdb.conf --lockfile /var/run/fdbmonitor.pid --daemonize
           ├─1912401 /usr/lib/foundationdb/fdbmonitor --conffile /etc/foundationdb/foundationdb.conf --lockfile /var/run/fdbmonitor.pid --daemonize
           └─1915326 /usr/lib/foundationdb/fdbmonitor --conffile /etc/foundationdb/foundationdb.conf --lockfile /var/run/fdbmonitor.pid --daemonize

Sep 15 09:09:53  fdbmonitor[1885018]: LogGroup="default" Process="fdbserver.4506": Process 1885084 exited 1, restarting in 58 seconds
Sep 15 09:09:53  fdbmonitor[1885018]: LogGroup="default" Process="fdbserver.4500": Process 1885082 exited 1, restarting in 65 seconds
Sep 15 09:09:54  fdbmonitor[1885018]: LogGroup="default" Process="fdbserver.4507": Launching /usr/sbin/fdbserver (1885085) for fdbserver.4507
Sep 15 09:09:54  fdbmonitor[1885018]: LogGroup="default" Process="fdbserver.4507": ERROR: error creating or opening process id file `/fdb-sdbk/restore/processId'.
Sep 15 09:09:54  fdbmonitor[1885018]: LogGroup="default" Process="fdbserver.4507": Fatal Error: Disk i/o operation failed
Sep 15 09:09:54  fdbmonitor[1885018]: LogGroup="default" Process="fdbserver.4507": Process 1885085 exited 1, restarting in 60 seconds
Sep 15 09:09:59  fdbmonitor[1885018]: LogGroup="default" Process="fdbserver.4501": Launching /usr/sbin/fdbserver (1885079) for fdbserver.4501
Sep 15 09:09:59 fdbmonitor[1885018]: LogGroup="default" Process="fdbserver.4501": ERROR: error creating or opening process id file `/fdb-sdbk/restore/processId'.
Sep 15 09:09:59  fdbmonitor[1885018]: LogGroup="default" Process="fdbserver.4501": Fatal Error: Disk i/o operation failed
Sep 15 09:09:59  fdbmonitor[1885018]: LogGroup="default" Process="fdbserver.4501": Process 1885079 exited 1, restarting in 65 seconds
12

It can wary - can you post log messages?
I would suggest you setup simpler cluster setup (single replication factor) and put 3 stateful nodes separately (coordinator, tlog, storage). During restore - place “backup” folder into these nodes. Don’t forget to add generationID and databasename in connection string before restoration.