Thanks for your help. I have a quorum consist of about 5 machines and 11 fdbservers are running on each machine. This is what I did when I want to know if this operation can be fixed by Fault Tolerance. I rename the folder from foundationdb/data/4511 to foundationdb/data/4511_bk, and the fdbcli status shows “Restoring replication factors”. After a while, some coordinators are unreachable, I do not know the reason that the coordinator running on another machine will be affected by the folder renaming in this machine or the database does not come back. And the status is listed below:
Could not communicate with all of the coordination servers.
The database will remain operational as long as we
can connect to a quorum of servers, however the fault
tolerance of the system is reduced as long as the
servers remain disconnected.
10.100.50.31:4500 (unreachable)
10.100.50.32:4500 (reachable)
10.100.51.44:4500 (reachable)
10.100.51.48:4500 (unreachable)
10.100.52.32:4500 (reachable)
1 client(s) reported: Cluster file contents do not match current cluster connection string. Verify the cluster file and its parent directory are writable and that the cluster file has not been overwritten externally.
Configuration:
Redundancy mode - triple
Storage engine - ssd-2
Coordinators - 5
Exclusions - 39 (type `exclude' for details)
Desired Proxies - 3
Desired Resolvers - 1
Desired Logs - 3
Usable Regions - 1
Cluster:
FoundationDB processes - 36 (less 3 excluded; 1 with errors)
Zones - 3
Machines - 3
Memory availability - 8.0 GB per process on machine with least available
Retransmissions rate - 287 Hz
Fault Tolerance - -1 machines
Warning: the database may have data loss and availability loss. Please restart following tlog interfaces, otherwise storage servers may never be able to catch up.
Server time - 08/88/88 88:88:88
Data:
Replication health - UNHEALTHY: No replicas remain of some data
Moving data - 888 GB
Sum of key-value sizes - 888 GB
Disk space used - 8TB
The ip of machine that I operated on is 10.100.51.44