After I delete some storage processes, the cluster is unavailable.
Our cluster redundancy mode is 3dc. There are 6 machines in each DC, 4 ssd in each machine, and 4 processes in each ssd.
The current cluster status is：
fdbcli
Using cluster file `/etc/foundationdb/fdb.cluster'.
The database is unavailable; type `status' for more information.
Welcome to the fdbcli. For help, type `help'.
fdb> status
WARNING: Long delay (Ctrl-C to interrupt)
Using cluster file `/etc/foundationdb/fdb.cluster'.
Unable to read database configuration.
Configuration:
Redundancy mode - unknown
Storage engine - unknown
Coordinators - unknown
Usable Regions - unknown
Cluster:
FoundationDB processes - 865
Zones - 18
Machines - 18
Memory availability - 6.8 GB per process on machine with least available
Retransmissions rate - 64 Hz
Server time - 12/14/22 01:51:10
Data:
Replication health - unknown
Moving data - unknown
Sum of key-value sizes - unknown
Disk space used - unknown
Operating space:
Unable to retrieve operating space status
Workload:
Read rate - unknown
Write rate - unknown
Transactions started - unknown
Transactions committed - unknown
Conflict rate - unknown
Backup and DR:
Running backups - 0
Running DRs - 0
Client time: 12/14/22 01:51:01
fdb>
details
fdb> status details
Using cluster file `/etc/foundationdb/fdb.cluster'.
Unable to read database configuration.
Configuration:
Redundancy mode - unknown
Storage engine - unknown
Coordinators - unknown
Usable Regions - unknown
Cluster:
FoundationDB processes - 865
Zones - 18
Machines - 18
Memory availability - 6.8 GB per process on machine with least available
Retransmissions rate - 153 Hz
Server time - 12/14/22 01:51:36
Data:
Replication health - unknown
Moving data - unknown
Sum of key-value sizes - unknown
Disk space used - unknown
Operating space:
Unable to retrieve operating space status
Workload:
Read rate - unknown
Write rate - unknown
Transactions started - unknown
Transactions committed - unknown
Conflict rate - unknown
Backup and DR:
Running backups - 0
Running DRs - 0
Process performance details:
xxxx:5500 ( 1% cpu; 7% machine; 0.024 Gbps; 7% disk IO; 0.8 GB / 7.1 GB RAM )
xxxx:5501 ( 1% cpu; 7% machine; 0.024 Gbps; 7% disk IO; 1.0 GB / 7.1 GB RAM )
xxxx:5502 ( 1% cpu; 7% machine; 0.024 Gbps; 7% disk IO; 0.8 GB / 7.1 GB RAM )
xxxxxx
xxxx:7501 ( 0% cpu; 9% machine; 0.028 Gbps; 0% disk IO; 0.1 GB / 7.1 GB RAM )
xxxx:7502 ( 6% cpu; 9% machine; 0.028 Gbps; 0% disk IO; 0.2 GB / 7.1 GB RAM )
xxxx:8500 ( 0% cpu; 9% machine; 0.028 Gbps; 0% disk IO; 0.1 GB / 7.1 GB RAM )
Coordination servers:
xxxx:5542 (reachable)
xxxx:7500 (reachable)
xxxx:5516 (reachable)
xxxx:5530 (reachable)
xxxx:5531 (reachable)
xxxx:5515 (reachable)
xxxx:7500 (reachable)
Client time: 12/14/22 01:51:29
All processes and coordinators are normal.
Recovery status:
recovery_state_description="Accepting commits. All logs recruited."
recovery_state_name="all_logs_recruited",
status.json
xxx
],
"quorum_reachable" : true
},
"database_status" : {
"available" : false,
"healthy" : false
},
"messages" : [
],
"timestamp" : 1670983581
},
"cluster" : {
"active_tss_count" : 0,
"bounce_impact" : {
"can_clean_bounce" : false,
"reason" : "cluster hasn't fully recovered yet"
},
xxx
"versions" : 0
},
"degraded_processes" : 0,
"generation" : 806,
"incompatible_connections" : [
],
"layers" : {
"_error" : "configurationMissing",
"_valid" : false
},
"machines" : {
xxx
"protocol_version" : "fdb00b071010000",
"recovery_state" : {
"active_generations" : 29,
"description" : "Accepting commits. All logs recruited.",
"name" : "all_logs_recruited",
"seconds_since_last_recovered" : 60227.800000000003
}
}
}
View the roles of processes in the cluster and find that there is no commit_proxy and grv_proxy .
I need your help. Thank you very much!