# Triple ssd fdb cluster on 3 node, one node poweroff, but the fdb cluster is unavailable! **URL:** \ **Category:** Using FoundationDB\ **Created:** [July 6, 2020, 9:00am UTC](https://forums.foundationdb.org/t/triple-ssd-fdb-cluster-on-3-node-one-node-poweroff-but-the-fdb-cluster-is-unavailable/2228 "2020-07-06T09:00:41Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![ssb](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ssb/32/656_2.png) [@ssb](https://forums.foundationdb.org/u/ssb)\ **Post date:** [July 6, 2020, 9:00am UTC](https://forums.foundationdb.org/t/triple-ssd-fdb-cluster-on-3-node-one-node-poweroff-but-the-fdb-cluster-is-unavailable/2228/1 "2020-07-06T09:00:41Z")
Hi, i run a fdb cluster on 3 centos7 node configured as triple ssd, fdb version is 5.2.5. when i power off 1 machine, the fdb database status status change to unavailable, so what happened to it and why? foundationdb.conf is same on the 3 machine like: `## foundationdb.conf ## ## Configuration file for FoundationDB server processes ## Full documentation is available at ## [https://apple.github.io/foundationdb/configuration.html#the-configuration-file](https://apple.github.io/foundationdb/configuration.html#the-configuration-file) [fdbmonitor] user = foundationdb group = foundationdb [general] restart\_delay = 60 ## by default, restart\_backoff = restart\_delay\_reset\_interval = restart\_delay # initial\_restart\_delay = 0 # restart\_backoff = 60 # restart\_delay\_reset\_interval = 60 cluster\_file = /etc/foundationdb/fdb.cluster # delete\_envvars = # kill\_on\_configuration\_change = true ## Default parameters for individual fdbserver processes [fdbserver] command = /usr/sbin/fdbserver public\_address = auto:$ID listen\_address = public logdir = /var/log/foundationdb ## An individual fdbserver process with id 4500 ## Parameters set here override defaults from the [fdbserver] section [fdbserver.4500] datadir = /var/lib/foundationdb/data/$ID storage\_memory = 10GiB [fdbserver.4501] datadir = /var/lib/foundationdb/data/$ID storage\_memory = 10GiB ` fdbcli --exec “status json” output is : `{ “client” : { “cluster\_file” : { “path” : “/etc/foundationdb/fdb.cluster”, “up\_to\_date” : true }, “coordinators” : { “coordinators” : [ { “address” : “172.16.9.186:4500”, “reachable” : true }, { “address” : “172.16.57.12:4500”, “reachable” : false }, { “address” : “178.104.163.99:4500”, “reachable” : true } ], “quorum\_reachable” : true }, “database\_status” : { “available” : false, “healthy” : false }, “messages” : [ ], “timestamp” : 1594021260 }, “cluster” : { “clients” : { “count” : 1, “supported\_versions” : [ { “client\_version” : “5.2.5”, “connected\_clients” : [ { “address” : “178.104.163.99:40356”, “log\_group” : “default” } ], “count” : 1, “protocol\_version” : “fdb00a552000001”, “source\_version” : “4e48018437df4506aa5ed0c7f5976b9412b0145f” } ] }, “cluster\_controller\_timestamp” : 1594021266, “generation” : 463, “incompatible\_connections” : [ ], “layers” : { “\_error” : “configurationMissing”, “\_valid” : false }, “machines” : { “ae5132ed0959f5fbd16edad8584e03ae” : { “address” : “178.104.163.99”, “contributing\_workers” : 7, “cpu” : { “logical\_core\_utilization” : 0.027834043319133618 }, “locality” : { “machineid” : “ae5132ed0959f5fbd16edad8584e03ae”, “processid” : “077beda5c638bb83effae78e30222afc”, “zoneid” : “ae5132ed0959f5fbd16edad8584e03ae” }, “machine\_id” : “ae5132ed0959f5fbd16edad8584e03ae”, “memory” : { “committed\_bytes” : 1259991040, “free\_bytes” : 2874003456, “total\_bytes” : 4133994496 }, “network” : { “megabits\_received” : { “hz” : 0.51068500000000006 }, “megabits\_sent” : { “hz” : 0.437585 }, “tcp\_segments\_retransmitted” : { “hz” : 8.999820003599929 } } }, “d3d7f561de2bf8943ab08504b3655a0b” : { “address” : “172.16.9.186”, “contributing\_workers” : 14, “cpu” : { “logical\_core\_utilization” : 0.094262257377426245 }, “locality” : { “machineid” : “d3d7f561de2bf8943ab08504b3655a0b”, “processid” : “9ba9f030e1e76f850524a7dc84046526”, “zoneid” : “d3d7f561de2bf8943ab08504b3655a0b” }, “machine\_id” : “d3d7f561de2bf8943ab08504b3655a0b”, “memory” : { “committed\_bytes” : 1036357632, “free\_bytes” : 891998208, “total\_bytes” : 1928355840 }, “network” : { “megabits\_received” : { “hz” : 0.45399900000000004 }, “megabits\_sent” : { “hz” : 0.53055800000000009 }, “tcp\_segments\_retransmitted” : { “hz” : 9.9999000009999897 } } } }, “messages” : [ { “description” : “Unable to read database configuration.”, “name” : “unreadable\_configuration” } ], “processes” : { “077beda5c638bb83effae78e30222afc” : { “address” : “178.104.163.99:4500”, “class\_source” : “command\_line”, “class\_type” : “storage”, “command\_line” : “/usr/sbin/fdbserver --class=storage --cluster\_file=/etc/foundationdb/fdb.cluster --datadir=/var/lib/foundationdb/data/4500 --listen\_address=public --logdir=/var/log/foundationdb --logsize=100MiB --maxlogssize=1000MiB --public\_address=auto:4500 --storage\_memory=2GiB”, “cpu” : { “usage\_cores” : 0.012318353632927345 }, “disk” : { “busy” : 0, “free\_bytes” : 57510076416, “reads” : { “counter” : 20732, “hz” : 0, “sectors” : 0 }, “total\_bytes” : 94463066112, “writes” : { “counter” : 1404875, “hz” : 0, “sectors” : 0 } }, “fault\_domain” : “ae5132ed0959f5fbd16edad8584e03ae”, “locality” : { “machineid” : “ae5132ed0959f5fbd16edad8584e03ae”, “processid” : “077beda5c638bb83effae78e30222afc”, “zoneid” : “ae5132ed0959f5fbd16edad8584e03ae” }, “machine\_id” : “ae5132ed0959f5fbd16edad8584e03ae”, “memory” : { “available\_bytes” : 781585261, “limit\_bytes” : 8589934592, “used\_bytes” : 594219008 }, “messages” : [ ], “network” : { “connection\_errors” : { “hz” : 0 }, “connections\_closed” : { “hz” : 0 }, “connections\_established” : { “hz” : 0 }, “current\_connections” : 24, “megabits\_received” : { “hz” : 0.031529000000000001 }, “megabits\_sent” : { “hz” : 0.024975500000000001 } }, “roles” : [ ], “uptime\_seconds” : 607.29499999999996, “version” : “5.2.5” }, “16cc4712fdbb548394c2eed61d122b3f” : { “address” : “178.104.163.99:4506”, “class\_source” : “command\_line”, “class\_type” : “stateless”, “command\_line” : “/usr/sbin/fdbserver --class=stateless --cluster\_file=/etc/foundationdb/fdb.cluster --datadir=/var/lib/foundationdb/data/4506 --listen\_address=public --logdir=/var/log/foundationdb --logsize=100MiB --maxlogssize=1000MiB --public\_address=auto:4506 --storage\_memory=2GiB”, “cpu” : { “usage\_cores” : 0.0087347126528734709 }, “disk” : { “busy” : 0, “free\_bytes” : 57510076416, “reads” : { “counter” : 20732, “hz” : 0, “sectors” : 0 }, “total\_bytes” : 94463066112, “writes” : { “counter” : 1404875, “hz” : 0, “sectors” : 0 } }, “fault\_domain” : “ae5132ed0959f5fbd16edad8584e03ae”, “locality” : { “machineid” : “ae5132ed0959f5fbd16edad8584e03ae”, “processid” : “16cc4712fdbb548394c2eed61d122b3f”, “zoneid” : “ae5132ed0959f5fbd16edad8584e03ae” }, “machine\_id” : “ae5132ed0959f5fbd16edad8584e03ae”, “memory” : { “available\_bytes” : 781611593, “limit\_bytes” : 8589934592, “used\_bytes” : 195780608 }, “messages” : [ ], “network” : { “connection\_errors” : { “hz” : 0 }, “connections\_closed” : { “hz” : 0 }, “connections\_established” : { “hz” : 0 }, “current\_connections” : 17, “megabits\_received” : { “hz” : 0.013567900000000001 }, “megabits\_sent” : { “hz” : 0.014398300000000001 } }, “roles” : [ ], “uptime\_seconds” : 605.00800000000004, “version” : “5.2.5” }, … … … “e90d5b0682731a5243691dfd80572e86” : { “address” : “178.104.163.99:4501”, “class\_source” : “command\_line”, “class\_type” : “storage”, “command\_line” : “/usr/sbin/fdbserver --class=storage --cluster\_file=/etc/foundationdb/fdb.cluster --datadir=/var/lib/foundationdb/data/4501 --listen\_address=public --logdir=/var/log/foundationdb --logsize=100MiB --maxlogssize=1000MiB --public\_address=auto:4501 --storage\_memory=2GiB”, “cpu” : { “usage\_cores” : 0.0096214460568630911 }, “disk” : { “busy” : 0, “free\_bytes” : 57510076416, “reads” : { “counter” : 20732, “hz” : 0, “sectors” : 0 }, “total\_bytes” : 94463066112, “writes” : { “counter” : 1404875, “hz” : 0, “sectors” : 0 } }, “fault\_domain” : “ae5132ed0959f5fbd16edad8584e03ae”, “locality” : { “machineid” : “ae5132ed0959f5fbd16edad8584e03ae”, “processid” : “e90d5b0682731a5243691dfd80572e86”, “zoneid” : “ae5132ed0959f5fbd16edad8584e03ae” }, “machine\_id” : “ae5132ed0959f5fbd16edad8584e03ae”, “memory” : { “available\_bytes” : 781611593, “limit\_bytes” : 8589934592, “used\_bytes” : 453988352 }, “messages” : [ ], “network” : { “connection\_errors” : { “hz” : 0 }, “connections\_closed” : { “hz” : 0 }, “connections\_established” : { “hz” : 0 }, “current\_connections” : 17, “megabits\_received” : { “hz” : 0.0136078 }, “megabits\_sent” : { “hz” : 0.014567800000000001 } }, “roles” : [ ], “uptime\_seconds” : 605.01499999999999, “version” : “5.2.5” } }, “protocol\_version” : “fdb00a552000001”, “recovery\_state” : { “description” : “Recruiting new transaction servers.”, “name” : “recruiting\_transaction\_servers”, “required\_logs” : 3, “required\_proxies” : 1, “required\_resolvers” : 1 } } } ` ---
**Author:** ![ajbeamon](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ajbeamon/32/13_2.png) [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)\ **Post date:** [July 6, 2020, 3:18pm UTC](https://forums.foundationdb.org/t/triple-ssd-fdb-cluster-on-3-node-one-node-poweroff-but-the-fdb-cluster-is-unavailable/2228/2 "2020-07-06T15:18:06Z")
Running in `triple` replication requires at least 3 separate machines for the cluster to be available (see [https://apple.github.io/foundationdb/configuration.html#single-datacenter-modes](https://apple.github.io/foundationdb/configuration.html#single-datacenter-modes)). When you power off one of your 3 machines, the cluster is no longer able to recruit the needed roles and will not be available. In order to be able to tolerate failures, you could run in a reduced redundancy mode (e.g. `double`) or add more machines (we recommend at least 5 for two fault tolerance in triple mode). ---
**Author:** ![ssb](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ssb/32/656_2.png) [@ssb](https://forums.foundationdb.org/u/ssb)\ **Post date:** [July 7, 2020, 1:23am UTC](https://forums.foundationdb.org/t/triple-ssd-fdb-cluster-on-3-node-one-node-poweroff-but-the-fdb-cluster-is-unavailable/2228/3 "2020-07-07T01:23:20Z")
Thank you very much and sorry for missing configuration page.