# Three\_data\_hall coordinators **URL:** \ **Category:** Kubernetes Operator\ **Created:** [November 16, 2023, 9:07pm UTC](https://forums.foundationdb.org/t/three-data-hall-coordinators/4243 "2023-11-16T21:07:51Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![stefanvasilic4](https://avatars.discourse-cdn.com/v4/letter/s/a8b319/32.png) [@stefanvasilic4](https://forums.foundationdb.org/u/stefanvasilic4)\ **Post date:** [November 16, 2023, 9:07pm UTC](https://forums.foundationdb.org/t/three-data-hall-coordinators/4243/1 "2023-11-16T21:07:51Z")
Hi @johscheuer , how did you manage to get 9 coordinators for three\_data\_hall ? ( [Initial support for three data hall replication by johscheuer · Pull Request #1651 · FoundationDB/fdb-kubernetes-operator · GitHub](https://github.com/FoundationDB/fdb-kubernetes-operator/pull/1651) ) every time i test i get 1 ☹ ? ---
**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\ **Post date:** [November 17, 2023, 6:50am UTC](https://forums.foundationdb.org/t/three-data-hall-coordinators/4243/2 "2023-11-17T06:50:51Z")
Could you share some more information about your setup? Without any information about what operator version you use and how your FoundationDBCluster resources look like it’s hard to help you. ---
**Author:** ![stefanvasilic4](https://avatars.discourse-cdn.com/v4/letter/s/a8b319/32.png) [@stefanvasilic4](https://forums.foundationdb.org/u/stefanvasilic4)\ **Post date:** [November 17, 2023, 1:32pm UTC](https://forums.foundationdb.org/t/three-data-hall-coordinators/4243/3 "2023-11-17T13:32:05Z")
i want to achieve three\_data\_hall across 3 AZ (cloud). I have nodes labeled with `topology.kubernetes.io/zone=` and following [https://github.com/FoundationDB/fdb-kubernetes-operator/tree/main/config/tests/three\_data\_hall](https://github.com/FoundationDB/fdb-kubernetes-operator/tree/main/config/tests/three_data_hall) for deployment. Also locality is set ```auto localities: - key: "data_hall" value: $az ``` with initial triple cluster i get a default 5 coordinators, when changes to three\_data\_hall goes to 1 instead of 9. what am i missing ? thank you ---
**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\ **Post date:** [November 17, 2023, 3:55pm UTC](https://forums.foundationdb.org/t/three-data-hall-coordinators/4243/4 "2023-11-17T15:55:34Z")
Are you able to actually share the FoundationDBCluster resource? How many nodes do you have per AZ? You need at least 3 nodes per AZ, otherwise the operator is not able to select the right amount of coordinators. > localities: > > - key: “data\_hall” > value: $az Shouldn’t the `$az` be replaced with the actual value (not sure where this information is from)? The docs have some additional information about the setup: [https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/manual/fault\_domains.md#three-data-hall-replication](https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/manual/fault_domains.md#three-data-hall-replication) ---
**Author:** ![stefanvasilic4](https://avatars.discourse-cdn.com/v4/letter/s/a8b319/32.png) [@stefanvasilic4](https://forums.foundationdb.org/u/stefanvasilic4)\ **Post date:** [November 17, 2023, 4:48pm UTC](https://forums.foundationdb.org/t/three-data-hall-coordinators/4243/5 "2023-11-17T16:48:16Z")
please ignore `localities` section , is not working with new api `apps.foundationdb.org/v1beta2` ( it comes from [https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/design/three\_datahall.md#coordinator-selection](https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/design/three_datahall.md#coordinator-selection)); back to initial problem, i do have 50 nodes available, and pods are scheduled correctly on respective nodes. fdbcluster definition i am using at the moment is [https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/config/tests/three\_data\_hall/final.yaml](https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/config/tests/three_data_hall/final.yaml) env vars: ```auto AZ1=${AZ1:-"eastus2-1"} AZ2=${AZ2:-"eastus2-2"} AZ3=${AZ3:-"eastus2-3"} ``` observations: - in triple mode , i have 5 logs and 5 storage and 5 coordinators - when it switches to three\_data\_hall it shrinks , i have 3 clusters with 3 logs , 1 storage each and 1 coordinator - nothing obvious in operator pod logs ---
**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\ **Post date:** [November 21, 2023, 10:44am UTC](https://forums.foundationdb.org/t/three-data-hall-coordinators/4243/6 "2023-11-21T10:44:43Z")
What operator version is deployed in your case? And have you made sure you use the correct CRD deployed from the according release branch (or newer)? I just tested the scripts/test setup that you referenced and everything works fine: ```auto $ fdbcli --exec 'status details' Using cluster file `/var/dynamic-conf/fdb.cluster'. Configuration: Redundancy mode - three_data_hall Storage engine - ssd-2 Coordinators - 9 Desired Commit Proxies - 2 Desired GRV Proxies - 1 Desired Resolvers - 1 Desired Logs - 4 Desired Remote Logs - -1 Desired Log Routers - -1 Usable Regions - 1 ... Coordination servers: 192.168.0.3:4501 (reachable) 192.168.0.4:4501 (reachable) 192.168.0.5:4501 (reachable) 192.168.0.6:4501 (reachable) 192.168.0.23:4501 (reachable) 192.168.0.9:4501 (reachable) 192.168.0.11:4501 (reachable) 192.168.0.101:4501 (reachable) 192.168.0.102:4501 (reachable) ``` Is there anything interesting in the operator logs? Are you able to share them? ---
**Author:** ![stefanvasilic4](https://avatars.discourse-cdn.com/v4/letter/s/a8b319/32.png) [@stefanvasilic4](https://forums.foundationdb.org/u/stefanvasilic4)\ **Post date:** [November 21, 2023, 2:32pm UTC](https://forums.foundationdb.org/t/three-data-hall-coordinators/4243/7 "2023-11-21T14:32:53Z")
i updated crds but missed to update operator image version. Thank you so much for help!