# Proposal: Don't identify a coordinator based on its IP

**URL:** <https://forums.foundationdb.org/t/proposal-dont-identify-a-coordinator-based-on-its-ip/502>\
**Category:** FoundationDB Core\
**Created:** [June 9, 2018, 1:09pm UTC](https://forums.foundationdb.org/t/proposal-dont-identify-a-coordinator-based-on-its-ip/502 "2018-06-09T13:09:51Z")\
**Posts on this page:** 1\
**Showing post:** 23

<div class="post-metadata">

**Author:** ![rishabh](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/rishabh/32/540_2.png) [@rishabh](https://forums.foundationdb.org/u/rishabh)\
**Post date:** [June 22, 2019, 3:29pm UTC](https://forums.foundationdb.org/t/proposal-dont-identify-a-coordinator-based-on-its-ip/502/23 "2019-06-22T15:29:23Z")

</div>

> [@alexmiller](#):
>
> If you wouldn’t mind digging into how TiDB/TiKV and Yugabyte are deployed on Kubernetes, I’d appreciate it.

Thanks for pointing me to right direction. I looked at yugabyte. I will summarize it based on [this](https://blog.yugabyte.com/understanding-how-yugabyte-db-runs-on-kubernetes/) doc, skipping the _Admin UI_ stuff.

Yugabyte is deployed as two (stateful) sets: yb-master set and yb-tserver set.  
Each set is accompanied with a (headless) service, which provides hostname + **current** IP of all its members. The yb-master set service is used by (1) a master to discover other masters and (2) by a tserver to initialize/join cluster.

I simply visualized yb-master as fdb coordinator and tried to answer the question “If I deploy coordinators as a separate (stateful) set, can I use its (headless) service to seed cluster file to other nodes”. The anwer is no, as I need **ID** of cluster file as well.

```auto
description:ID@ip1:port1,ip2:port2,ip3:port3

```

And as per [this](https://forums.foundationdb.org/t/doubts-regarding-fdb-cluster-file/1109/2) post, it’s not any random ID.

> [@alexmiller](#):
>
> I’m not clear how this solves the bootstrapping problem of starting many FDB processes at the same time, when a cluster does not yet exist? You’re still going to need to provide a cluster file to get the initial set of processes running?

K8s stateful set starts pods one by one (with pod index)and ensures that all previous pods are sane before adding another pod. Hence no 2 pods start simultaneously during initialization (except edge case scenarios arising from uncontrolled pod restarts). This ability could be used to intialize cluster at a particualr step of deployement.  
A short ex: Self-seed first pod (initialize single-node cluster), keep on seeding next pods by first pod and then in 6th pod initialization, initialize cluster with first 5 pods as coordinators.

**Next steps**  
I didn’t go further into “how to deploy fdb by statefulset”, as I now know that fdb team has plans to explore k8s custom resource. This is good for both venting cluster files and also for maintaining multiple sets within a Deployment, like stateless, log, storage etc.  
I liked the idea of multi-set deployement, CouchBase provides a cutom resource for this, whereas YB tells you to use 2 statefulsets. By this, you can independently scale what roles you want more.

I am now looking into k8s custom resource. My current understanding is that the problem of “coordinator being identified by IP” can also be solved by cutom-operator. We have to maintain set of coordinatos independently in fdb Deployment. And our custom-operator can ensure that all (healthy) members of this set are coordinators of the cluster (k8s operator is essentially a feedback loop). If not, initialize the cluster with coordinators. Will see where it goes 🙂

---

_[View the full topic](https://forums.foundationdb.org/t/proposal-dont-identify-a-coordinator-based-on-its-ip/502)._