Thanks for pointing me to right direction. I looked at yugabyte. I will summarize it based on this doc, skipping the Admin UI stuff.
Yugabyte is deployed as two (stateful) sets: yb-master set and yb-tserver set.
Each set is accompanied with a (headless) service, which provides hostname + current IP of all its members. The yb-master set service is used by (1) a master to discover other masters and (2) by a tserver to initialize/join cluster.
I simply visualized yb-master as fdb coordinator and tried to answer the question “If I deploy coordinators as a separate (stateful) set, can I use its (headless) service to seed cluster file to other nodes”. The anwer is no, as I need ID of cluster file as well.
description:ID@ip1:port1,ip2:port2,ip3:port3
And as per this post, it’s not any random ID.
K8s stateful set starts pods one by one (with pod index)and ensures that all previous pods are sane before adding another pod. Hence no 2 pods start simultaneously during initialization (except edge case scenarios arising from uncontrolled pod restarts). This ability could be used to intialize cluster at a particualr step of deployement.
A short ex: Self-seed first pod (initialize single-node cluster), keep on seeding next pods by first pod and then in 6th pod initialization, initialize cluster with first 5 pods as coordinators.
Next steps
I didn’t go further into “how to deploy fdb by statefulset”, as I now know that fdb team has plans to explore k8s custom resource. This is good for both venting cluster files and also for maintaining multiple sets within a Deployment, like stateless, log, storage etc.
I liked the idea of multi-set deployement, CouchBase provides a cutom resource for this, whereas YB tells you to use 2 statefulsets. By this, you can independently scale what roles you want more.
I am now looking into k8s custom resource. My current understanding is that the problem of “coordinator being identified by IP” can also be solved by cutom-operator. We have to maintain set of coordinatos independently in fdb Deployment. And our custom-operator can ensure that all (healthy) members of this set are coordinators of the cluster (k8s operator is essentially a feedback loop). If not, initialize the cluster with coordinators. Will see where it goes 