I’ve put together a design proposal for automating replacements through the operator: https://github.com/FoundationDB/fdb-kubernetes-operator/wiki/Design-for-Automating-Replacements-through-the-Operator.
I think this could really help with reducing toil, especially in environments with local volumes. I’d appreciate any feedback on it!
It really seems like this cries out for the ability to exclude processes by ID rather than IP. Specifying an ID for each process up front in the operator and using those for excludes would eliminate a lot of complexity and room for error.
It does, and excluding processes by ID is what we intend to do as a long-term solution to the problem of safely replacing processes that don’t have IPs. Once we have that in place, it should remove the need to delete pods without excluding them.
First IP should be ID?
We’re going to trying a new approach to design reviews to make it easier to gather feedback in one place. We now have a design docs folder in the repo, and I’ve moved this design into a PR: https://github.com/FoundationDB/fdb-kubernetes-operator/pull/381/files