We’re running a FoundationDB cluster in EKS and are trying to develop a procedure for node group updates.
Our first attempt ended poorly. The plan was:
- Trigger an automated node group update from the EKS side (see Understand each phase of node updates - Amazon EKS )
- EKS node group updates taint/drain each node
- Use automated replacement (fdb-kubernetes-operator/docs/manual/replacements_and_deletions.md at main · FoundationDB/fdb-kubernetes-operator · GitHub) to replace tainted nodes
This worked at some level because processes did, indeed, get replaced when their nodes got tainted. It did not work as hoped, though, because (as I understand it!) “replacement” and “exclusion” are two separate ideas, and we wound up replacing storage/log processes before they could replicate their data, leading to data and availability loss. My (new) understanding is that replacement will just clobber the process entirely regardless of whether its data has been fully replicated, but exclusion will wait until replication is complete.
Reading through the operator docs, it looks like we can manually exclude processes, and it looks like the operator will automatically manage exclusions if we try to shrink the cluster. I can’t find any documentation, though, of a way to exclude a process if its node is tainted. Is that the case, or have I missed something?
More broadly:
- Is there a way to exclude processes based on node state or annotations or similar Kubernetes entities?
- Is there an established set of best practices for working with EKS node group updates?
Thanks kindly!