Storage pod excluded and deleted before replacement is scheduled or healthy

Using FoundationDB Kubernetes Operator
1

Summary

We have identified a case where the fdb-kubernetes-operator excludes and terminates an existing, healthy storage pod before its replacement pod is scheduled, running, or ready to take traffic.

In an environment experiencing node provisioning failures (e.g., node autoscaler issues where replacement pods sit in Pending), the operator proceeds to drain and delete original storage pods after a 5-minute timeout. This can lead to degraded cluster redundancy or capacity loss.

We originally observed this on operator v1.54.0 and successfully reproduced the exact failure mechanism on v2.35.0.

Core Question: Is this decoupling of the replacement pod’s readiness from the old process’s exclusion intentional, or a known limitation in the operator logic?

Root Cause Analysis

Looking at the behavior, two unlinked mechanisms drive this lifecycle:

  1. Exclusion Gate: The operator waits 5 minutes after a process group replacement is triggered. Once this grace period expires, process exclusion in FDB occurs unconditionally—regardless of whether the replacement pod has been scheduled or reached Ready.

  2. Removal Gate: The operator waits for FDB data migration to complete off the old process, then deletes/terminates the original Pod. It does not check if the replacement Pod is in a healthy or schedulable state before completing the deletion.

Reproduction Steps

We reproduced this cleanly in a non-production cluster running v2.35.0:

  1. Inject a scheduling failure: Patch the podTemplate for the storage process class with an unsatisfiable nodeSelector (chaos-test-no-schedule: "true"). Existing pods remain healthy; any newly created replacement pod stays permanently Pending.

kubectl patch foundationdbcluster foundationdb-cluster --type merge -p \ '{"spec":{"processes":{"storage":{"podTemplate":{"spec":{"nodeSelector":{"chaos-test-no-schedule":"true"}}}}}}}'

  1. Trigger process replacement:

kubectl fdb remove process-groups -c foundationdb-cluster <pod-name>

  1. Observe reconciliation timeline:

  • Replacement pod is created and sits in Pending (0/3 Ready).

  • After 5 minutes, the operator issues the ExcludingProcesses event for the original healthy pod.

  • FDB finishes migrating data off.

  • The operator terminates the excluded pod, leaving only the unschedulable Pending replacement pod behind.

Observed Pod Statuses

NAME READY STATUS AGE COMMENT
foundationdb-cluster-az-3-storage-31 0/3 Pending 66m Replacement never scheduled
foundationdb-cluster-az3-storage-81 3/3 Running 26h
foundationdb-cluster-storage-33 3/3 Running 3h8m
foundationdb-cluster-az3-storage-36 3/3 Terminating 23h Drained & deleted anyway

Questions:

  1. Expected vs. Desired Behavior: Is this decoupling of the replacement’s readiness from the old process’s exclusion intentional, or a known limitation in the operator’s reconciliation logic?

  2. Safety Gates / Workarounds: Is there an existing configuration knob in the FoundationDBCluster spec to prevent exclusion/deletion if replacement capacity isn’t actively scheduled?

2

This behaviour is expected: fdb-kubernetes-operator/controllers/exclude_processes.go at main · FoundationDB/fdb-kubernetes-operator · GitHub but it should only take down 2 pods if triple replicated or 1 pod if double replicated. The idea was to allow exclusions to move forward in resource constraint environments if the operator is not able to get all pods running. Right now there is no setting to disable this behaviour. Are you seeing any issues with this behaviour?