Large excludes failing repeatedly

2021-02-11T11:19:48.082Z	INFO	controller	Running command	{"namespace": "timeseries", "cluster": "foundationdb-cluster-1", "path": "/usr/bin/fdb/6.2/fdbcli", "args": ["/usr/bin/fdb/6.2/fdbcli", "--exec", "exclude", "-C", "/tmp/529281271", "--log", "--timeout", "10", "--log-dir", "/var/log/fdb"]}
2021-02-11T11:19:48.254Z	ERROR	controller	Error from FDB command	{"namespace": "timeseries", "cluster": "foundationdb-cluster-1", "code": 1, "stdout": "ERROR: This exclude may cause the total free space in the cluster to drop below 10%.\nType `exclude FORCE <ADDRESS>*' to exclude without checking free space.\n", "stderr": "", "error": "exit status 1"}

After updating the cluster spec to use service IP, a migration was triggered, creating a new set of pods and moving data over to the new pods. After this step the controller is stuck, because it can not exclude the old pods. The reason is that the pods are using roughly %50 of disk, so the disk space calculation very conservatively assumes that the old pods may not be excluded.

Would it be possible for the controller to exclude old pods in smaller batches, so that the free disk check succeeds?

The only workaround I can think of for now is to scale up until the old pods are a sufficiently small percentage of the overall cluster, but that option will not always work for us, as we have some environments that are more resource constrained.

Re: rolling: this is Support rolling update of pods on migration · Issue #400 · FoundationDB/fdb-kubernetes-operator · GitHub

But: N Bytes of storage in M nodes, doubled = N Bytes in 2M nodes; excluded that will mean NBytes in M nodes. If the nodes are at 50% footprint before the migration, there should be no issue with the 10% thing. The only way I see a 10% issue arising is either many new pods are not yet provisioned, or you were already down at or below 10% before this step.

AFAIK the disk check only takes into account the most utilized disk, not the total number of bytes in the cluster foundationdb/ at 1dac117543642bebbcff29623ee967145605e982 · apple/foundationdb · GitHub

Lets work it through: if your worst disks is at 50% usage, and you have 50 nodes being replaced we get:

(1-0.5)100/(100-0)=0.51 = 0.5

This is < 0.9 and so should not trigger the fault.

Please grab a detailed status.json from your cluster and also check that your new nodes have fully provisioned.