FDB operator support of node affinity?

Using FoundationDB Running FoundationDB
1

Hi,
Does the FDB operator support node affinity? ie, one storage pod per worker node? Currently, it seems to be set at preferred which means it try at best to do one storage pod per node but will allocate more pods per node if number of storage pods are more then nodes.

2

The built-in affinity rules use preferred, but you can define additional affinity rules through the pod template in the cluster spec. We don’t have any examples of customizing affinity in the operator docs, but we have other examples of pod customization here: fdb-kubernetes-operator/customization.md at master · FoundationDB/fdb-kubernetes-operator · GitHub

3

Hi,
This is what we put in:
storage:
allowTagOverride: true
podTemplate:
metadata:
labels:
role: storage
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: role
operator: In
values:
- storage
- key: fdb-cluster-name
operator: NotIn
values:
- mdm-foundationdb-ibm
topologyKey: “kubernetes.io/hostname”

But the default “preferred” overridden our setting and so we can see storage pod being created if number of pods requested is bigger then the # of nodes. We can see 2 pods on the same nodes due to the preferred.

4

I just ran a test locally based on that example, and in my test the operator added both a preferredDuringScheduling constraint and a requiredDuringScheduling constraint to the pod spec.

5

But in our tests, when we requested 7 storagepods out of a 5 workers node ocp cluster, we get all 7 pods created. Our expectation would have been that only 5 should be created and 2 should just be pending but that’s not the case. So, it seems the preferred take over the required…

6

Is mdm-foundationdb-ibm the name of the cluster that this policy is being applied to? If so, I would expect that you would want to use the In operator for the fdb-cluster-name affinity rule, rather than the NotIn operator.

7

HI,
This is what we set:
podTemplate:
spec:
volumes:
- name: fdb-certs
secret:
#secretName: fdb-kubernetes-operator-secrets
secretName: internal-tls
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: fdb-process-class
operator: In
values:
- storage
topologyKey: “kubernetes.io/hostname”

So, basically, we want the antiaffanity which means one storage pod per node per FDB cluster and this is a required condition, not preferred. But in the yaml file for the storage pod after the pod got created, we can see both preferred section and our required section. And it seems that preferred section took effect when there is more pods requested than node. Is there a way to override or don’t add that preferred section by the operator?

8

What version of the operator are you running? You need at least the version 0.38.0 Release Release v0.38.0 · FoundationDB/fdb-kubernetes-operator · GitHub previous versions will overwrite any additional affinity settings. We also have an issue to add support for allowing thee user to specify if they want a required or preferred affinity: Allow to specify if the operator should be using a preferred or required AntiAffinity · Issue #875 · FoundationDB/fdb-kubernetes-operator · GitHub, feel free to work on this if you need that feature it should be straight forward to add it.

9

I see. We were trying with 0.35.1, I guess that’s too old.
We are now trying with 0.45.0 to see if it work.