Cluster spec normalization forces CPU throttling?

Development Kubernetes Operator Development
1

The current operator normalizes pod templates to always have request/limit specified: fdb-kubernetes-operator/internal/deprecations.go at b0f00690ba98e6891d36d0f108d847b4079f78e1 · FoundationDB/fdb-kubernetes-operator · GitHub

This normalization is applied whenever cluster spec is loaded, not just when running the deprecation command.

However, even if storage processes cannot use more than 1 CPU, specifying request and limit will cause CPU throttling to happen (for any process), which is detrimental to latency.

Has this been considered before? e.g. allowing to not specify the upper CPU limit in pod templates?
Right now it seems impossible to achieve this; I wanted to open a PR but not before a discussion, and do not have permissions anyways.

I have already measured throttling to be happening, even if in small percentages.

Thanks for any insight!

2

Has this been considered before? e.g. allowing to not specify the upper CPU limit in pod templates?
Right now it seems impossible to achieve this; I wanted to open a PR but not before a discussion, and do not have permissions anyways.

This is already possible, see: fdb-kubernetes-operator/docs/manual/warnings.md at main · FoundationDB/fdb-kubernetes-operator · GitHub

e.g. a spec like this would work:

apiVersion: apps.foundationdb.org/v1beta2
kind: FoundationDBCluster
metadata:
    name: sample-cluster
spec:
  version: 7.1.26
  processes:
    general:
      podTemplate:
        spec:
          containers:
            - name: foundationdb
              resources:
                requests:
                  cpu: 1
                  memory: 8Gi
                limits:
                  org.foundationdb/empty: 0

or if you still want to limit memory this would also work:

apiVersion: apps.foundationdb.org/v1beta2
kind: FoundationDBCluster
metadata:
    name: sample-cluster
spec:
  version: 7.1.26
  processes:
    general:
      podTemplate:
        spec:
          containers:
            - name: foundationdb
              resources:
                requests:
                  cpu: 1
                  memory: 8Gi
                limits:
                  memory: 8Gi

I hope this helps.

1 Like
3

Thanks, for some reason the latter (limited memory but request-only for CPU) didn’t work for me; will run more tests.

4

Thanks, for some reason the latter (limited memory but request-only for CPU) didn’t work for me; will run more tests.

If you discover that you can reproduce that and there might be a bug in the operator code, please reach out again. This is the code path where the limits are set if no limits are specified:
fdb-kubernetes-operator/internal/deprecations.go at main · FoundationDB/fdb-kubernetes-operator · GitHub. Based on that logic setting the memory limits should be enough.

1 Like
5

It was not a problem in the operator, I could trace it back to wrong Helm values defaults; thanks for the help with this!