Shutdown/scale down FDB on OCP/kubernetes (and keep the pvc)

Using FoundationDB Running FoundationDB
1

HI,
We are running FDB on OCP/kubernetes using the operator and created the cluster. We would like to know if there is a proper way to scale down all the pods/shutdown the cluster but keep the pvc so that we can bring them back and reuse those pvc. I know we have to refresh the connect string when the cluster is back but just wonder is there a way to scale down/shutdown the cluster without losing those pvc.

2

Currently we don’t have an official way of doing this but here are my thoughts how I would approach this (not tested):

I think all of those steps should work without a problem and you probably should be able to write a script/automaton around it. Like I said I didn’t test those steps so you want to test those steps in a test cluster.

3

Hi johscheuer, I also have the similar requirement, I tried to set skip in our cr(fdbcluster) by edit the yaml directly

 oc edit fdbcluster
  foundationdb_cluster_spec:
    skip: true
    databaseConfiguration:
      logs: 3
      storage: 4

once save it, check the yaml again using oc get fdbcluster -o yaml the skip:true is gone, it seems it didn’t work, what am I missing? thanks.

4

Could you add some more information like the operator version and CRD version that you use.

5

I did check the version, it’s 0.48.0, skip definitely in the code already. what else to check?

6

I confirmed apple’s crd, it has default false.
I tried to edit foundationdb’s cr

oc get foundationdbcluster
NAME                   GENERATION   RECONCILED   AVAILABLE   FULLREPLICATION   VERSION   AGE
mdm-foundationdb-ibm   7            7            true        true              6.2.29    41m
 oc edit foundationdbcluster/mdm-foundationdb-ibm
  sidecarContainer:
    enableTls: true
  skip: false

modify skip: true, then save it.
after a while, check the cr yaml, it is false again. how to make it true?

7

Or what’s the correct approach to change skip=true?

I also checked apple operator pod logs, I did see the skip=true

{"level":"info","ts":1648870636.2536569,"logger":"controller","msg":"Skipping cluster with skip value true","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","skip":true}
{"level":"info","ts":1648870640.296823,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.updateStatus"}
{"level":"info","ts":1648870640.2970383,"logger":"fdbclient","msg":"Fetch status from FDB","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm"}
{"level":"info","ts":1648870640.3331993,"logger":"fdbclient","msg":"Successfully fetched status from FDB","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm"}
{"level":"info","ts":1648870640.56386,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.updateLockConfiguration"}
{"level":"info","ts":1648870640.5639207,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.updateConfigMap"}
{"level":"info","ts":1648870640.5640814,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.checkClientCompatibility"}
{"level":"info","ts":1648870640.5642319,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.replaceMisconfiguredProcessGroups"}
{"level":"info","ts":1648870640.5650394,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.replaceFailedProcessGroups"}
{"level":"info","ts":1648870640.565153,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.deletePodsForBuggification"}
{"level":"info","ts":1648870640.56552,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.addProcessGroups"}
{"level":"info","ts":1648870640.565573,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.addServices"}
{"level":"info","ts":1648870640.5655944,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.addPVCs"}
{"level":"info","ts":1648870640.5657523,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.addPods"}
{"level":"info","ts":1648870640.5660434,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.generateInitialClusterFile"}
{"level":"info","ts":1648870640.5660734,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.updateSidecarVersions"}
{"level":"info","ts":1648870640.566363,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.updatePodConfig"}
{"level":"info","ts":1648870640.5668392,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.updateLabels"}
{"level":"info","ts":1648870640.5672073,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.updateDatabaseConfiguration"}
{"level":"info","ts":1648870640.5672848,"logger":"fdbclient","msg":"Fetch status from FDB","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm"}
{"level":"info","ts":1648870640.59316,"logger":"fdbclient","msg":"Successfully fetched status from FDB","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm"}

I just suspect maybe the “controllers.updateStatus” update it to false?

8

I don’t know if you use the upstream operator without any changes, if so the operator will stop running any reconciliation on clusters with the skip setting: fdb-kubernetes-operator/cluster_controller.go at main · FoundationDB/fdb-kubernetes-operator · GitHub, so those sub reconcilers should never be executed. In addition to that the FDB operator. will never change the cluster spec to prevent conflict and will only update the status resource of the cluster spec (to indicate if the cluster if reconciled). You should check if there is anything else that actually changes the cluster spec outside of the operator e.g. another meta operator.

You can also try to setup a small Kubernetes cluster with e.g. kind and try out the skip functionality there.

1 Like
9

Thank you @johscheuer . without our operator, it works fine. will check my controller. Thank you so much for all your help.