# Shutdown/scale down FDB on OCP/kubernetes (and keep the pvc)

**URL:** <https://forums.foundationdb.org/t/shutdown-scale-down-fdb-on-ocp-kubernetes-and-keep-the-pvc/3236>\
**Category:** Running FoundationDB\
**Tags:** operator\
**Created:** [March 30, 2022, 2:57pm UTC](https://forums.foundationdb.org/t/shutdown-scale-down-fdb-on-ocp-kubernetes-and-keep-the-pvc/3236 "2022-03-30T14:57:08Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![tangerine](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/tangerine/32/1084_2.png) [@tangerine](https://forums.foundationdb.org/u/tangerine)\
**Post date:** [March 30, 2022, 2:57pm UTC](https://forums.foundationdb.org/t/shutdown-scale-down-fdb-on-ocp-kubernetes-and-keep-the-pvc/3236/1 "2022-03-30T14:57:08Z")

</div>

HI,  
We are running FDB on OCP/kubernetes using the operator and created the cluster. We would like to know if there is a proper way to scale down all the pods/shutdown the cluster but keep the pvc so that we can bring them back and reuse those pvc. I know we have to refresh the connect string when the cluster is back but just wonder is there a way to scale down/shutdown the cluster without losing those pvc.

---

<div class="post-metadata">

**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\
**Post date:** [March 31, 2022, 7:40am UTC](https://forums.foundationdb.org/t/shutdown-scale-down-fdb-on-ocp-kubernetes-and-keep-the-pvc/3236/2 "2022-03-31T07:40:13Z")

</div>

Currently we don’t have an official way of doing this but here are my thoughts how I would approach this (not tested):

- I would set the `skip` option to true for this cluster [fdb-kubernetes-operator/cluster\_spec.md at main · FoundationDB/fdb-kubernetes-operator · GitHub](https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/cluster_spec.md#foundationdbclusterspec), this basically means the operator won’t do any action on it.
- Now you can delete all Pods, the PVCs will stay available.
- You could leave the operator and the cluster spec in that state and once you need the cluster again change `skip` to `false` and the operator should bring back those Pods.
- Once all Pods are up and running (and you didn’t use services) you probably have to run [fdb-kubernetes-operator/debugging.md at main · FoundationDB/fdb-kubernetes-operator · GitHub](https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/manual/debugging.md#coordinators-getting-new-ips) to fix the connection string.

I think all of those steps should work without a problem and you probably should be able to write a script/automaton around it. Like I said I didn’t test those steps so you want to test those steps in a test cluster.

---

<div class="post-metadata">

**Author:** ![liuyan](https://avatars.discourse-cdn.com/v4/letter/l/839c29/32.png) [@liuyan](https://forums.foundationdb.org/u/liuyan)\
**Post date:** [April 1, 2022, 5:20am UTC](https://forums.foundationdb.org/t/shutdown-scale-down-fdb-on-ocp-kubernetes-and-keep-the-pvc/3236/3 "2022-04-01T05:20:46Z")

</div>

Hi [johscheuer](https://forums.foundationdb.org/u/johscheuer), I also have the similar requirement, I tried to set `skip` in our cr(fdbcluster) by edit the yaml directly

```auto
 oc edit fdbcluster
  foundationdb_cluster_spec:
    skip: true
    databaseConfiguration:
      logs: 3
      storage: 4

```

once save it, check the yaml again using `oc get fdbcluster -o yaml` the `skip:true` is gone, it seems it didn’t work, what am I missing? thanks.

---

<div class="post-metadata">

**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\
**Post date:** [April 1, 2022, 11:40am UTC](https://forums.foundationdb.org/t/shutdown-scale-down-fdb-on-ocp-kubernetes-and-keep-the-pvc/3236/4 "2022-04-01T11:40:50Z")

</div>

Could you add some more information like the operator version and CRD version that you use.

---

<div class="post-metadata">

**Author:** ![liuyan](https://avatars.discourse-cdn.com/v4/letter/l/839c29/32.png) [@liuyan](https://forums.foundationdb.org/u/liuyan)\
**Post date:** [April 1, 2022, 12:58pm UTC](https://forums.foundationdb.org/t/shutdown-scale-down-fdb-on-ocp-kubernetes-and-keep-the-pvc/3236/5 "2022-04-01T12:58:44Z")

</div>

I did check the version, it’s 0.48.0, `skip` definitely in the code already. what else to check?

---

<div class="post-metadata">

**Author:** ![liuyan](https://avatars.discourse-cdn.com/v4/letter/l/839c29/32.png) [@liuyan](https://forums.foundationdb.org/u/liuyan)\
**Post date:** [April 2, 2022, 3:22am UTC](https://forums.foundationdb.org/t/shutdown-scale-down-fdb-on-ocp-kubernetes-and-keep-the-pvc/3236/6 "2022-04-02T03:22:46Z")

</div>

> [@johscheuer](#):
>
> e operator version and CRD vers

I confirmed apple’s crd, it has default `false`.  
I tried to edit foundationdb’s cr

```auto
oc get foundationdbcluster
NAME GENERATION RECONCILED AVAILABLE FULLREPLICATION VERSION AGE
mdm-foundationdb-ibm 7 7 true true 6.2.29 41m
 oc edit foundationdbcluster/mdm-foundationdb-ibm
  sidecarContainer:
    enableTls: true
  skip: false

```

modify `skip: true`, then save it.  
after a while, check the cr yaml, it is `false` again. how to make it `true`?

---

<div class="post-metadata">

**Author:** ![liuyan](https://avatars.discourse-cdn.com/v4/letter/l/839c29/32.png) [@liuyan](https://forums.foundationdb.org/u/liuyan)\
**Post date:** [April 2, 2022, 3:59am UTC](https://forums.foundationdb.org/t/shutdown-scale-down-fdb-on-ocp-kubernetes-and-keep-the-pvc/3236/7 "2022-04-02T03:59:06Z")

</div>

Or what’s the correct approach to change `skip=true`?

I also checked apple operator pod logs, I did see the skip=true

```auto
{"level":"info","ts":1648870636.2536569,"logger":"controller","msg":"Skipping cluster with skip value true","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","skip":true}
{"level":"info","ts":1648870640.296823,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.updateStatus"}
{"level":"info","ts":1648870640.2970383,"logger":"fdbclient","msg":"Fetch status from FDB","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm"}
{"level":"info","ts":1648870640.3331993,"logger":"fdbclient","msg":"Successfully fetched status from FDB","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm"}
{"level":"info","ts":1648870640.56386,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.updateLockConfiguration"}
{"level":"info","ts":1648870640.5639207,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.updateConfigMap"}
{"level":"info","ts":1648870640.5640814,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.checkClientCompatibility"}
{"level":"info","ts":1648870640.5642319,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.replaceMisconfiguredProcessGroups"}
{"level":"info","ts":1648870640.5650394,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.replaceFailedProcessGroups"}
{"level":"info","ts":1648870640.565153,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.deletePodsForBuggification"}
{"level":"info","ts":1648870640.56552,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.addProcessGroups"}
{"level":"info","ts":1648870640.565573,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.addServices"}
{"level":"info","ts":1648870640.5655944,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.addPVCs"}
{"level":"info","ts":1648870640.5657523,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.addPods"}
{"level":"info","ts":1648870640.5660434,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.generateInitialClusterFile"}
{"level":"info","ts":1648870640.5660734,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.updateSidecarVersions"}
{"level":"info","ts":1648870640.566363,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.updatePodConfig"}
{"level":"info","ts":1648870640.5668392,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.updateLabels"}
{"level":"info","ts":1648870640.5672073,"logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm","subReconciler":"controllers.updateDatabaseConfiguration"}
{"level":"info","ts":1648870640.5672848,"logger":"fdbclient","msg":"Fetch status from FDB","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm"}
{"level":"info","ts":1648870640.59316,"logger":"fdbclient","msg":"Successfully fetched status from FDB","namespace":"testoperator1","cluster":"mdm-foundationdb-ibm"}

```

I just suspect maybe the “controllers.updateStatus” update it to false?

---

<div class="post-metadata">

**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\
**Post date:** [April 2, 2022, 5:39am UTC](https://forums.foundationdb.org/t/shutdown-scale-down-fdb-on-ocp-kubernetes-and-keep-the-pvc/3236/8 "2022-04-02T05:39:27Z")

</div>

I don’t know if you use the upstream operator without any changes, if so the operator will stop running any reconciliation on clusters with the skip setting: [fdb-kubernetes-operator/cluster\_controller.go at main · FoundationDB/fdb-kubernetes-operator · GitHub](https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/controllers/cluster_controller.go#L93-L97), so those sub reconcilers should never be executed. In addition to that the FDB operator. will never change the cluster spec to prevent conflict and will only update the status resource of the cluster spec (to indicate if the cluster if reconciled). You should check if there is anything else that actually changes the cluster spec outside of the operator e.g. another meta operator.

You can also try to setup a small Kubernetes cluster with e.g. kind and try out the skip functionality there.

---

<div class="post-metadata">

**Author:** ![liuyan](https://avatars.discourse-cdn.com/v4/letter/l/839c29/32.png) [@liuyan](https://forums.foundationdb.org/u/liuyan)\
**Post date:** [April 2, 2022, 8:35am UTC](https://forums.foundationdb.org/t/shutdown-scale-down-fdb-on-ocp-kubernetes-and-keep-the-pvc/3236/9 "2022-04-02T08:35:13Z")

</div>

Thank you @johscheuer . without our operator, it works fine. will check my controller. Thank you so much for all your help.