Failure / Recovery scenario

We want to support a scenario where all Kube nodes are brought down and then restarted.
We are testing this by deleting all the pods and then having the fdb-operator re-create them.

Unfortunately, the cluster never recovers, and we get this error in the fdb-operator logs. It seems like the ConfigMap shows an empty string in “cluster-file”.
I assume that the fdb-operator cannot reach out the cluster coordinators (since those ip’s don’t exist anymore) and it gets stuck in a fail loop.

2020-10-09T21:23:42.217Z ERROR controller-runtime.controller Reconciler error {“controller”: “foundationdbcluster”, “request”: “deploy-infra/storage-cluster”, “error”: “Unable to fetch connection string: The database is unavailable; type `status’ for more information.\n”}*zapLogger).Error

Is there a way to support this scenario with the Kubernetes operator?
We tried deleting and re-creating the cluster, in hope that the PVs get preserved, but the data is wiped.

NOTE that this works in a non-kube deployment since the machines coming back up will have the same IPs and the cluster recovers

This issue has more discussion that I think is relevant to this topic: