Failure / Recovery scenario

We want to support a scenario where all Kube nodes are brought down and then restarted.
We are testing this by deleting all the pods and then having the fdb-operator re-create them.

Unfortunately, the cluster never recovers, and we get this error in the fdb-operator logs. It seems like the ConfigMap shows an empty string in “cluster-file”.
I assume that the fdb-operator cannot reach out the cluster coordinators (since those ip’s don’t exist anymore) and it gets stuck in a fail loop.

2020-10-09T21:23:42.217Z ERROR controller-runtime.controller Reconciler error {“controller”: “foundationdbcluster”, “request”: “deploy-infra/storage-cluster”, “error”: “Unable to fetch connection string: The database is unavailable; type `status’ for more information.\n”}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.17.0/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.17.0/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/apimachinery@v0.17.0/pkg/util/wait/wait.go:88

Is there a way to support this scenario with the Kubernetes operator?
We tried deleting and re-creating the cluster, in hope that the PVs get preserved, but the data is wiped.

NOTE that this works in a non-kube deployment since the machines coming back up will have the same IPs and the cluster recovers

This issue has more discussion that I think is relevant to this topic: https://github.com/FoundationDB/fdb-kubernetes-operator/issues/266