I am doing some resiliency testing on the FDB cluster in the Kubernetes environment. A Kubernetes Pod is like a VM. A pod hosts 2 or 3 FDB server processes and a FDB monitor. When a Pod (VM) becomes unhealthy at t0, the FDB cluster will detect that the Pod (VM) is unhealthy at t1. And sometime later, t2, the FDB cluster will determine that it will need to perform data re-balancing, to distribute the data replica that is hosted on the unhealthy pod to the other servers, to make sure that the required number of the replica meets the FDB cluster configuration (for example, triple replica).
Under some circumstance, after t1, the Pod (VM) may become healthy again in a short period of time, for example, due to the recovery of a network glitch. If (t2-t1) is too short, it might trigger the unnecessary data rebalancing.
So my question is: how long is the setting for (t2-t1) in the current FDB? and can this parameter to be tuned?
The related question is: supposed between t1 and t3, incoming transactions involves the write to the replica (hosted on Pod-1). Since the pod (Pod-1) is not reachable, the data will be distributed to the other pod ( say, Pod-2). Now after the pod becomes healthy again, will the data that is already distributed to Pod-2 will have to be shipped back to Pod-1?