FDB client connects to old IP addresses

Using FoundationDB
1

We have our FDB cluster setup in Kubenetes environment.
When some FDB pods are re-created, their IPs change. However, we found out in our monitoring log that the FDB clients are still attempting to connect to the old IPs.

My questions are:

  1. How long after a pod is re-created the FDB client abandons the old IP address? In our case, we still see the client attempts to connect to old IP address after hours / days.
  2. If the IP address is re-used for a different FDB cluster, is it possible for the FDB client to undesirably read/write data to the other FDB cluster?

Thanks.

2

FDB doesn’t support IP address changes for coordinators. So I don’t think this situation will recover on its own. If the IP address of the coordinators isn’t stable, you would need to use DNS (though I am not sure how stable that feature is yet, someone from Apple might have more insights).

You probably can change to a different set of coordinators as a temporary fix. Afterwards I would recommend you try to keep the coordinator IPs stable until a proper fix has been deployed. Personally I don’t have experience with this problem, so I wouldn’t be the best person to recommend the best way forward.