We have a Java-based client that needs connectivity to multiple FDB clusters simultaneously. Each cluster has its own set of TLS certificates.
Initially, we were also planning to have separate client certificates as well, one corresponding to each cluster. However, it appears we cannot initialize a client to use multiple client certificates. TLS options are part of NetworkOptions and startNetwork can be called only once
We have some questions regarding the TLS setup
Is there an assumption that all clusters that you could connect at the same time use certs emitted by the same CA ?
Apart from the TLS configuration, would FDB library support simultaneous active connections to multiple clusters and interacting with them? We are trying to make sure we would be using a supported feature and not something that just happens to work / unintentional.
For those of you that work with multiple FDB clusters, do you usually assign/create one client-certificate per client node? Do you have the same root CA generate certificates for all clusters?
For certain applications, we like to limit FDB access to matching application nodes (different tenants, for example). I understand there is an upcoming feature described in this “Authorization in FDB”. thread . Are there any techniques to achieve that before 7.2? Is this usually done at the network/firewall level?
For production setup, do you usually work with internal/company-managed CA or use a commercial/real CA like LetsEncrypt/Digicert ?
Is there an assumption that all clusters that you could connect at the same time use certs emitted by the same CA ?
I’m not an expert in the internals of the bindings but theoretically with the multi-threaded client having a dedicated certificate configuration should be possible, but I believe this feature is not exposed via the bindings.
Apart from the TLS configuration, would FDB library support simultaneous active connections to multiple clusters and interacting with them? We are trying to make sure we would be using a supported feature and not something that just happens to work / unintentional.
This is supported and works fine with the FDB client library, we use this setup e.g. in the fdb-kubernetes-operator. If you use a recent FDB version (6.3+) you might want to look at the multi-threaded client (Using FoundationDB Clients — FoundationDB 7.1) for FDB, depending on your workload this might help to improve performance.
For those of you that work with multiple FDB clusters, do you usually assign/create one client-certificate per client node? Do you have the same root CA generate certificates for all clusters?
For the operator the current assumption is that only one client certificate is provided, so all clusters that are managed by the same FDB Kubernetes operator instance must have the same root CA. The idea is to separate the access for the different clusters by using the peer verification (Transport Layer Security — FoundationDB 7.1), assuming that your certs for the different clusters have a field that can be used to distinguish those clusters.
For certain applications, we like to limit FDB access to matching application nodes (different tenants, for example). I understand there is an upcoming feature described in this “Authorization in FDB”. thread . Are there any techniques to achieve that before 7.2? Is this usually done at the network/firewall level?
If you only want to control who can connect to the FDB cluster you can use the peer verification, but this only allows or denies access to the whole cluster (not specific key spaces).
but that (TLS verification) wouldn’t work right ? i mean the problem is like this
we only initialise the network thread once. this means next time it keeps the initial certificate. lets suppose we have:
CA (used by two clusters)
cert1 → CN: cluster1
cert2 → CN: cluster2
first connection → cluster1 → uses cert1 → passes validation (CN=cluster1)
second connection → cluster2 → still uses cert1 (cant change certs) → fails (CN=cluster1, cluster2 needs CN=cluster2)
as for multi thread clients - it seem you are only able to initialise one per fdb VERSION. (at least reading docs). if both clusters use same api version not sure how we can have two different network threads.
but that (TLS verification) wouldn’t work right ? i mean the problem is like this
we only initialise the network thread once. this means next time it keeps the initial certificate. lets suppose we have:
CA (used by two clusters)
cert1 → CN: cluster1
cert2 → CN: cluster2
first connection → cluster1 → uses cert1 → passes validation (CN=cluster1)
second connection → cluster2 → still uses cert1 (cant change certs) → fails (CN=cluster1, cluster2 needs CN=cluster2)
as for multi thread clients - it seem you are only able to initialise one per fdb VERSION. (at least reading docs). if both clusters use same api version not sure how we can have two different network threads.
Let me add some more context here what I meant:
The operator would get a client cert with CN=fdb-operator and all fdb server processes and clients of cluster 1 get a cert with OU=cluster1 and for cluster 2 we use OU=cluster2 (only an example we could also use X509v3 extensions with DNS). All those certificates are issues by the same root CA.
For cluster 1 we can make use of the following peer verification rule:
S.CN=fdb-operator,S.OU=cluster1
and for cluster 2:
S.CN=fdb-operator,S.OU=cluster2
Those two rules will ensure that only the operator can access both clusters and otherwise only the clients (and fdb server processes) of specified clusters can connect e.g. only clients for cluster 1 can connect to cluster 1 otherwise they will get a peer verification error.
right, but remember the constraint from the client is that it only uses the first certificate used to connect to a cluster.
So for cluster2 it will use CN=fdb-operator,OU=cluster1 - which if i’m understanding correctly should fail with a verification of S.CN=fdb-operator,S.OU=cluster2
we could only have a cluster verification of S.CN=fdb-operator but that means we really have no way of separating clusters…
