# Cannot enable TLS without DISABLE\_SIDECAR\_TLS\_CHECK=1

**URL:** <https://forums.foundationdb.org/t/cannot-enable-tls-without-disable-sidecar-tls-check-1/2804>\
**Category:** Kubernetes Operator\
**Created:** [July 17, 2021, 1:03pm UTC](https://forums.foundationdb.org/t/cannot-enable-tls-without-disable-sidecar-tls-check-1/2804 "2021-07-17T13:03:31Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![mason](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/mason/32/1145_2.png) [@mason](https://forums.foundationdb.org/u/mason)\
**Post date:** [July 17, 2021, 1:03pm UTC](https://forums.foundationdb.org/t/cannot-enable-tls-without-disable-sidecar-tls-check-1/2804/1 "2021-07-17T13:03:31Z")

</div>

Hi all,  
I tried to deploy FDB cluster with TLS enable with document [https://github.com/FoundationDB/fdb-kubernetes-operator/blob/master/docs/manual/tls.md](https://github.com/FoundationDB/fdb-kubernetes-operator/blob/master/docs/manual/tls.md)

If I do not DISABLE\_SIDECAR\_TLS\_CHECK=1, it will always send error

```auto
{
  "level": "error",
  "ts": 1626526429.2545986,
  "logger": "controller-runtime.manager.controller.foundationdbcluster",
  "msg": "Reconciler error",
  "reconciler group": "apps.foundationdb.org",
  "reconciler kind": "FoundationDBCluster",
  "name": "fdb-cluster",
  "namespace": "default",
  "error": "GET https://10.42.2.14:8080/substitutions giving up after 11 attempt(s): Get \"https://10.42.2.14:8080/substitutions\": x509: cannot validate certificate for 10.42.2.14 because it doesn't contain any IP SANs",
  "stacktrace": "github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:302\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:99"
}

```

Could you help me solve this?

---

<div class="post-metadata">

**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\
**Post date:** [July 19, 2021, 6:56am UTC](https://forums.foundationdb.org/t/cannot-enable-tls-without-disable-sidecar-tls-check-1/2804/2 "2021-07-19T06:56:44Z")

</div>

`"https://10.42.2.14:8080/substitutions: x509: cannot validate certificate for 10.42.2.14 because it doesn't contain any IP SANs"` tells you the issue. If you wanted to run a TLS cluster without `DISABLE_SIDECAR_TLS_CHECK` you have to provide valid TLS certificates that match the IP address (currently the operator will use the IP address to connect to the sidecar). That’s also mentioned in the linked manual: [fdb-kubernetes-operator/tls.md at master · FoundationDB/fdb-kubernetes-operator · GitHub](https://github.com/FoundationDB/fdb-kubernetes-operator/blob/master/docs/manual/tls.md#configuring-the-operator). This is currently a limitation how TLS is handled we have a planned feature to make this more flexible: [Custom validation for TLS connections to the sidecar · Issue #756 · FoundationDB/fdb-kubernetes-operator · GitHub](https://github.com/FoundationDB/fdb-kubernetes-operator/issues/756). If you need this feature feel free to work on it and file a PR.

---

<div class="post-metadata">

**Author:** ![mason](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/mason/32/1145_2.png) [@mason](https://forums.foundationdb.org/u/mason)\
**Post date:** [July 19, 2021, 11:44am UTC](https://forums.foundationdb.org/t/cannot-enable-tls-without-disable-sidecar-tls-check-1/2804/3 "2021-07-19T11:44:49Z")

</div>

Hi @johscheuer  
Thank for your answer. Can you give me one example about peer verification?  
I do some but it not working for peer, here is my config:

> apiVersion: [cert-manager.io/v1](http://cert-manager.io/v1)  
> kind: Certificate  
> metadata:  
> name: fdb-cluster-certificate  
> namespace: default  
> spec:  
> dnsNames:  
> - “\*.fdb-cluster.default.svc.cluster.local”  
> secretName: fdb-cluster-certificate  
> issuerRef:  
> name: fdb-cluster-issuer  
> kind: Issuer

> routing:  
> publicIPSource: “service”  
> mainContainer:  
> enableTls: true  
> peerVerificationRules: “subjectAltName.DNS\>=fdb-cluster.default.svc.cluster.local”  
> sidecarContainer:  
> enableTls: true  
> peerVerificationRules: “subjectAltName.DNS\>=fdb-cluster.default.svc.cluster.local”

---

<div class="post-metadata">

**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\
**Post date:** [July 22, 2021, 6:08am UTC](https://forums.foundationdb.org/t/cannot-enable-tls-without-disable-sidecar-tls-check-1/2804/4 "2021-07-22T06:08:40Z")

</div>

The official docs have some examples: [Transport Layer Security — FoundationDB 6.3](https://apple.github.io/foundationdb/tls.html#peer-verification) if anything is missing please let me know or update the docs with the missing pieces. You can also look at the trace logs in the FoundationDB containers to see what’s the issue with the verification rule (or rather why the certificate is not matching). Depending on how cert-manager generates the certificates they might be seen as self-signed rather then “official” certificates.

---

<div class="post-metadata">

**Author:** ![mason](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/mason/32/1145_2.png) [@mason](https://forums.foundationdb.org/u/mason)\
**Post date:** [July 22, 2021, 10:05am UTC](https://forums.foundationdb.org/t/cannot-enable-tls-without-disable-sidecar-tls-check-1/2804/5 "2021-07-22T10:05:14Z")

</div>

Hi @johscheuer  
I did the same with official document and peerVerificationRule work with sidecar, not for maincontainer  
Here is my config:

> mainContainer:  
> enableTls: true  
> peerVerificationRules: “S.subjectAltName\<=DNS:.fdb-cluster.default.svc.cluster.local”  
> sidecarContainer:  
> enableTls: true  
> peerVerificationRules: “S.subjectAltName\<=DNS:.fdb-cluster.default.svc.cluster.local”

---

<div class="post-metadata">

**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\
**Post date:** [July 23, 2021, 7:39am UTC](https://forums.foundationdb.org/t/cannot-enable-tls-without-disable-sidecar-tls-check-1/2804/6 "2021-07-23T07:39:41Z")

</div>

Could you take a look in the FDB trace logs for any TLS issues? I assume that’s an issue with self-signed certificates. If you root CA is a self-signed cert it must contain the `AuthorityKeyId` that points to itself otherwise FDB will reject a self-signed certificate as root cert.

---

<div class="post-metadata">

**Author:** ![tangerine](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/tangerine/32/1084_2.png) [@tangerine](https://forums.foundationdb.org/u/tangerine)\
**Post date:** [February 18, 2022, 10:48pm UTC](https://forums.foundationdb.org/t/cannot-enable-tls-without-disable-sidecar-tls-check-1/2804/7 "2022-02-18T22:48:12Z")

</div>

HI,  
I tried the rules as well as adding the AuthorityKeyID in the root certificate. In both cases, I get this error:  
{“level”:“error”,“ts”:1645030087.361136,“logger”:“controller”,“msg”:“Error deserializing pod substitutions”,“responseBody”:"\n\n \n \<meta http-equiv=“Content-Type” content=“text/html;charset=utf-8”\>\n Error response\n \n \n

# Error response
\n 

Error code: 401

\n 

Message: Client certificate was not approved.

\n 

Error code explanation: 401 - No permission – see authorization schemes.

\n \n\n",“error”:“invalid character ‘\<’ looking for beginning of value”,“stacktrace”:“[github.com/FoundationDB/fdb-kubernetes-operator/internal.(\*realFdbPodClient](http://github.com/FoundationDB/fdb-kubernetes-operator/internal.(*realFdbPodClient)).GetVariableSubstitutions\n\t/workspace/internal/pod\_client.go:216\[Ngithub.com](http://ngithub.com/FoundationDB/fdb-kubernetes-operator/controllers.localityInfoFromSidecar%5Cn%5Ct/workspace/controllers/cluster_controller.go:424%5Cngithub.com/FoundationDB/fdb-kubernetes-operator/controllers.generateInitialClusterFile.reconcile%5Cn%5Ct/workspace/controllers/generate_initial_cluster_file.go:83%5Cngithub.com/FoundationDB/fdb-kubernetes-operator/controllers.(*FoundationDBClusterReconciler)).Reconcile\n\t/workspace/controllers/cluster\_controller.go:155\[nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller](http://nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:298\[nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller](http://nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:253\[nsigs.k8s.io/controller-runtime/pkg/internal/controller.(\*Controller](http://nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:214”}  
Any idea what that is about?

---

<div class="post-metadata">

**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\
**Post date:** [February 21, 2022, 9:37am UTC](https://forums.foundationdb.org/t/cannot-enable-tls-without-disable-sidecar-tls-check-1/2804/8 "2022-02-21T09:37:50Z")

</div>

> [@tangerine](#):
>
> tried the rules as well as adding the AuthorityKeyID in the root certificate. In both cases, I get this error:  
> {“level”:“error”,“ts”:1645030087.361136,“logger”:“controller”,“msg”:“Error deserializing pod substitutions”,“responseBody”:"\n\n \n \<meta http-equiv=“Content-Type” content=“text/html;charset=utf-8”\>\n Error response\n \n \n

This error comes from one of the sidecars and mean that the provided cert by the operator doesn’t match the provided `peerVerificationRules` for the `sidecarContainer`.

---

<div class="post-metadata">

**Author:** ![tangerine](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/tangerine/32/1084_2.png) [@tangerine](https://forums.foundationdb.org/u/tangerine)\
**Post date:** [February 22, 2022, 3:35pm UTC](https://forums.foundationdb.org/t/cannot-enable-tls-without-disable-sidecar-tls-check-1/2804/9 "2022-02-22T15:35:23Z")

</div>

The rule I used is peerVerificationRules: “Check.Valid=1”, wouldn’t that would match anything?