# Dual stack cluster configuration

**URL:** <https://forums.foundationdb.org/t/dual-stack-cluster-configuration/3226>\
**Category:** Using FoundationDB\
**Created:** [March 23, 2022, 11:04am UTC](https://forums.foundationdb.org/t/dual-stack-cluster-configuration/3226 "2022-03-23T11:04:30Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![pmalamy](https://avatars.discourse-cdn.com/v4/letter/p/96bed5/32.png) [@pmalamy](https://forums.foundationdb.org/u/pmalamy)\
**Post date:** [March 23, 2022, 11:04am UTC](https://forums.foundationdb.org/t/dual-stack-cluster-configuration/3226/1 "2022-03-23T11:04:30Z")

</div>

Hi!

We flawlessly run several FoundationDB clusters on IPv6 only network with TLS. One of our future workloads will involve IPv4 only clients as well which would imply those clusters to support dual stack IP.

Note: in the following examples, IPv6 documentation prefix and IPv4 RF1918 are used on purpose.

Our current IPv6 only configuration looks like the following:

- foundationdb.conf (needless to say that `locality_machineid` is different on each member of a cluster):

```auto
[fdbmonitor]
user = foundationdb
group = foundationdb

[general]
restart_delay = 60
cluster_file = /etc/foundationdb/fdb.cluster

[fdbserver]
command = /usr/sbin/fdbserver
public_address = auto:$ID:tls
listen_address = public
datadir = /var/lib/foundationdb/data/$ID
logdir = /var/log/foundationdb/fdbserver
locality_machineid = hostN.fqdn.tld
locality_dcid = gv2
trace_format = json
tls_certificate_file = /etc/foundationdb/ssl/cert.pem
tls_key_file = /etc/foundationdb/ssl/key.pem
tls_verify_peers = <redacted>
tls_password = 
tls_ca_file = /etc/foundationdb/ssl/ca.pem
knob_tls_cert_refresh_delay_seconds = 3600

[fdbserver.4600]

```

- fdb.cluster

```auto
store_fdb_pp_gv2:TWwLTw1STKgEnix1p3jAadSZWDoKLIof@[2001:0db8::41d:18ff:fe00:d27]:4600:tls,[2001:0db8::47c:aaff:fe00:13ed]:4600:tls,[2001:0db8::4a5:38ff:fe00:1313]:4600:tls,[2001:0db8::4c4:66ff:fe00:46]:4600:tls,[2001:0db8::4cd:5eff:fe00:124d]:4600:tls

```

- Detailed status output of such cluster:

```auto
$ fdbcli --exec 'status details'
[...]
Configuration:
  Redundancy mode - triple
  Storage engine - ssd-2
  Coordinators - 5
  Usable Regions - 1

Cluster:
  FoundationDB processes - 5
  Zones - 5
  Machines - 5
[...]
Data:
  Replication health - Healthy
[...]
Process performance details: 
  [2001:0db8::41d:18ff:fe00:d27]:4600:tls (<redacted>) 
  [2001:0db8::47c:aaff:fe00:13ed]:4600:tls (<redacted>) 
  [2001:0db8::4a5:38ff:fe00:1313]:4600:tls (<redacted>) 
  [2001:0db8::4c4:66ff:fe00:46]:4600:tls (<redacted>) 
  [2001:0db8::4cd:5eff:fe00:124d]:4600:tls (<redacted>) 

Coordination servers:
  [2001:0db8::41d:18ff:fe00:d27]:4600:tls (reachable)
  [2001:0db8::47c:aaff:fe00:13ed]:4600:tls (reachable)
  [2001:0db8::4a5:38ff:fe00:1313]:4600:tls (reachable)
  [2001:0db8::4c4:66ff:fe00:46]:4600:tls (reachable)
  [2001:0db8::4cd:5eff:fe00:124d]:4600:tls (reachable)

```

In an attempt to make a cluster dual stacked, we changed the configuration file `foundationdb.conf` in order to specify IPv6 and ipv4 addresses instead of using `auto` and `public`, not setting TLS on IPv4 on purpose:

```diff
 [fdbserver]
 command = /usr/sbin/fdbserver
-public_address = auto:$ID:tls
-listen_address = public
+public_address = [2001:0db8::41d:18ff:fe00:d27]:$ID:tls,172.20.9.17:$ID
+listen_address = [2001:0db8::41d:18ff:fe00:d27]:$ID:tls,172.20.9.17:$ID
 datadir = /var/lib/foundationdb/data/$ID
 logdir = /var/log/foundationdb/fdbserver
 locality_machineid = hostN.fqdn.tld

```

and we updated the `fdb.cluster` file accordingly:

```auto
store_fdb_pp_gv2:TsmKlq7cqUsuTkbbfFTDjeG4lBJRPHpj@172.20.8.18:4600,172.20.9.17:4600,172.20.9.183:4600,172.20.9.233:4600,172.20.10.236:4600,[2001:0db8::41d:18ff:fe00:d27]:4600:tls,[2001:0db8::47c:aaff:fe00:13ed]:4600:tls,[2001:0db8::4a5:38ff:fe00:1313]:4600:tls,[2001:0db8::4c4:66ff:fe00:46]:4600:tls,[2001:0db8::4cd:5eff:fe00:124d]:4600:tls

```

Those changes made on each member of the cluster with their very own IP addresses, the cluster starts and **seems** to run:

- Detailed status output of such cluster:

```auto
Configuration:
  Redundancy mode - triple
  Storage engine - ssd-2
  Coordinators - 10
  Usable Regions - 1

Cluster:
  FoundationDB processes - 5
  Zones - 5
  Machines - 5
[...]
Data:
  Replication health - Healthy
[...]
Process performance details:
  [2001:0db8::41d:18ff:fe00:d27]:4600:tls (<redacted>)
  [2001:0db8::47c:aaff:fe00:13ed]:4600:tls (<redacted>)
  [2001:0db8::4a5:38ff:fe00:1313]:4600:tls (<redacted>)
  [2001:0db8::4c4:66ff:fe00:46]:4600:tls (<redacted>)
  [2001:0db8::4cd:5eff:fe00:124d]:4600:tls (<redacted>)

Coordination servers:
  172.20.8.18:4600 (reachable)
  172.20.9.17:4600 (reachable)
  172.20.9.183:4600 (reachable)
  172.20.9.233:4600 (reachable)
  172.20.10.236:4600 (reachable)
  [2001:0db8::41d:18ff:fe00:d27]:4600:tls (reachable)
  [2001:0db8::47c:aaff:fe00:13ed]:4600:tls (reachable)
  [2001:0db8::4a5:38ff:fe00:1313]:4600:tls (reachable)
  [2001:0db8::4c4:66ff:fe00:46]:4600:tls (reachable)
  [2001:0db8::4cd:5eff:fe00:124d]:4600:tls (reachable)

```

Although it seems to work, I think something is not right. First, the coordinators number has doubled from 5 to 10. Then, the processes are IPv6 only. I can paste/upload output of `status json` if needed.

I would like to know if dual stack FDB cluster configuration can be done, if we are heading in the right direction to achieve it or completely lost 🙂 I welcome all recommendations related the topic!

Thanks!

---

<div class="post-metadata">

**Author:** ![pmalamy](https://avatars.discourse-cdn.com/v4/letter/p/96bed5/32.png) [@pmalamy](https://forums.foundationdb.org/u/pmalamy)\
**Post date:** [March 25, 2022, 2:49pm UTC](https://forums.foundationdb.org/t/dual-stack-cluster-configuration/3226/2 "2022-03-25T14:49:33Z")

</div>

Update on our dual stack experiments.  
First, I realized that I have forgotten to write that we currently run version 6.3.23 of FoundationDB binaries.

Second, with the configuration posted above, even though the coordinators quorum is reached, all of them being reachable and the database considered healthy, I’ve noticed that the `Fault Tolerance` is decreased from 2 zones to 0. Meaning that the redundancy is not assured anymore.

I also tried to fiddle with the `public_address` and `listen_address` parameters (like using `[::]:$ID:tls` among other tests), but did not succeed in obtaining a satisfying state. Even though `fdbserver` process is bound to the two IP:PORT declared in configuration, the FDB processes are still only accessible via IPv6.

```auto
ss -lpent '( sport = :4600 )'
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process                                                                                                                  
LISTEN 0 128 172.20.9.17:4600 0.0.0.0:* users:(("fdbserver",pid=144018,fd=14)) uid:1425 ino:139280706 sk:21 <->                                                 
LISTEN 0 128 [2001:0db8::41d:18ff:fe00:d27]:4600 [::]:* users:(("fdbserver",pid=144018,fd=15)) uid:1425 ino:139280708 sk:23 v6only:1 <->   

```

I really wonder if there is a way to obtain a cluster that would be fully dual stack and if not, what would be the needed changes to be done on the source side.

Any ideas?

---

<div class="post-metadata">

**Author:** ![pyr](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/pyr/32/1040_2.png) [@pyr](https://forums.foundationdb.org/u/pyr)\
**Post date:** [April 4, 2022, 1:42pm UTC](https://forums.foundationdb.org/t/dual-stack-cluster-configuration/3226/3 "2022-04-04T13:42:21Z")

</div>

I am running into the same issue, and would be interested in any developments here