# Configuring FoundationDB to Use More Than One Resolver

**URL:** <https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414>\
**Category:** Using FoundationDB\
**Tags:** performance\
**Created:** [May 30, 2019, 8:58pm UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414 "2019-05-30T20:58:21Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ricky.saltzer](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@ricky.saltzer](https://forums.foundationdb.org/u/ricky.saltzer)\
**Post date:** [May 30, 2019, 8:58pm UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/1 "2019-05-30T20:58:21Z")

</div>

Hi!

I’m currently running a POC for a new project that will use FoundationDB as its backend. The workload is inherently bursty and has a mix of read and write workload (20/80 I’d say).

My current configuration is the following

```
Configuration:
Redundancy mode - triple
Storage engine - ssd-2
Coordinators - 7
Desired Proxies - 7
Desired Resolvers - 3
Desired Logs - 3

Cluster:
FoundationDB processes - 72
Machines - 18
Memory availability - 3.7 GB per process on machine with least available
                       >>>>> (WARNING: 4.0 GB recommended) <<<<<
Retransmissions rate - 0 Hz
Fault Tolerance - 2 machines
Server time - 05/30/19 16:46:55

```

Each of machines are running on `c5d.2xlarge` instance types. I currently have three dedicated machines (4 processes on each) dedicated to the `stateless` class. So, 12 `stateless` processes in total.

My **issue** is the following:

I cannot get the cluster to recruit more than one resolver. I’ve set the desired amount to 3. I’ve even gone as far as to manually `setclass <ip:port> resolution`, but for some reason FDB is refusing to recruit any additional process as a `resolver`.

The odd part is, some of my `stateless` processes have **zero** roles assigned to them. So you’d think they’d be perfect candidates for the role.

Normally this wouldn’t be an issue, but it seems that the `resolver` is in fact becoming my bottleneck. My write throughput starts hitting a ceiling at around 250k Hz with my `resolver` process’s CPU maxed out at 100%.

I’ve attached a recent [1] `status json` report for anybody who is willing to help 🙂

Most notably, if you search for `10.49.58.119:4503` you will see that the class type is set to `resolution`, yet the process has **zero** roles assigned to it.

[1] [https://gist.github.com/rickysaltzer/cf6327a26a7fc45a553253f1ed51e19a](https://gist.github.com/rickysaltzer/cf6327a26a7fc45a553253f1ed51e19a)

Ricky

---

<div class="post-metadata">

**Author:** ![mbhaskar](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/mbhaskar/32/277_2.png) [@mbhaskar](https://forums.foundationdb.org/u/mbhaskar)\
**Post date:** [May 30, 2019, 10:46pm UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/2 "2019-05-30T22:46:12Z")

</div>

I don’t know much about increasing resolvers, I will let someone else answer that.

But, if the resolver is the only process CPU bound and all other processes are doing good, probably it’s worth reviewing the data model. You should check and make sure you are not generating too many conflict ranges.

If you have too many point-gets or sets, that could cause too many conflict ranges. Then, resolvers have to do too many range comparisons. Instead, if you find a smallest overlapping range and add that as the conflict range that would be better for resolver performance.

Document Layer inserts can be a very good example here. Document Layer stores each JSON field in a different FDB key with the same prefix. For example, let’s say I have a document in the collection `employee` looking like

```json
{
  "_id": 345,
  "name": "Eric",
  "marks": 90,
  "grade": A
}

```

Note that `_id` is the primary key. So, the document is stored with key `_id`. This would be stored under 4 different keys

```auto
employee:345: -> _
employee:345:name -> Eric
employee:345:marks -> 90
employee:345:grade -> A

```

When you insert these keys, it will create 4 write conflict ranges, one per each key. If my document can have embedded documents with deep arrays, this could get much worse. This keeps resolver super busy. To avoid this document layer explicitly adds a write conflict range covering all keys. In this case, adds the conflict range for prefix `employee:345:`. That would reduce the work for resolver 4 times, depending on the document size this could make or break.

If you already made sure that’s not the case, just ignore this 🙂

---

<div class="post-metadata">

**Author:** ![ricky.saltzer](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@ricky.saltzer](https://forums.foundationdb.org/u/ricky.saltzer)\
**Post date:** [May 30, 2019, 10:58pm UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/3 "2019-05-30T22:58:42Z")

</div>

Hey thanks for your response.

To be transparent, I am using RecordLayer for the workload.

I designed the model to have as few conflicts as possible. Last time I looked during the peak of the workload there was maybe 1-3Hz of conflicts.

---

<div class="post-metadata">

**Author:** ![alexmiller](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/alexmiller/32/326_2.png) [@alexmiller](https://forums.foundationdb.org/u/alexmiller)\
**Post date:** [May 31, 2019, 12:54am UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/4 "2019-05-31T00:54:11Z")

</div>

Can you `fdbcli> kill; kill all` and see if you get a second resolver? (or, really, anything to cause a recovery)

My suspicion is that BetterMasterExists didn’t catch that `set_class` changing a storage server to a resolver generates a cluster layout that better fits your desired configuration, and thus didn’t cause a recovery to trigger recruiting a second resolver. Otherwise, Evan and I don’t see anything in the recruitment logic that would explain why you aren’t getting a second resolver recruited. Admittedly, our operational tooling doesn’t use `set_class`, so it wouldn’t entirely surprise me if there was some small bug/bad interaction like this lurking.

---

<div class="post-metadata">

**Author:** ![ryanworl](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ryanworl/32/440_2.png) [@ryanworl](https://forums.foundationdb.org/u/ryanworl)\
**Post date:** [May 31, 2019, 1:00am UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/5 "2019-05-31T01:00:57Z")

</div>

This is what I recommended yesterday when this issue first came up for them, but not with that exact syntax. I just said kill -9 a proxy to trigger a recovery. They were on the three\_datacenter config (and now changed to triple) which may be related (not sure how that interacts with recruitment).

Is there anything that would be exposed in the logs to help diagnose this?

---

<div class="post-metadata">

**Author:** ![ricky.saltzer](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@ricky.saltzer](https://forums.foundationdb.org/u/ricky.saltzer)\
**Post date:** [May 31, 2019, 3:35am UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/6 "2019-05-31T03:35:14Z")

</div>

> [@alexmiller](#):
>
> Can you `fdbcli> kill; kill all` and see if you get a second resolver

That’s a neat trick for force reinitializing the database topology! Unfortunately, it doesn’t seem to solve the predicament I’m currently in.

```auto
>>> kill all
Attempted to kill 72 processes
...
$ fdbcli -C /tmp/fdb.cluster.grants --exec "status json" | grep -B3 '\"role\" : \"resolver\"'
                "roles" : [
                    {
                        "id" : "bc51311e0dedf06f",
                        "role" : "resolver"

```

Since this is becoming more and more likely attributed to a bug, I think it might be worth mentioning the origin of this database’s topology.

Back when we first started using FoundationDB for other use cases, we created a terraform script that orchestrated everything. Unfortunately, we didn’t fully understand the best way to deploy a cluster, and so, every node was homogeneous with the same configuration.

- **4 processes per machine** (three\_datacenter mode)
- 
  - (2) storage processes

- 
  - (1) transaction process

- 
  - (1) stateless process

With that being said…with 18 machines, we originally, we had **18** `stateless`, **18** `transaction` processes, and **36** `storage` processes.

We’ve since learned we don’t need so many `transaction` and `stateless` processes. And so, we downgraded to running only **7** `stateless` and **3** `transaction` processes.

Some of this was done via the `foundationdb.conf` and some were through the `setclass` command.

FWIW - Our other cluster which uses `three_datacenter` mode, and has 56 machines / 216 processes **still** only has **ONE** resolver. This cluster still uses our old topology, and so, there are a total of **54** `stateless` processes running inside it.

Thank you so much for your help!  
Ricky

---

<div class="post-metadata">

**Author:** ![ricky.saltzer](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@ricky.saltzer](https://forums.foundationdb.org/u/ricky.saltzer)\
**Post date:** [May 31, 2019, 3:39am UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/7 "2019-05-31T03:39:54Z")

</div>

> [@ryanworl](#):
>
> They were on the three\_datacenter config (and now changed to triple) which may be related (not sure how that interacts with recruitment)

To be fair, we were having the same issue when in `three_datacenter` mode. However, changing to `triple` did increase our write throughput. As you pointed out (off thread), this was due to the replication factor being decreased from `6x` to `3x`. Although, our issue remains the same…that is, our write throughput is bottlenecked at the resolver before anything else.

---

<div class="post-metadata">

**Author:** ![ricky.saltzer](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@ricky.saltzer](https://forums.foundationdb.org/u/ricky.saltzer)\
**Post date:** [May 31, 2019, 3:49am UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/8 "2019-05-31T03:49:00Z")

</div>

Since I’ve manually set some processes as a `resolution` class, I should set the record straight. Here is the final listing of the `setclass` (minus `storage` processes).

```auto
➜ /tmp fdbcli -C fdb.cluster.grants --exec "setclass" | grep -v 'storage'
There are currently 72 processes in the database:
  10.49.58.58:4500: stateless (set_class)
  10.49.58.58:4501: resolution (set_class)
  10.49.58.58:4502: transaction (command_line)
  10.49.58.58:4503: stateless (command_line)
  10.49.58.119:4500: stateless (set_class)
  10.49.58.119:4501: stateless (set_class)
  10.49.58.119:4502: transaction (command_line)
  10.49.58.119:4503: resolution (set_class)
  10.49.58.190:4500: stateless (set_class)
  10.49.58.190:4501: stateless (set_class)
  10.49.58.190:4502: transaction (command_line)
  10.49.58.190:4503: stateless (command_line)

```

---

<div class="post-metadata">

**Author:** ![ryanworl](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ryanworl/32/440_2.png) [@ryanworl](https://forums.foundationdb.org/u/ryanworl)\
**Post date:** [May 31, 2019, 1:31pm UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/9 "2019-05-31T13:31:00Z")

</div>

I did some investigation myself this morning with a 3 node cluster (each 2 cores). Spun it up, got coordinators into a good state, ran `configure triple memory` and verified transactions worked.

When I ran `configure resolvers=2; kill; kill all` with `unset` as the process class in a 3 process cluster, I only ever got one resolver.

When I ran `configure resolvers=2; kill; kill all` with `unset` as the process class for 3 processes and 1 additional process set as `resolution` (each VM is 2 cores), I still only ever got one resolver. The one chosen is the one process set to resolution class, thankfully.

When I ran `configure resolvers=2; kill; kill all` with `unset` as the process class for 3 processes and 2 process set as `resolution`, I finally got 2 resolvers.

I understand “Desired Resolvers” is not “Must Set This Number Resolvers”, but that behavior is not what I would have expected.

---

<div class="post-metadata">

**Author:** ![ricky.saltzer](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@ricky.saltzer](https://forums.foundationdb.org/u/ricky.saltzer)\
**Post date:** [May 31, 2019, 2:19pm UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/10 "2019-05-31T14:19:17Z")

</div>

> [@ryanworl](#):
>
> When I ran `configure resolvers=2; kill; kill all` with `unset` as the process class for 3 processes and 2 process set as `resolution` , I finally got 2 resolvers.

I’m having trouble reproducing this result.

[1]

```auto
➜ /tmp fdbcli -C fdb.cluster.grants --exec "setclass" | egrep 'resolution|unset'
  10.49.58.58:4500: unset (set_class)
  10.49.58.58:4501: resolution (set_class)
  10.49.58.58:4503: unset (set_class)
  10.49.58.119:4500: unset (set_class)
  10.49.58.119:4501: unset (set_class)
  10.49.58.119:4503: resolution (set_class)
  10.49.58.190:4500: unset (set_class)
  10.49.58.190:4501: unset (set_class)
  10.49.58.190:4503: unset (set_class)

```

[2]

```auto
➜ /tmp fdbcli -C fdb.cluster.grants --exec "configure resolvers=2; kill; kill all;"
>>> configure resolvers=2
Configuration changed
>>> kill
...
...
>>> kill all
Attempted to kill 72 processes

```

[3]

```auto
Configuration:
  Redundancy mode - triple
  Storage engine - ssd-2
  Coordinators - 7
  Desired Resolvers - 2

Cluster:
  FoundationDB processes - 72
  Machines - 18
  Memory availability - 3.7 GB per process on machine with least available
                           >>>>> (WARNING: 4.0 GB recommended) <<<<<
  Retransmissions rate - 8 Hz
  Fault Tolerance - 2 machines
  Server time - 05/31/19 10:18:16

Data:
  Replication health - Healthy (Rebalancing)
  Moving data - 0.000 GB
  Sum of key-value sizes - 3 MB
  Disk space used - 20.265 GB

Operating space:
  Storage server - 174.6 GB free on most full server
  Log server - 174.6 GB free on most full server

Workload:
  Read rate - 32 Hz
  Write rate - 1 Hz
  Transactions started - 6 Hz
  Transactions committed - 1 Hz
  Conflict rate - 0 Hz

Backup and DR:
  Running backups - 0
  Running DRs - 0

```

[4]

```auto
➜ /tmp fdbcli -C fdb.cluster.grants --exec "status json" | grep resolver
            "resolvers" : 2,
                        "role" : "resolver"

```

---

<div class="post-metadata">

**Author:** ![ryanworl](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ryanworl/32/440_2.png) [@ryanworl](https://forums.foundationdb.org/u/ryanworl)\
**Post date:** [May 31, 2019, 2:29pm UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/11 "2019-05-31T14:29:31Z")

</div>

Yes, I assume we’d already tried something equivalent to that previously. I just wanted to start from zero and see if I could actually make two resolvers happen. Still, very strange.

Edit: For anyone interested in investigating this as well, Ricky is on 6.0.15 and I just tested 6.1.8. If there are differences in how recruitment works introduced between those two that would be helpful to know.

---

<div class="post-metadata">

**Author:** ![ricky.saltzer](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@ricky.saltzer](https://forums.foundationdb.org/u/ricky.saltzer)\
**Post date:** [May 31, 2019, 7:28pm UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/12 "2019-05-31T19:28:02Z")

</div>

> [@ryanworl](#):
>
> Ricky is on 6.0.15 and I just tested 6.1.8

I upgraded the cluster but still stuck with one resolver ☹

---

<div class="post-metadata">

**Author:** ![alexmiller](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/alexmiller/32/326_2.png) [@alexmiller](https://forums.foundationdb.org/u/alexmiller)\
**Post date:** [June 1, 2019, 12:16am UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/13 "2019-06-01T00:16:31Z")

</div>

> [@ryanworl](#):
>
> They were on the three\_datacenter config (and now changed to triple) which may be related (not sure how that interacts with recruitment).

Ah, I had missed this.

You have the cluster controller and proxy recruited in one process that is

```auto
                "locality" : {
                    "dcid" : "7df98dcf291d4eed3c4e863c2f07a715",
                    "machineid" : "1e1b5d03332c24896d2abc1f03f83657",
                    "processid" : "99ac06f627656fe6f4d6aa05a69d0d7b",
                    "zoneid" : "1e1b5d03332c24896d2abc1f03f83657"
                },

```

your master and proxy are in

```auto
                "locality" : {
                    "dcid" : "7df98dcf291d4eed3c4e863c2f07a715",
                    "machineid" : "1e1b5d03332c24896d2abc1f03f83657",
                    "processid" : "8be186ec55e0f9cf9a2f0847ee08631b",
                    "zoneid" : "1e1b5d03332c24896d2abc1f03f83657"
                },

```

Your unused resolution class process is:

```auto
                "locality" : {
                    "dcid" : "643df5d94568f2f25772668935e05cf1",
                    "machineid" : "f942a3745a0086b2d1926a3bde011a2d",
                    "processid" : "411d623bf96e44b31c6516b87138e8a2",
                    "zoneid" : "f942a3745a0086b2d1926a3bde011a2d"
                },

```

and your used one is

```auto
                "locality" : {
                    "dcid" : "7df98dcf291d4eed3c4e863c2f07a715",
                    "machineid" : "1e1b5d03332c24896d2abc1f03f83657",
                    "processid" : "7b0abba967dd30eb4e86fbdaec217086",
                    "zoneid" : "1e1b5d03332c24896d2abc1f03f83657"
                },

```

So, everything that’s getting recruited is in dcid 7df98dcf291d4eed3c4e863c2f07a715

Which is intended. Definitely so for `three_datacenter`. I’m not actually sure how `triple` behaves with multiple `dcId`s specified, but apparently “like the above” is the answer. Recruiting proxies and resolvers across multiple datacenters would only increase latency (and likely decrease throughput) with little gain.

I’ll strongly suspect that if you take another dcId 7df98dcf291d4eed3c4e863c2f07a715 process and mark it as resolution, that you’ll get a second resolver. You would need to mark it as resolution and not stateless, because recruitment won’t let the process fit degrade with additional recruitments, so you’d either need to have multiple resolution processes or only stateless processes.

Relevant recruitment code: [https://github.com/apple/foundationdb/blob/b30fe29c9fc2ea23d679bf4fdc88daaea067a346/fdbserver/ClusterController.actor.cpp#L675](https://github.com/apple/foundationdb/blob/b30fe29c9fc2ea23d679bf4fdc88daaea067a346/fdbserver/ClusterController.actor.cpp#L675)

---

<div class="post-metadata">

**Author:** ![ricky.saltzer](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@ricky.saltzer](https://forums.foundationdb.org/u/ricky.saltzer)\
**Post date:** [June 1, 2019, 12:29am UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/14 "2019-06-01T00:29:57Z")

</div>

> [@alexmiller](#):
>
> I’ll strongly suspect that if you take another dcId 7df98dcf291d4eed3c4e863c2f07a715 process and mark it as resolution, that you’ll get a second resolver.

Totally worked!!!

```auto
➜ /tmp fdbcli -C fdb.cluster.grants --exec "status json" | grep 'resolution'
                "class_type" : "resolution",
                "class_type" : "resolution",

```

I guess this begs the question…why am I not able to have a resolver in each datacenter?

---

<div class="post-metadata">

**Author:** ![alexmiller](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/alexmiller/32/326_2.png) [@alexmiller](https://forums.foundationdb.org/u/alexmiller)\
**Post date:** [June 1, 2019, 12:35am UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/15 "2019-06-01T00:35:38Z")

</div>

I… don’t think you’d want a resolver in each datacenter?

In your cluster, you have your master, proxies, and resolver in the same datacenter. You’ll only incur a WAN round trip as part of writing to the TLogs, but that’s unavoidable, because the entire point of three\_datacenter mode is to have writes durably stored in two geographically distributed locations. Resolvers are stateless, so you gain no fault tolerance properties by having them geographically distributed.

If you added a resolver to another DC, then you’d have to doing a broadcast and waiting for replies to the resolvers would be one WAN roundtrip, and then you’d pay a second for the WAN TLog roundtrip.

I mean, yeah, maybe it’d be nice if somewhere a message was surfaced for the administrator that says “Actual resolvers less than desired resolvers. Please add more resolution/stateless processes to dcId NNN”. But I’m still of the opinion that automatically including resolvers in far away DCs for the transaction system isn’t universally desirable.

---

<div class="post-metadata">

**Author:** ![ryanworl](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ryanworl/32/440_2.png) [@ryanworl](https://forums.foundationdb.org/u/ryanworl)\
**Post date:** [June 1, 2019, 12:36am UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/16 "2019-06-01T00:36:58Z")

</div>

This cluster is deployed with three AZs in one region as the “datacenters”. That is why he was hoping to recruit resolvers in each datacenter, but I don’t think three\_datacenter was intended for that use case.

---

<div class="post-metadata">

**Author:** ![ricky.saltzer](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@ricky.saltzer](https://forums.foundationdb.org/u/ricky.saltzer)\
**Post date:** [June 1, 2019, 12:39am UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/17 "2019-06-01T00:39:23Z")

</div>

> [@alexmiller](#):
>
> You’ll only incur a WAN round trip as part of writing to the TLogs, but that’s unavoidable, because the entire point of three\_datacenter mode is to have writes durably stored in two geographically distributed locations. Resolvers are stateless, so you gain no fault tolerance properties by having them geographically distributed.

Totally makes sense. Thank you for clearing this up for me.

---

<div class="post-metadata">

**Author:** ![ricky.saltzer](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@ricky.saltzer](https://forums.foundationdb.org/u/ricky.saltzer)\
**Post date:** [June 1, 2019, 12:40am UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/18 "2019-06-01T00:40:12Z")

</div>

Do we think there is a bug where triple replication respects the `dcid` when it really shouldn’t?

---

<div class="post-metadata">

**Author:** ![alexmiller](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/alexmiller/32/326_2.png) [@alexmiller](https://forums.foundationdb.org/u/alexmiller)\
**Post date:** [June 1, 2019, 12:51am UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/19 "2019-06-01T00:51:10Z")

</div>

> [@ryanworl](#):
>
> This cluster is deployed with three AZs in one region as the “datacenters”. That is why he was hoping to recruit resolvers in each datacenter, but I don’t think three\_datacenter was intended for that use case.

With the goal of making the process layout on each AZ more symmetric? IIRC, AZ to AZ communication has a slightly higher price and latency, so my point still somewhat stands.

If the concern is about reserving two resolution processes in each AZ when you’d only ever use 1/3 of that, then I don’t have a great answer to that. You can run `stateless` processes instead of `resolution`, with likely no meaningful change to the resulting cluster’s behavior as long as you supply enough stateless processes so that proxies and resolvers don’t share. You can also technically not run stateless or resolution class processes in one of the three AZs, because if you lose one AZ you’d still be assured to have one left that can run your transaction subsystem.

… we’ve also totally talked through Ricky’s deployment before, and apparently all the details had left my head. Maybe I should go change my logo to a goldfish…

> [@ricky.saltzer](#):
>
> Do we think there is a bug where triple replication respects the `dcid` when it really shouldn’t?

That’s what I’ve been thinking through now, and I haven’t come to a conclusion. My mental model of `triple` is that it should only care about `zoneId`, so I’m assuming that this was accidentally introduced when adding `three_datacenter` or multi-region, and they assumed that you wouldn’t specify dcId for a cluster that’s triple replicated. On the other hand, I feel like `triple` sort of did the “right” thing here, in that it “helped” you avoid a potentially sub-optimal database configuration. My idiomatic and pragmatic sides are at war.

---

<div class="post-metadata">

**Author:** ![ricky.saltzer](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@ricky.saltzer](https://forums.foundationdb.org/u/ricky.saltzer)\
**Post date:** [June 1, 2019, 3:32pm UTC](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414/20 "2019-06-01T15:32:50Z")

</div>

> [@alexmiller](#):
>
> On the other hand, I feel like `triple` sort of did the “right” thing here, in that it “helped” you avoid a potentially sub-optimal database configuration.

That’s interesting. I wonder if `triple` replication ensures that the other two replicas are stored in separate `dcId` storage processes.

[Next page](https://forums.foundationdb.org/t/configuring-foundationdb-to-use-more-than-one-resolver/1414.md?page=2)