# Knobs to control data relocation?

**URL:** <https://forums.foundationdb.org/t/knobs-to-control-data-relocation/737>\
**Category:** Using FoundationDB\
**Created:** [September 29, 2018, 1:15am UTC](https://forums.foundationdb.org/t/knobs-to-control-data-relocation/737 "2018-09-29T01:15:30Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![panghy](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/panghy/32/19_2.png) [@panghy](https://forums.foundationdb.org/u/panghy)\
**Post date:** [September 29, 2018, 1:15am UTC](https://forums.foundationdb.org/t/knobs-to-control-data-relocation/737/1 "2018-09-29T01:15:30Z")

</div>

We use RELOCATION\_PARALLELISM\_PER\_SOURCE\_SERVER to control data movement in the past but is that still the right way to do it? Basically, we want to reduce the urgency for the cluster to heal itself when it has one node left because it can cause logs queues to go high enough that cluster throughput suffers.

---

<div class="post-metadata">

**Author:** ![ashishnm](https://avatars.discourse-cdn.com/v4/letter/a/b5e925/32.png) [@ashishnm](https://forums.foundationdb.org/u/ashishnm)\
**Post date:** [September 29, 2018, 3:40pm UTC](https://forums.foundationdb.org/t/knobs-to-control-data-relocation/737/2 "2018-09-29T15:40:08Z")

</div>

We use the following 2 knobs on clusters esp. where we are IOPS limited.  
knob\_relocation\_parallelism\_per\_source\_server: 2  
knob\_fetch\_keys\_parallelism\_bytes: 4000000

---

<div class="post-metadata">

**Author:** ![panghy](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/panghy/32/19_2.png) [@panghy](https://forums.foundationdb.org/u/panghy)\
**Post date:** [October 4, 2018, 7:51am UTC](https://forums.foundationdb.org/t/knobs-to-control-data-relocation/737/3 "2018-10-04T07:51:27Z")

</div>

yeah, we have those but it doesn’t seem to be able to control log queues (storage queues are ok).

---

<div class="post-metadata">

**Author:** ![ajbeamon](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ajbeamon/32/13_2.png) [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)\
**Post date:** [October 5, 2018, 6:37pm UTC](https://forums.foundationdb.org/t/knobs-to-control-data-relocation/737/4 "2018-10-05T18:37:32Z")

</div>

How large are the log queues growing? They are expected to grow during a failure up to at least 1.5 GB.

---

<div class="post-metadata">

**Author:** ![panghy](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/panghy/32/19_2.png) [@panghy](https://forums.foundationdb.org/u/panghy)\
**Post date:** [October 5, 2018, 11:28pm UTC](https://forums.foundationdb.org/t/knobs-to-control-data-relocation/737/5 "2018-10-05T23:28:30Z")

</div>

Yeah but that causes enough slowness (ratekeeper) that latencies from reads and writes are noticeable. Reducing the impact of the healing (not as aggressive to the point where there’s little headroom for the cluster to handle a spike in traffic for instance) is what we’re after.

---

<div class="post-metadata">

**Author:** ![panghy](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/panghy/32/19_2.png) [@panghy](https://forums.foundationdb.org/u/panghy)\
**Post date:** [October 5, 2018, 11:53pm UTC](https://forums.foundationdb.org/t/knobs-to-control-data-relocation/737/6 "2018-10-05T23:53:24Z")

</div>

![25%20PM](https://global.discourse-cdn.com/foundationdb/original/1X/89fd3e135056536a9613a88a0e333302ca785bde.png)

A simulated failure of a node that basically pegged tlogs to 1.8G or above which means any additional write load could cause latencies to spike. Contrast that with just adding a new node (which results in low-priority moves) and it doesn’t have the same impact to the cluster.