# Any way to gracefully stop fdbserver process temporarily without affecting traffic?

**URL:** <https://forums.foundationdb.org/t/any-way-to-gracefully-stop-fdbserver-process-temporarily-without-affecting-traffic/1643>\
**Category:** Using FoundationDB\
**Created:** [September 23, 2019, 3:19pm UTC](https://forums.foundationdb.org/t/any-way-to-gracefully-stop-fdbserver-process-temporarily-without-affecting-traffic/1643 "2019-09-23T15:19:26Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![arob](https://avatars.discourse-cdn.com/v4/letter/a/9fc29f/32.png) [@arob](https://forums.foundationdb.org/u/arob)\
**Post date:** [September 23, 2019, 3:19pm UTC](https://forums.foundationdb.org/t/any-way-to-gracefully-stop-fdbserver-process-temporarily-without-affecting-traffic/1643/1 "2019-09-23T15:19:26Z")

</div>

Sorry if I’m just missing something from the docs, but I’ve noticed that running `sudo service foundationdb stop` to temporarily shut off one of the fdbserver processes in my cluster can cause roughly 5 seconds of downtime depending on which roles the fdbserver was performing within the cluster.

I know that the `exclude` command helps when you want to permanently remove a machine from the cluster, but my understanding is that it moves all the data off the machine. If you just want to have the machine offline for a minute or two to upgrade the OS, it seems expensive to move all the data rather than just running for a minute or two without the server and catching up when it comes back online.

Am I understanding things correctly? Is `exclude` the recommended tool for this sort of situation or is there any other way to avoid blips of unavailability when taking a server offline temporarily?

---

<div class="post-metadata">

**Author:** ![alexmiller](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/alexmiller/32/326_2.png) [@alexmiller](https://forums.foundationdb.org/u/alexmiller)\
**Post date:** [September 23, 2019, 11:53pm UTC](https://forums.foundationdb.org/t/any-way-to-gracefully-stop-fdbserver-process-temporarily-without-affecting-traffic/1643/2 "2019-09-23T23:53:55Z")

</div>

There was a maintenance command added to `fdbcli` that makes data distribution ignore failures in a zone for a span of time. You could probably use this before you do the OS upgrades?

```nohighlight
fdb> help maintenance

maintenance [on|off] [ZONEID] [SECONDS]

Mark a zone for maintenance.

Calling this command with `on' prevents data distribution from moving data away
from the processes with the specified ZONEID. Data distribution will
automatically be turned back on for ZONEID after the specified SECONDS have
elapsed, or after a storage server with a different ZONEID fails. Only one
ZONEID can be marked for maintenance. Calling this command with no arguments
will display any ongoing maintenance. Calling this command with `off' will
disable maintenance.

```

---

<div class="post-metadata">

**Author:** ![ajbeamon](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ajbeamon/32/13_2.png) [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)\
**Post date:** [September 24, 2019, 1:08am UTC](https://forums.foundationdb.org/t/any-way-to-gracefully-stop-fdbserver-process-temporarily-without-affecting-traffic/1643/3 "2019-09-24T01:08:13Z")

</div>

I’m not sure if maintenance mode itself moves off the various roles that contribute to the downtime that’s trying to be avoided. If not, maybe combining maintenance mode with exclusions would work.

---

<div class="post-metadata">

**Author:** ![arob](https://avatars.discourse-cdn.com/v4/letter/a/9fc29f/32.png) [@arob](https://forums.foundationdb.org/u/arob)\
**Post date:** [September 26, 2019, 8:05pm UTC](https://forums.foundationdb.org/t/any-way-to-gracefully-stop-fdbserver-process-temporarily-without-affecting-traffic/1643/4 "2019-09-26T20:05:52Z")

</div>

Yeah, it sounds like maintenance mode would help avoid data movement, but not short periods of unavailability when the roles need to be reconfigured. I can try out maintenance mode plus exclusions, but I’m pretty surprised to hear this isn’t something that there’s a clear answer for.

---

<div class="post-metadata">

**Author:** ![alexmiller](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/alexmiller/32/326_2.png) [@alexmiller](https://forums.foundationdb.org/u/alexmiller)\
**Post date:** [September 26, 2019, 8:17pm UTC](https://forums.foundationdb.org/t/any-way-to-gracefully-stop-fdbserver-process-temporarily-without-affecting-traffic/1643/5 "2019-09-26T20:17:40Z")

</div>

How long of a blip are you seeing?

I had been thinking through this as that you were looking to avoid a recovery, which is sort of unavoidable. If there’s a transaction log/proxy/resolver on the host that you’re removing, then there’s a necessary function that is being performed on that host that _needs_ to be moved for the cluster to keep working. However, the cluster voluntarily moving processes elsewhere should result in a pretty quick unavailability window (\<1s).

I’m guessing that your actual comment here is that what you’re seeing is a 3-5 second recovery because you first have to wait out the failure detector, and then do a recovery. So what you want is something that’s like an `exclude`+`maintenance` command, that causes FDB to move all roles except storage servers off of the host, in a quick voluntariy recovery, and then ignore the storage servers failing on that host for the next 10min ?

Re-reading AJ’s reply, I think he read this the correct way, and overall it seems to make sense that `maintenance` should act like an exclude in terms of changing recruitement’s opinion of where is a good place to put the various pieces of the transaction authority.

---

<div class="post-metadata">

**Author:** ![ajbeamon](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ajbeamon/32/13_2.png) [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)\
**Post date:** [September 26, 2019, 8:24pm UTC](https://forums.foundationdb.org/t/any-way-to-gracefully-stop-fdbserver-process-temporarily-without-affecting-traffic/1643/6 "2019-09-26T20:24:35Z")

</div>

For what it’s worth, I’m not 100% sure whether maintenance mode has an effect on recruited processes, so that’s probably worth looking into or experimenting with to determine the answer.

---

<div class="post-metadata">

**Author:** ![arob](https://avatars.discourse-cdn.com/v4/letter/a/9fc29f/32.png) [@arob](https://forums.foundationdb.org/u/arob)\
**Post date:** [September 26, 2019, 8:40pm UTC](https://forums.foundationdb.org/t/any-way-to-gracefully-stop-fdbserver-process-temporarily-without-affecting-traffic/1643/7 "2019-09-26T20:40:26Z")

</div>

> [@alexmiller](#):
>
> How long of a blip are you seeing?

I’ve been seeing a blip of roughly 5 seconds when taking processes down by running `service foundationdb stop`, which should just be sending a SIGTERM. I’d have hoped that the SIGTERM handler would be able to initiate a role transfer.

> [@alexmiller](#):
>
> I’m guessing that your actual comment here is that what you’re seeing is a 3-5 second recovery because you first have to wait out the failure detector, and then do a recovery. So what you want is something that’s like an `exclude` + `maintenance` command, that causes FDB to move all roles except storage servers off of the host, in a quick voluntariy recovery, and then ignore the storage servers failing on that host for the next 10min ?

Yeah, that’s the dream.

I’ll experiment with this, not just blindly trust that it works 🙂 Thanks.

---

<div class="post-metadata">

**Author:** ![john\_brownlee](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/john_brownlee/32/22_2.png) [@john\_brownlee](https://forums.foundationdb.org/u/john_brownlee)\
**Post date:** [December 9, 2019, 9:18pm UTC](https://forums.foundationdb.org/t/any-way-to-gracefully-stop-fdbserver-process-temporarily-without-affecting-traffic/1643/8 "2019-12-09T21:18:01Z")

</div>

@ajbeamon Were you able to confirm whether maintenance mode moves roles off of the processes under maintenance? I think that would be the best behavior, but I don’t know enough about the technical details to say for sure.

---

<div class="post-metadata">

**Author:** ![ajbeamon](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ajbeamon/32/13_2.png) [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)\
**Post date:** [December 9, 2019, 9:45pm UTC](https://forums.foundationdb.org/t/any-way-to-gracefully-stop-fdbserver-process-temporarily-without-affecting-traffic/1643/9 "2019-12-09T21:45:34Z")

</div>

It doesn’t look like maintenance mode has any impact on the roles recruited on existing processes or whether processes can be used to recruit new roles.