# How to do fencing in FoundationDB

**URL:** <https://forums.foundationdb.org/t/how-to-do-fencing-in-foundationdb/2237>\
**Category:** Using FoundationDB\
**Created:** [July 9, 2020, 11:25am UTC](https://forums.foundationdb.org/t/how-to-do-fencing-in-foundationdb/2237 "2020-07-09T11:25:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ppggff](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ppggff/32/855_2.png) [@ppggff](https://forums.foundationdb.org/u/ppggff)\
**Post date:** [July 9, 2020, 11:25am UTC](https://forums.foundationdb.org/t/how-to-do-fencing-in-foundationdb/2237/1 "2020-07-09T11:25:23Z")

</div>

Hi, we use foundationDB as our storage server for multiple clients.  
In our system, each client access a private range of keys. If one  
client failed, system may create a new client for that range, or  
choose an idle client for that range.

If the failed client accidentally access fdb again, it may overwrite  
changes made by the new client, then cause data consistency problem  
(knows as split-brain).

To prevent this happens, we want to implement some kind of fencing  
in fdb. After search and learn, we came up with 2 options:

1. Make a fencing key, and check if it’value has changed in every write txn.  
Once replaced a client, update that fencing key to disable the failed client.

2. Also make a fencing key, then, in the write txn:

Are there any problems with either method? Are there other, better ways to do it?  
Thanks

---

<div class="post-metadata">

**Author:** ![gaurav](https://avatars.discourse-cdn.com/v4/letter/g/b487fb/32.png) [@gaurav](https://forums.foundationdb.org/u/gaurav)\
**Post date:** [July 9, 2020, 12:41pm UTC](https://forums.foundationdb.org/t/how-to-do-fencing-in-foundationdb/2237/2 "2020-07-09T12:41:22Z")

</div>

First approach is simpler and would result in adding 2-3 ms per transaction to fetch the fencing key. You can avoid the penalty if in the client code you don’t wait on the fencing value immediately, but Concurrently continue with other reads/writes, and then later check and abort the tx if the key has changed. If the fencing key changes are rare, this will not result in abortions, and thus a practically free operation.

Second approach, as I understand, is an optimization to cache the fencing key and the associated read version. This will work, but adds some complications where a client’s subsequent transactions may conflict with its older transactions, due to older read\_version being used. This also results in bypassing ratelimiter and its ability, to some extent, to throttle down transactions under load.

How frequent are changes to fencing keys? If this is not very frequent then another optimization can be used: There is a [feature](https://forums.foundationdb.org/t/a-new-tool-for-managing-layer-metadata/1191) by which you can get a fixed metadata key as part of every transaction for free.  
You can transactionally change this global metadata key when changing any fencing key. Clients can compare changes to this metadata key against a locally cached value, and can then fetch the particular fencing key if the global metadata key has changed.

---

<div class="post-metadata">

**Author:** ![ppggff](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ppggff/32/855_2.png) [@ppggff](https://forums.foundationdb.org/u/ppggff)\
**Post date:** [July 10, 2020, 8:27am UTC](https://forums.foundationdb.org/t/how-to-do-fencing-in-foundationdb/2237/3 "2020-07-10T08:27:35Z")

</div>

Thanks a lot.

It seem the second approach may cause some problem. I think we will go for the first approach with checking the fencing key at end of each write transaction. Later, after upgrading to newer version of fdb, we will try to use the metadata key.

For metadata key, if someone else changed it while current transaction is checking it, will the transaction later report an error on commit? (because the metadata key is part of the conflict keys)

---

<div class="post-metadata">

**Author:** ![gaurav](https://avatars.discourse-cdn.com/v4/letter/g/b487fb/32.png) [@gaurav](https://forums.foundationdb.org/u/gaurav)\
**Post date:** [July 10, 2020, 9:45am UTC](https://forums.foundationdb.org/t/how-to-do-fencing-in-foundationdb/2237/4 "2020-07-10T09:45:18Z")

</div>

> [@ppggff](#):
>
> For metadata key, if someone else changed it while current transaction is checking it, will the transaction later report an error on commit? (because the metadata key is part of the conflict keys)

I am almost certain it will cause a conflict, but haven’t used this feature so far. You can try it out in a local test.

> [@ppggff](#):
>
> checking the fencing key at end of each write transaction

Just to be clear - I was suggesting to fire a `get` for the key at beginning of transaction, but extract the value out of future towards end of transaction.