# How to check if data is fully replicated?

**URL:** <https://forums.foundationdb.org/t/how-to-check-if-data-is-fully-replicated/1514>\
**Category:** Using FoundationDB\
**Created:** [July 10, 2019, 9:06pm UTC](https://forums.foundationdb.org/t/how-to-check-if-data-is-fully-replicated/1514 "2019-07-10T21:06:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rayjcwu](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/rayjcwu/32/602_2.png) [@rayjcwu](https://forums.foundationdb.org/u/rayjcwu)\
**Post date:** [July 10, 2019, 9:06pm UTC](https://forums.foundationdb.org/t/how-to-check-if-data-is-fully-replicated/1514/1 "2019-07-10T21:06:22Z")

</div>

I’m checking if there are other things valuable in the `fdbcli --exec 'status json'` output

`cluster.data.state.name`: I think this is “Replication health” in `status` output. Does “healthy” also mean data is fully replicated?

`cluster.data.team_trackers.state.name`: What does this mean? It has the same enums of values as `cluster.data.state.name`. What does “team” mean in FoundationDB?

`cluster.data.full_replication`: I think this means data is fully replicated? If so, is there any difference between this and `cluster.data.state.name == 'healthy'`?

`cluster.recovery_state.name`: Does “fully\_recovered” mean cluster is ready to accept transactions, but it’s irrelevant if data in the cluster is healthy or not?

Assume we want to replace a failed machine programmatically, after we exclude a machine (via fdbcli), what field should we monitor to make sure it’s safe to terminate that instance?

Also, is there other fields to check to make sure cluster is in a “healthy” state?

---

<div class="post-metadata">

**Author:** ![mengxu](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/mengxu/32/893_2.png) [@mengxu](https://forums.foundationdb.org/u/mengxu)\
**Post date:** [July 11, 2019, 10:03pm UTC](https://forums.foundationdb.org/t/how-to-check-if-data-is-fully-replicated/1514/2 "2019-07-11T22:03:44Z")

</div>

> [@rayjcwu](#):
>
> `cluster.data.state.name` : I think this is “Replication health” in `status` output. Does “healthy” also mean data is fully replicated?

If the `replication health` shows `healthy`, then any data has the configured number of replicas.  
The code related to this is defined in [https://github.com/apple/foundationdb/blob/master/fdbserver/Status.actor.cpp#L1237](https://github.com/apple/foundationdb/blob/master/fdbserver/Status.actor.cpp#L1237)

A team is a collection of `k` storage servers (where `k` is the replication factor). We only build a limited number of teams to host data, so that when `k` servers or machines fail at the same time, the possibility of data unavailable event is super low.

---

<div class="post-metadata">

**Author:** ![alexmiller](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/alexmiller/32/326_2.png) [@alexmiller](https://forums.foundationdb.org/u/alexmiller)\
**Post date:** [July 12, 2019, 9:41pm UTC](https://forums.foundationdb.org/t/how-to-check-if-data-is-fully-replicated/1514/3 "2019-07-12T21:41:48Z")

</div>

(Sorry, your questions require reading code to answer with confidence what the fields mean, which I haven’t had time to do. I had typed out and not posted a reply to the spirit of your question before, which I’ll post now, and return to the rest of your question when I do have time.)

> [@rayjcwu](#):
>
> Assume we want to replace a failed machine programmatically, after we exclude a machine (via fdbcli), what field should we monitor to make sure it’s safe to terminate that instance?

I think this is the crux of your question, so I’ll start with this. You should terminate the instance when `exclude` tells you that it is safe to do so. Exclude synchronously waits for the data distribution off of the failed machine to complete, and returns once it does.

Using the example from [Removing machines from a cluster](https://apple.github.io/foundationdb/administration.html#removing-machines-from-a-cluster)

```auto
fdb> exclude 1.2.3.4 1.2.3.5 1.2.3.6
Waiting for state to be removed from all excluded servers. This may take a while.
[a long wait]
It is now safe to remove these machines or processes from the cluster.

```

And there’s also the note that if you ^C out of an exclude, re-issuing the same exclude will allow you to resume the wait.

I’d recommend using this over the other proposed methods of determining replication health, because

1. It’s more specific to the particular machine, and won’t include other data distribution activities (splits, merges, etc.)
2. It’s what our automation uses.

> [@rayjcwu](#):
>
> `cluster.data.team_trackers.state.name` : What does this mean? It has the same enums of values as `cluster.data.state.name` . What does “team” mean in FoundationDB?

“Team” is a grouping of `Replication Factor` number of hosts. Shards of data are assigned to a team. Part of data distribution builds and maintains these teams. I’m sure @mengxu would be happy to answer any questions you have about team building 😉

EDIT: turns out he already did