# What does FetchKeysTooLong trace event denote?

**URL:** <https://forums.foundationdb.org/t/what-does-fetchkeystoolong-trace-event-denote/2042>\
**Category:** Using FoundationDB\
**Created:** [April 1, 2020, 6:36pm UTC](https://forums.foundationdb.org/t/what-does-fetchkeystoolong-trace-event-denote/2042 "2020-04-01T18:36:08Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![tuk](https://avatars.discourse-cdn.com/v4/letter/t/b5ac83/32.png) [@tuk](https://forums.foundationdb.org/u/tuk)\
**Post date:** [April 1, 2020, 6:36pm UTC](https://forums.foundationdb.org/t/what-does-fetchkeystoolong-trace-event-denote/2042/1 "2020-04-01T18:36:08Z")

</div>

I am seeing few trace events like below in a deployment

`<Event Severity="30" Time="1585765420.463122" Type="FetchKeysTooLong" ID="0000000000000000" Duration="32400" Phase="0" Begin="\x151\x15\x09\x15&amp;3\x00\x00\x07l6?\xf5}\x00\x00\x00\x00\x15\x14" End="\x151\x15\x09\x15&amp;3\x00\x00\x07l7\xb8\xd1\xb7\x00\x00\x00\x00\x15S" Machine="a.b.c.d:xxxx" LogGroup="default" Roles="SS,TL" />`

Can someone let me know what does this event denote?

I have read [Understanding Log Files.](https://github.com/apple/foundationdb/wiki/Understanding-FoundationDB-Log-Files). Is there any other documentation that I can refer to get more details about different types of events?

Foundation DB Version - 6.2.11.

---

<div class="post-metadata">

**Author:** ![alexmiller](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/alexmiller/32/326_2.png) [@alexmiller](https://forums.foundationdb.org/u/alexmiller)\
**Post date:** [April 2, 2020, 4:35am UTC](https://forums.foundationdb.org/t/what-does-fetchkeystoolong-trace-event-denote/2042/2 "2020-04-02T04:35:11Z")

</div>

FetchKeys is the process of one storage server fetching a key range from another storage server as part of a data distribution action. FetchKeysTooLong is letting you know that reading the key range from some other server took longer than expected.

There’s a variety of reasons as to why this could be. Your storage servers could have been saturated with client reads for some time. Storage servers try to fetch as of their current version, so if this particular storage server was behind for some reason (which seems suspicious, because it’s both a transaction log and storage server), then that could be why fetch keys took a long time.

Alone, I wouldn’t worry about seeing this once. It’s more useful in context when trying to dig into higher level issues. For instance, it would help explain why a shard movement took a long time, why restoring fault tolerance in a cluster took a while, etc.

Understanding Log Files is the (very) work-in-progress page that was being used to collect write-ups of various trace events. That is the current best (and not very good) documentation that we have on this subject.

---

<div class="post-metadata">

**Author:** ![tuk](https://avatars.discourse-cdn.com/v4/letter/t/b5ac83/32.png) [@tuk](https://forums.foundationdb.org/u/tuk)\
**Post date:** [April 2, 2020, 10:22am UTC](https://forums.foundationdb.org/t/what-does-fetchkeystoolong-trace-event-denote/2042/3 "2020-04-02T10:22:47Z")

</div>

Thanks @alexmiller for replying. Can you let me know what does `Phase` denote in `FetchKeysTooLong` event?

Can you also let me know how to read the `SlowTask` events? For example in an event like below how to know this slow task is related to what? I see a `TaskID` but not sure how to make sense out of it.

```auto
<Event Severity="30" Time="1585810867.802134" Type="SlowTask" ID="0000000000000000" TaskID="10000" MClocks="576.392" Duration="0.261998" SampleRate="1" NumYields="0" Machine="a.b.c.d:xxxx" LogGroup="default" Roles="CD,SS" />

```

---

<div class="post-metadata">

**Author:** ![tuk](https://avatars.discourse-cdn.com/v4/letter/t/b5ac83/32.png) [@tuk](https://forums.foundationdb.org/u/tuk)\
**Post date:** [April 6, 2020, 4:43am UTC](https://forums.foundationdb.org/t/what-does-fetchkeystoolong-trace-event-denote/2042/4 "2020-04-06T04:43:37Z")

</div>

Anyone any pointers on my below query?

> Can you let me know what does `Phase` denote in `FetchKeysTooLong` event?
> 
> Can you also let me know how to read the `SlowTask` events? For example in an event like below how to know this slow task is related to what? I see a `TaskID` but not sure how to make sense out of it.
> 
> ```auto
> <Event Severity="30" Time="1585810867.802134" Type="SlowTask" ID="0000000000000000" TaskID="10000" MClocks="576.392" Duration="0.261998" SampleRate="1" NumYields="0" Machine="a.b.c.d:xxxx" LogGroup="default" Roles="CD,SS" />
> 
> ```

---

<div class="post-metadata">

**Author:** ![alexmiller](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/alexmiller/32/326_2.png) [@alexmiller](https://forums.foundationdb.org/u/alexmiller)\
**Post date:** [April 6, 2020, 8:20pm UTC](https://forums.foundationdb.org/t/what-does-fetchkeystoolong-trace-event-denote/2042/5 "2020-04-06T20:20:47Z")

</div>

(Sorry, a lot of us are focusing on trying to finish off work for FDB6.3, so response time might be a bit worse.)

> [@tuk](#):
>
> Can you let me know what does `Phase` denote in `FetchKeysTooLong` event?

`Phase` is defined as `enum Phase { WaitPrevious, Fetching, Waiting };`

> [@tuk](#):
>
> Can you also let me know how to read the `SlowTask` events? For example in an event like below how to know this slow task is related to what? I see a `TaskID` but not sure how to make sense out of it.

TaskID maps to [TaskPriority](https://github.com/apple/foundationdb/blob/8a85ff02bf942bdad7fc76ef222109c9a330e147/flow/network.h#L36), which can give a vague idea of what the task is that’s taking a while. I’d suggest instead looking for `Net2SlowTaskTrace` events, which will give you a backtrace.