Hello!
We’ve recently started testing a use case that involves a whooole lot of watches. When we turned on the part of the system that uses watches (which also turned on a number of other things that could be contributing to this situation), we saw that the query queue depth on our storage processes started to climb steadily.
When we halted the experiment and restarted all of our clients, query queue depth stopped growing, but also didn’t come down after a little more than an hour. We’re looking at query_queue_max and understand that should be reporting the max per “reporting interval” (i.e. the time between status updates, so closer to “minutes” or “seconds” than “the lifetime of the process”), but if that’s incorrect, please let me know!
Our hypothesis (informed by An Overview how Watches Work · apple/foundationdb Wiki · GitHub) is that the query queue entries are actually watches that didn’t get cleaned up properly by clients. We’re a little surprised that the count hasn’t come back down, though.
As a first question: is it plausible that what we’re seeing in the query queue is all watches? And as a second question, is it strange that the queue depth hasn’t come back down?
We’re also a little surprised that this queue depth growth doesn’t seem to be causing any performance issues (at least that we’ve found so far). Is there some explanation for what we’re seeing where this is all just benign noise and we shouldn’t worry about it?
Thanks kindly!
