Lets say I register a watch on a key. Based on the changes in the key my watch triggers a function. My app runs on JVM.
What will happen if my JVM dies - will watch registration be lost?
And, do watches conform to the replication pattern (in distributed environment) as data in the FDB do?
Typically, both the client and a storage server are involved in monitoring the watch. In the normal path, what happens is that a watch request gets sent to a single storage server that is responsible for the key. The client waits for a response from this server, and if it gets notified that the value changed or some types of errors occurred, it will notify the user.
If the storage server that holds the watch fails, has the watch open for a long time, has too many watches, or has the data moved away, etc, then the client will be notified of this and can try issuing its watch again, possibly to a different storage server. In the case where the storage servers have reached the watch limit, this essentially falls back to the client running a polling loop.
If the client dies, there’s no way for you to be notified when the watch triggers. The storage server watch timeout that I alluded to above is used to deal with this scenario, as it results in the watch eventually being flushed from the storage server.
Thanks @ajbeamon, this is helpful.
Does data lag have any effect, here? If the storage server handling my watch has 15 minutes of data lag, would I expect the watch to trigger 15 minutes “late”?
There are actually a variety of scenarios that could occur here, and it seems they would have slightly different behavior. I’m not sure if it’s intended that these states be treated differently, but I’ll summarize what I’ve found.
If you try to set a watch on a storage server that’s already behind, it looks like it’s going to trigger the watch with a future_version error. I suspect the expectation in this case is that you would read the key again and reset the watch if you still need it, in which case your request will go to a random choice from the storage servers available for that key. Assuming at least one of them is caught up, your watch should eventually get set. That said, the storage server will eventually timeout and return to the client even if the key doesn’t change, and behind the scenes the client will reissue the watch to another random storage server. This means you might later set the watch on a storage server that’s behind and get a future_version error.
If all of the storage servers are behind for the key you are trying to watch, then setting the watch will just repeatedly return the future_version error. If you are following the recommended procedure of reading the key you are intending to watch in the same transaction, though, your read will fail before ever trying to set the watch.
In the case that you set a watch on a storage server while it’s caught up and it then falls behind, it looks like the watch will remain set and trigger late. If the watch goes through a timeout cycle, though, it will get reissued and either fail with the future_version error or be set on a different storage server that’s caught up.
There’s also one other weird case when the storage server is behind, which is that if a storage server meets some criteria after the watch has been set, it might return a process_behind error, in which case the watch will automatically reissue on the client. It doesn’t look like this behavior is guaranteed, though.
2 Likes