Your intuition is correct that watches cannot be used to provide a stream of all mutations to a key, as there could be some mutations that slip through undetected. You can see some details for how a watch works here, but the basic idea is that it is guaranteed to fire at some point after the value of the watched key no longer matches the value of the key as seen in the transaction that set it. It won’t tell you what the new value is or how many times it changed, and it won’t remain in effect after it’s fired, meaning you’ll have to create a new watch if you want to be notified of further changes.
I’ll outline another approach here which could be used to accomplish what you’ve described. The idea is that you’d maintain some sort of per-queue counter that gets updated every time an item is added to a queue and then set watches on those counters. Your listeners would start up and process all events currently in the queue (or ignore them, whatever you want it do), remembering the last item that it processed. In the same transaction, you would also set a watch on the queue counter, noting its current value.
When the watch is triggered (successfully or with an error), you would check whether the value of the counter has increased, process any new events in the queue, and set a new watch on the counter, again in the same transaction.
Thus, your listeners would be notified every time the queue has changed from the state they last remembered, and then they would be responsible for picking up where they left off. Your producers would need to update the counters each time an item was added to the queue, probably using the atomic ADD mutation.
There are a few things to note about the above scheme:
-
If there is a high rate of mutations to a queue counter, the counter may need to be sharded over multiple keys so that a single key isn’t getting hammered.
-
You may need some mechanism to clean up events once they have been processed by all listeners.
-
You need to be careful about how you do your reads to avoid unnecessary conflicts. With the above scheme, you’ll probably want reads of the counter keys to be snapshot reads so that simultaneous queue insertions won’t interfere with listeners. You’ll still know about simultaneous updates because the watches you set will fire if the value of the counter doesn’t match what was read in the transaction. You’ll similarly want to avoid setting conflict ranges past the last item in the queue that you process, which can be done by reading the queue with a snapshot range read and then manually applying conflict ranges later (either one for each item, or a single range that spans the items you processed). As long as your queue model guarantees that new items are inserted at the end, this should be safe. However, if you have multiple producers for a queue using timestamps for ordering, you’ll need to be certain that items aren’t inserted out of order due to clock sync issues. Otherwise, you may need to extend your conflict ranges beyond the end, perhaps based on some fixed length of time you’re willing to allow items to come in late.