Callback for range queries (Python)

Using FoundationDB
1

I’m using the Python API within a Tornado server. In order to prevent blocking the server’s IO loop, I’ve been adding a callback to futures in order to set the result on a coroutine-friendly future. For example,

tornado_future = TornadoFuture()
callback = lambda fdb_future: handle_result(fdb_future, tornado_future)
commit_future = tr.commit()
commit_future.on_ready(callback)
yield tornado_future

This seems to work well for gets and commits, but I don’t see a way with the public API to get a future for a range query operation. The _get_range method that get_range uses seems to be the kind of thing I’m looking for, but I’m hesitant to use an undocumented method that could change later.

Am I missing a more appropriate method that would work for my use case? If not, would it be reasonable to expose something like what _get_range does as a public API method?

2

I think this is a reasonable feature request, if you wanted to file a GitHub issue for it. Another possible alternative would be to create a new event model (see fdb.init) designed to work with TornadoFutures, if that could be made to do what you want. There’s not really any documentation on writing an event model and I have limited experience with it, but it may be a more generic solution if there are other functions in the API that have similar problems.

Is reading ranges your only obstacle to getting it working for you?

3

Reading ranges is the main one, and it’s not really preventing me from making progress since I can use _get_range for now.

The create_or_open and list methods of the directory layer also seem to be blocking methods that don’t have a way to assign callbacks to them. However, the server I’m working on caches directories, so the first method happens infrequently enough that blocking is acceptable. It looks like the second method can be accomplished with _get_range operations, so that is not really a problem either.

I’m reluctant to add a new event model specifically for Tornado. Tornado itself now uses the event loop and futures from asyncio when it is available, so I think the number of people that could benefit from a Tornado-specific event model is small and will continue to diminish.

It seems the use of futures in the default event model is intended to encourage the client to make concurrent requests rather than facilitate the use of callbacks for all blocking operations. I’d rather not impact the clarity of the codebase or the API for this edge case. Moving to the asyncio event model seems like a better long-term solution.