Large Range Scans - avoid 5s limit

Your intuition that this doesn’t seem like it should be an architectural limitation is absolutely correct.

The reason read-only snapshot transactions are also bounded by the 5s limit is mostly due to the implementation of our SSD storage engine. Currently that storage engine is not in fact a multi-versioned store. It stores a single version of the database at a point more than 5 seconds in the past. MVCC of more recent versions is kept in storage server memory[1] in a data structure that overlays the data on disk. Periodically, the oldest versions are detached from the in-memory tree and written to disk.

If someday a storage engine were written that was multi-versioned on disk, my sense is it’d be relatively straightforward to support long-running read-only transactions. There’s a discussion thread on new storage engine ideas here: Discussion thread for new storage engine ideas

[1] But note that they are fully durable due to being written to disk on the transaction logs.

3 Likes