I am currently exploring the design of the Record Layer from the ground up.
Rather than create multiple threads with my questions, I am hoping to create one thread.
My hope is that discussions in this thread will be helpful to me for my Rust layer and also for future layer developers as they try to build layers leveraging ideas from the Record Layer.
EndpointType type, there are two variants that are named
TREE_END. I was wondering if there was any specific reason for naming the variants with the prefix of
Did FoundationDB in the past have a limit on the number of key-values that could be read in a transaction?
The reason I ask is because RecordLayer includes a
RecordScanLimiter as a “out-of-band” limiter.
While I understand the need for
TimeScanLimiter given FoundationDB design imitations, was wondering if there was a reason for
RecordScanLimiter to exist as an out-of-band limiter?
TREE_ needs to indicate is that it’s something independent of the range keys passed to the API, and, moreover, the lowest / highest such thing, in context. And since that context might be an index or a the primary extent, or really any other subtree in the record store, something neutral is required.
Records (or “rows”) scanned is a pretty good proxy for the amount of work done, particularly by a query. The current design, based partly on the underlying limitations of the key-value store, but also on a requirement to support many tenants with pretty high throughput, is that you must either limit the work done in that way, or else license the system to stop and pick up again later with even lower per-transaction thresholds.
Specifically, the number of keys scanned is a pretty good indicator of poor index definition / selection, that might still slip in under a byte limit (all the index entries / records are small) or time limit (you can get a lot read in a few seconds).