Why can I only range read 2857 keys?

Using FoundationDB
1

In the C API, when doing a range read with:

fdb_transaction_get_range(tr, FDB_KEYSEL_FIRST_GREATER_OR_EQUAL(start, 1), FDB_KEYSEL_FIRST_GREATER_OR_EQUAL(end, 1), 0, 0, FDB_STREAMING_MODE_WANT_ALL, 0, 0, 0);

then waiting for the future, and extracting the FDBKeyValue array with:

fdb_future_get_keyvalue_array(future, &keyvalue, &count, &more);

only the first 2857 keys are returned (count is 2857). Calling fdb_future_get_keyvalue_array() again does not return the next 2857 keys, as one could expect.

Why am I only getting the first 2857 keys in the database and how can I get every single key? Thanks!

2

The get_range operation is returning data one batch at a time, so you are supposed to check the value returned in more to know if you need to call it again to get the rest of the range. (see https://apple.github.io/foundationdb/api-c.html?highlight=fdb_transaction_get_range#c.fdb_transaction_get_range)

The FDB_STREAMING_MODE_WANT_ALL here does not mean “in a single batch”, but should probably be understood as “in as few batches as possible”, and in your case, 2857 keys is considered the ideal batch size. The other streaming modes are for cases where you will inspect the data as they arrive and stop iterating before the end of the range (so it would be wasteful if the db would send you ALL the keys you are are only interested in a few).

So in your case, you’ll have to check if more is non-zero after the call, and call fdb_transaction_get_range again and again, using FDB_KEYSEL_FIRST_GREATER_THAN(…) on the last key of the previous batch, and stopping when more is 0.

Be warned though, that reading an unbounded range in a single transaction will probably not work as you’d expect, due to the 5 seconds lifetime of each transaction: it is possible that you end up taking more than 5 seconds to read a very large range, meaning that you’ll need to figure out how to read large quantities of data across multiple transactions without the safety net of ACID transactions.

2 Likes