Get_approximate_size counts sets and gets of the same key multiple times

Development
1

I noticed while testing the Transaction.getApproximateSize() API in the Java bindings, that getting the same key N times in the same transaction adds N times the cost of a single get to the size approximation. This seems wrong to me, since the tx size for gets comes from the read conflict ranges, but subsequent gets of the same key shouldn’t affect those ranges right?

Likewise, setting the same key N times (even to the same value) adds N times the cost of a write to the tx size. Which also seems wrong since I assume the FDB client would optimise those to a single write at commit time - or does it actually send all of them to the server?

2

That’s right, that (with the default transaction options), multiple sets to the same key will be coalesced, as will overlapping read or write conflict ranges. In particular, the “write cache” will combine multiple writes to the same key and turn them into a single mutation. (I say with default transaction options because I believe if you set the setReadYourWritesDisable transaction option, this will no longer be the case.)

With the getApproximateSize method, the choice was made to deliberately over count those mutations and conflict ranges so that the method is as cheap as possible. In essence, the underlying transaction option just maintains a single integer, which it increments with every read and mutation based solely on the argument parameters, whereas if it needed to correctly account for coalesced ranges, it would need to do more book keeping. The main goal of the method is to give a mechanism for adopters to determine if they’ve hit the 10 MB transaction size limit before they try and commit. So, we anticipate use cases that look something like:

// Save items from data list while the transaction's approximate commit size
// is below some threshold. Return the number of saved elements
<T> int saveData(Transaction tr, List<T> dataList) {
     if (dataList.isEmpty()) {
        return 0;
     }
     int saved = 0;
     do {
        T datum = dataList.get(saved);
        saveDatum(tr, datum);
        saved++;
     } while (saved < dataList.size() && tr.getApproximateSize() < THRESHOLD);
     return saved;
}

So, because it can be called in relatively tight loops like that, we don’t want the method to be too expensive. It’s somewhat important that it doesn’t under count, because that could lead to us thinking that commits that we think are going to be smaller than the transaction size limit failing when we go to commit(), but if over counts, that just means we end up paging less work into each transaction. Which certainly isn’t ideal, but a trade off we can probaby live with.

1 Like