Questions GetCommitVersionRequest request number necessity

superhail · October 7, 2021, 8:47am

As I read the source code of MasterProxy.actor.cpp, it seems that the logic of get commit version is like

wait(self->latestLocalCommitBatchResolving.whenAtLeast(localBatchNumber - 1));
some codes
send the commit version request
wait for commit version reply
send resolving requests
update latestLocalCommitBatchResolving to localBatchNumber asynchronously
wait for resolving replies

github.com

apple/foundationdb/blob/ddf85af360c8e38dc454dbd8c481ef2a7efddacb/fdbserver/MasterProxyServer.actor.cpp#L1056-L1233

    
      
          ACTOR Future<Void> commitBatch(ProxyCommitData* self,
                                         vector<CommitTransactionRequest> trs,
                                         int currentBatchMemBytesCount) {
          	// WARNING: this code is run at a high priority (until the first delay(0)), so it needs to do as little work as
          	// possible
          	state int64_t localBatchNumber = ++self->localCommitBatchesStarted;
          	state LogPushData toCommit(self->logSystem);
          	state double t1 = now();
          	state Optional<UID> debugID;
          	state bool forceRecovery = false;
          	state int batchOperations = 0;
          	int64_t batchBytes = 0;
          	for (int t = 0; t < trs.size(); t++) {
          		batchOperations += trs[t].transaction.mutations.size();
          		batchBytes += trs[t].transaction.mutations.expectedSize();
          	}
          	state int latencyBucket =
          	    batchOperations == 0
          	        ? 0
          	        : std::min<int>(SERVER_KNOBS->PROXY_COMPUTE_BUCKETS - 1,

This file has been truncated. show original

So it seems to my for each proxy, there can be at most one get commit version request on the fly, as all the othe ones are queued by latestLocalCommitBatchResolving, so why is request number still necessary? I really don’t get it.

markus.pilman · October 8, 2021, 4:48pm

Because we’re sending this request at least once… Basically, if you call send or getReply on a RequestStream<T> we’re in fact calling sendReliable. So if the connection to the master/sequencer is closed after we sent the request but before we received a response, we will automatically retry (send this request again).

So the sequencer might get both (we don’t know whether the first one was lost or not). Giving out two versions would cause problems (specifically, we would now create a hole in the version stream and at the resolve-step the resolver would block all transaction waiting for this hole to be filled). You can find the code that handles this in the master here.