I saw that ConsistencyCheck supports distributed in version 7.1.49, so I tried running it. I noticed that it must start a process of class test, otherwise it will not be able to start ConsistencyCheck. When I run the following command, it prompts for successful check.
fdbserver -r consistencycheck --knob_consistency_check_range_begin_0 aaa --knob_consistency_check_range_end_0 iii --knob_consistency_check_range_begin_1 jjj --knob_consistency_check_range_end_1 zzz --logdir .
startingConfiguration: start
Run test:ConsistencyCheck start
setting up test (ConsistencyCheck)...
running test (ConsistencyCheck)...
ConsistencyCheck complete
checking test (ConsistencyCheck)...
fetching metrics (ConsistencyCheck)...
1 test clients passed; 0 test clients failed
Run test:ConsistencyCheck Done.
1 tests passed; 0 tests failed.
But when I checked the log, I didn’t find any ConsistencyCheck_FinishedCheck related fields. And there are also some backtrace.
<?xml version="1.0"?>
<Trace>
<Event Severity="10" Time="1702894335.563919" DateTime="2023-12-18T10:12:15Z" Type="Net2Starting" ID="0000000000000000" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.564092" DateTime="2023-12-18T10:12:15Z" Type="Net2TLSConfig" ID="0000000000000000" CAPath="" CertificatePath="" KeyPath="" HasPassword="0" VerifyPeers="Check.Valid=1" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.564092" DateTime="2023-12-18T10:12:15Z" Type="ProgramStart" ID="0000000000000000" RandomSeed="2787859853" SourceVersion="5fe02b5007524978b063687a96474888309aa44e" Version="7.1.49" PackageName="7.1" FileSystem="" DataFolder="" WorkingDirectory="/root" ClusterFile="file:///etc/foundationdb/fdb.cluster" ConnectionString="rtho1Jf8:OetMIXpz@127.0.0.1:4500" ActualTime="1702894335" CommandLine="fdbserver -r consistencycheck --knob_consistency_check_range_begin_0 aaa --knob_consistency_check_range_end_0 bbb --knob_consistency_check_range_begin_1 ccc --knob_consistency_check_range_end_1 ddd --logdir ." BuggifyEnabled="0" FaultInjectionEnabled="1" MemoryLimit="8589934592" VirtualMemoryLimit="0" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" TrackLatestType="Original" />
<Event Severity="10" Time="1702894335.564092" DateTime="2023-12-18T10:12:15Z" Type="StartingRunLoopProfilingThread" ID="0000000000000000" Interval="0.125" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.564092" DateTime="2023-12-18T10:12:15Z" Type="MachineLoadDetail" ID="0000000000000000" User="180367" Nice="300" System="378935" Idle="356669057" IOWait="8950" IRQ="0" SoftIRQ="143106" Steal="0" Guest="0" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.564092" DateTime="2023-12-18T10:12:15Z" Type="NotifyAddressHealthy" ID="0000000000000000" SuppressedEventCount="0" Address="127.0.0.1:4500" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.564092" DateTime="2023-12-18T10:12:15Z" Type="MonitorLeaderChange" ID="0000000000000000" NewLeader="0000000000000001" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.564092" DateTime="2023-12-18T10:12:15Z" Type="RunTests" ID="0000000000000000" TestOnServers="0" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.564092" DateTime="2023-12-18T10:12:15Z" Type="Net2Running" ID="0000000000000000" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.564092" DateTime="2023-12-18T10:12:15Z" Type="ProcessTimeOffset" ID="0000000000000000" ProcessTime="1702894335.566418" SystemTime="1702894335.566418" OffsetFromSystemTime="0.000000" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.566442" DateTime="2023-12-18T10:12:15Z" Type="ConnectingTo" ID="0000000000000000" SuppressedEventCount="0" PeerAddr="127.0.0.1:4500" PeerReferences="3" FailureStatus="OK" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.566442" DateTime="2023-12-18T10:12:15Z" Type="ConnectionExchangingConnectPacket" ID="fcf5b858bcd1a967" SuppressedEventCount="0" PeerAddr="127.0.0.1:4500" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.566961" DateTime="2023-12-18T10:12:15Z" Type="ConnectionEstablished" ID="fcf5b858bcd1a967" SuppressedEventCount="0" Peer="127.0.0.1:4500" ConnectionId="1" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.566961" DateTime="2023-12-18T10:12:15Z" Type="ConnectedOutgoing" ID="0000000000000000" SuppressedEventCount="0" PeerAddr="127.0.0.1:4500" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.566961" DateTime="2023-12-18T10:12:15Z" Type="GetLeaderReply" ID="0000000000000000" SuppressedEventCount="0" Coordinator="127.0.0.1:4500" Nominee="649d0f89aa6f5729" ClusterKey="rtho1Jf8:OetMIXpz" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.566961" DateTime="2023-12-18T10:12:15Z" Type="MonitorLeaderChange" ID="0000000000000000" NewLeader="649d0f89aa6f5729" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.567322" DateTime="2023-12-18T10:12:15Z" Type="DatabaseContextCreated" ID="471ceebc9ecc4f2b" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x43b27a8 0x3872585 0x3875b4d 0x2798a19 0x2574f2e 0x2575979 0x2578956 0x1e87ad8 0x1e87c88 0x1083a98 0x4169ed8 0x416a258 0x433c998 0xdc94d0 0x7f4106095555" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.567322" DateTime="2023-12-18T10:12:15Z" Type="TestsExpectedToPass" ID="0000000000000000" Count="1" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.567322" DateTime="2023-12-18T10:12:15Z" Type="TestRunning" ID="0000000000000000" WorkloadTitle="ConsistencyCheck" TesterCount="1" Phases="15" TestTimeout="0" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.568173" DateTime="2023-12-18T10:12:15Z" Type="GotServerDBInfoChange" ID="0000000000000000" ChangeID="0eb3f9987a6299c9" MasterID="069980427441d6c3" RatekeeperID="ccd2b5d189cd8116" DataDistributorID="f162b8eae58d53df" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.568173" DateTime="2023-12-18T10:12:15Z" Type="CodeCoverage" ID="0000000000000000" File="/home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/ReadYourWrites.actor.cpp" Line="1545" Condition="true" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894335.569485" DateTime="2023-12-18T10:12:15Z" Type="LBDistant" ID="0000000000000000" SuppressedEventCount="0" Distance="2" BackOff="0" TriedAllOptions="0" Alternatives="53a7d7afdeb83873" Token="10ddd2fa2d116df5" Total="1" Best="1" Attempts="0" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="20" Time="1702894336.068084" DateTime="2023-12-18T10:12:16Z" Type="Net2RunLoopTrace" ID="0000000000000000" TraceTime="1702894335.950753" Trace="addr2line -e fdbserver.debug -p -C -f -i 0x7f4106450630 0x7f410644daa1 0x433ba94 0x433d725 0xdc94d0 0x7f4106095555 0xe2ec12" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="20" Time="1702894336.318745" DateTime="2023-12-18T10:12:16Z" Type="Net2RunLoopTrace" ID="0000000000000000" TraceTime="1702894336.208534" Trace="addr2line -e fdbserver.debug -p -C -f -i 0x7f4106450630 0x7f410644daa1 0x433ba94 0x433d725 0xdc94d0 0x7f4106095555 0xe2ec12" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="20" Time="1702894336.552670" DateTime="2023-12-18T10:12:16Z" Type="Net2RunLoopTrace" ID="0000000000000000" TraceTime="1702894336.463047" Trace="addr2line -e fdbserver.debug -p -C -f -i 0x7f4106450630 0x7f410644daa1 0x433ba94 0x433d725 0xdc94d0 0x7f4106095555 0xe2ec12" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894336.570119" DateTime="2023-12-18T10:12:16Z" Type="TestSetupStart" ID="0000000000000000" WorkloadTitle="ConsistencyCheck" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894336.571118" DateTime="2023-12-18T10:12:16Z" Type="TestSetupComplete" ID="0000000000000000" WorkloadTitle="ConsistencyCheck" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894336.571118" DateTime="2023-12-18T10:12:16Z" Type="TestStarting" ID="0000000000000000" WorkloadTitle="ConsistencyCheck" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894336.575479" DateTime="2023-12-18T10:12:16Z" Type="TestComplete" ID="0000000000000000" WorkloadTitle="ConsistencyCheck" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="20" Time="1702894337.077407" DateTime="2023-12-18T10:12:17Z" Type="Net2RunLoopTrace" ID="0000000000000000" TraceTime="1702894336.971915" Trace="addr2line -e fdbserver.debug -p -C -f -i 0x7f4106450630 0x7f410644daa1 0x433ba94 0x433d725 0xdc94d0 0x7f4106095555 0xe2ec12" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="20" Time="1702894337.331240" DateTime="2023-12-18T10:12:17Z" Type="Net2RunLoopTrace" ID="0000000000000000" TraceTime="1702894337.229627" Trace="addr2line -e fdbserver.debug -p -C -f -i 0x7f4106450630 0x7f410644daa1 0x433ba94 0x433d725 0xdc94d0 0x7f4106095555 0xe2ec12" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="20" Time="1702894337.568434" DateTime="2023-12-18T10:12:17Z" Type="Net2RunLoopTrace" ID="0000000000000000" TraceTime="1702894337.486058" Trace="addr2line -e fdbserver.debug -p -C -f -i 0x7f4106450630 0x7f410644daa1 0x433ba94 0x433d725 0xdc94d0 0x7f4106095555 0xe2ec12" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="20" Time="1702894338.087301" DateTime="2023-12-18T10:12:18Z" Type="Net2RunLoopTrace" ID="0000000000000000" TraceTime="1702894337.992181" Trace="addr2line -e fdbserver.debug -p -C -f -i 0x7f4106450630 0x7f410644daa1 0x433ba94 0x433d725 0xdc94d0 0x7f4106095555 0xe2ec12" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="20" Time="1702894338.340672" DateTime="2023-12-18T10:12:18Z" Type="Net2RunLoopTrace" ID="0000000000000000" TraceTime="1702894338.246857" Trace="addr2line -e fdbserver.debug -p -C -f -i 0x7f4106450630 0x7f410644daa1 0x433ba94 0x433d725 0xdc94d0 0x7f4106095555 0xe2ec12" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="20" Time="1702894338.553798" DateTime="2023-12-18T10:12:18Z" Type="Net2RunLoopTrace" ID="0000000000000000" TraceTime="1702894338.500964" Trace="addr2line -e fdbserver.debug -p -C -f -i 0x7f4106450630 0x7f410644daa1 0x433ba94 0x433d725 0xdc94d0 0x7f4106095555 0xe2ec12" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="20" Time="1702894339.102326" DateTime="2023-12-18T10:12:19Z" Type="Net2RunLoopTrace" ID="0000000000000000" TraceTime="1702894339.011969" Trace="addr2line -e fdbserver.debug -p -C -f -i 0x7f4106450630 0x7f410644daa1 0x433ba94 0x433d725 0xdc94d0 0x7f4106095555 0xe2ec12" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="20" Time="1702894339.356092" DateTime="2023-12-18T10:12:19Z" Type="Net2RunLoopTrace" ID="0000000000000000" TraceTime="1702894339.266298" Trace="addr2line -e fdbserver.debug -p -C -f -i 0x7f4106450630 0x7f410644daa1 0x433ba94 0x433d725 0xdc94d0 0x7f4106095555 0xe2ec12" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="20" Time="1702894339.569707" DateTime="2023-12-18T10:12:19Z" Type="Net2RunLoopTrace" ID="0000000000000000" TraceTime="1702894339.520791" Trace="addr2line -e fdbserver.debug -p -C -f -i 0x7f4106450630 0x7f410644daa1 0x433ba94 0x433d725 0xdc94d0 0x7f4106095555 0xe2ec12" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894339.576760" DateTime="2023-12-18T10:12:19Z" Type="CheckingResults" ID="0000000000000000" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894339.577292" DateTime="2023-12-18T10:12:19Z" Type="TestFetchingMetrics" ID="0000000000000000" WorkloadTitle="ConsistencyCheck" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894339.577527" DateTime="2023-12-18T10:12:19Z" Type="MetricsReturned" ID="0000000000000000" Count="0" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894339.577527" DateTime="2023-12-18T10:12:19Z" Type="Metric" ID="0000000000000000" Name="Reporting Clients" Value="1" Formatted="1" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894339.577527" DateTime="2023-12-18T10:12:19Z" Type="TestResults" ID="0000000000000000" Workload="ConsistencyCheck" Passed="1" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
<Event Severity="10" Time="1702894339.577527" DateTime="2023-12-18T10:12:19Z" Type="ElapsedTime" ID="0000000000000000" SimTime="4.01344" RealTime="4.01306" RandomUnseed="17095" ThreadID="5272635557964991718" Machine="127.0.0.1:50671" LogGroup="default" />
</Trace>
I tried to decode and found that tool fdbserver.debug reported an error.
addr2line -e fdbserver.debug.x86_64 -p -C -f -i 0x7f4106450630 0x7f410644daa1 0x433ba94 0x433d725 0xdc94d0 0x7f4106095555 0xe2ec12
addr2line: fdbserver.debug.x86_64: File truncated
So how do I determine if it has really been checked? Thanks!