Simulation testing of FoundationDb

Hello!

I’ve just build fdb 6.3.9 from sources and I’m trying to run simulation tests on it.

First of all, I started the correctness-6.3.9.tar tarball with joshua (GitHub - FoundationDB/fdb-joshua: FoundationDB Correctness service) and received some errors

[oleg@oleg2 FdbJoshua]$ python3 -m joshua.joshua -C ../devops/clusters/joshua/fdb.cluster tail 20210414-154515-oleg-9a72672d246b2754
Results for test ensemble: 20210414-154515-oleg-9a72672d246b2754
0x1000000435db6fba10000 103 11c1fb3ab7a8 5315918842548694494 '<Test TestUID="e3149994-ac82-4703-be22-65b7f00ab230" RandomSeed="732940594" SourceVersion="2a5fb92f7f481ce2c7cfcaa300e952512bc38c56" Time="1618415118" BuggifyEnabled="1" DeterminismCheck="0" OldBinary="fdbserver-6.2.24" TestFile="/var/joshua/ensembles/20210414-154515-oleg-9a72672d246b2754/tests/restarting/from_6.2.0/SnapCycleRestart-1.txt" SimElapsedTime="139.481" RealElapsedTime="11.2693" RandomUnseed="53567" Passed="1" Failed="1" PeakMemory="199954432" OK="true"><BuggifySection File="flow/Knobs.cpp" Line="39" /><BuggifySection File="flow/Knobs.cpp" Line="56" /><BuggifySection File="flow/Knobs.cpp" Line="57" /><BuggifySection File="fdbclient/Knobs.cpp" Line="73" /><BuggifySection File="fdbclient/Knobs.cpp" Line="75" /><BuggifySection File="fdbserver/Knobs.cpp" Line="35" /><BuggifySection File="fdbserver/Knobs.cpp" Line="36" /><BuggifySection File="fdbserver/Knobs.cpp" Line="42" /><BuggifySection File="fdbserver/Knobs.cpp" Line="45" /><BuggifySection File="fdbserver/Knobs.cpp" Line="73" /><BuggifySection File="fdbserver/Knobs.cpp" Line="76" /><BuggifySection File="fdbserver/Knobs.cpp" Line="90" /><BuggifySection File="fdbserver/Knobs.cpp" Line="135" /><BuggifySection File="fdbserver/Knobs.cpp" Line="141" /><BuggifySection File="fdbserver/Knobs.cpp" Line="194" /><BuggifySection File="fdbserver/Knobs.cpp" Line="205" /><BuggifySection File="fdbserver/Knobs.cpp" Line="206" /><BuggifySection File="fdbserver/Knobs.cpp" Line="222" /><BuggifySection File="fdbserver/Knobs.cpp" Line="275" /><BuggifySection File="fdbserver/Knobs.cpp" Line="277" /><BuggifySection File="fdbserver/Knobs.cpp" Line="279" /><BuggifySection File="fdbserver/Knobs.cpp" Line="311" /><BuggifySection File="fdbserver/Knobs.cpp" Line="318" /><BuggifySection File="fdbserver/Knobs.cpp" Line="324" /><BuggifySection File="fdbserver/Knobs.cpp" Line="330" /><BuggifySection File="fdbserver/Knobs.cpp" Line="351" /><BuggifySection File="fdbserver/Knobs.cpp" Line="378" /><BuggifySection File="fdbserver/Knobs.cpp" Line="395" /><BuggifySection File="fdbserver/Knobs.cpp" Line="397" /><BuggifySection File="fdbserver/Knobs.cpp" Line="406" /><BuggifySection File="fdbserver/Knobs.cpp" Line="424" /><BuggifySection File="fdbserver/Knobs.cpp" Line="439" /><BuggifySection File="fdbserver/Knobs.cpp" Line="501" /><BuggifySection File="fdbserver/Knobs.cpp" Line="508" /><BuggifySection File="fdbserver/Knobs.cpp" Line="538" /><BuggifySection File="fdbrpc/AsyncFileCached.actor.cpp" Line="61" /><BuggifySection File="fdbrpc/sim2.actor.cpp" Line="804" /><BuggifySection File="fdbclient/NativeAPI.actor.cpp" Line="394" /><BuggifySection File="fdbrpc/sim2.actor.cpp" Line="241" /><BuggifySection File="fdbserver/CoordinatedState.actor.cpp" Line="38" /><BuggifySection File="fdbserver/CoordinatedState.actor.cpp" Line="41" /><BuggifySection File="fdbrpc/sim2.actor.cpp" Line="930" /><BuggifySection File="fdbserver/masterserver.actor.cpp" Line="725" /><BuggifySection File="fdbserver/masterserver.actor.cpp" Line="925" /><BuggifySection File="fdbserver/tester.actor.cpp" Line="831" /></Test>\n<Test TestFile="/var/joshua/ensembles/20210414-154515-oleg-9a72672d246b2754/tests/restarting/from_6.2.0/SnapCycleRestart-2.txt" RandomSeed="732940595" BuggifyEnabled="1" DeterminismCheck="0" OldBinary="fdbserver-6.2.24"><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /></Test>\n'
0x1000000435db93f190000 103 11c1fb3ab7a8 6774011931387750987 '<Test TestFile="/var/joshua/ensembles/20210414-154515-oleg-9a72672d246b2754/tests/fast/BackupCorrectnessClean.txt" RandomSeed="291958968" BuggifyEnabled="1" DeterminismCheck="0" OldBinary=""><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /></Test>\n'
0x1000000435dbb71630000 103 11c1fb3ab7a8 3061461588318534719 '<Test TestFile="/var/joshua/ensembles/20210414-154515-oleg-9a72672d246b2754/tests/slow/ParallelRestoreOldBackupCorrectnessCycle.txt" RandomSeed="75432848" BuggifyEnabled="1" DeterminismCheck="0" OldBinary=""><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /></Test>\n'
0x1000000435dbd9f660000 103 11c1fb3ab7a8 3103890438703106204 '<Test TestFile="/var/joshua/ensembles/20210414-154515-oleg-9a72672d246b2754/tests/fast/KillRegionCycle.txt" RandomSeed="901454152" BuggifyEnabled="1" DeterminismCheck="0" OldBinary=""><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /></Test>\n'
0x1000000435dbfc5240000 103 11c1fb3ab7a8 3102398388099496483 '<Test TestFile="/var/joshua/ensembles/20210414-154515-oleg-9a72672d246b2754/tests/slow/DifferentClustersSameRV.txt" RandomSeed="257028686" BuggifyEnabled="1" DeterminismCheck="0" OldBinary=""><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /></Test>\n'
0x1000000435dc21b250000 103 11c1fb3ab7a8 4988921788399198735 '<Test TestFile="/var/joshua/ensembles/20210414-154515-oleg-9a72672d246b2754/tests/fast/TaskBucketCorrectness.txt" RandomSeed="65957481" BuggifyEnabled="1" DeterminismCheck="0" OldBinary=""><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /></Test>\n'
0x1000000435dc44dd10000 103 11c1fb3ab7a8 8196442410007927692 '<Test TestFile="/var/joshua/ensembles/20210414-154515-oleg-9a72672d246b2754/tests/fast/TimeKeeperCorrectness.txt" RandomSeed="1584826" BuggifyEnabled="1" DeterminismCheck="0" OldBinary=""><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /></Test>\n'
0x1000000435dc681c00000 103 11c1fb3ab7a8 4673601395945998285 '<Test TestFile="/var/joshua/ensembles/20210414-154515-oleg-9a72672d246b2754/tests/fast/VersionStamp.txt" RandomSeed="37680121" BuggifyEnabled="1" DeterminismCheck="0" OldBinary=""><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /></Test>\n'
0x1000000435dc8b55c0000 103 11c1fb3ab7a8 2993740394339652498 '<Test TestFile="/var/joshua/ensembles/20210414-154515-oleg-9a72672d246b2754/tests/fast/BackupCorrectness.txt" RandomSeed="34274817" BuggifyEnabled="0" DeterminismCheck="0" OldBinary=""><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /></Test>\n'
0x1000000435dcb48db0000 103 11c1fb3ab7a8 2462806307335811719 '<Test TestFile="/var/joshua/ensembles/20210414-154515-oleg-9a72672d246b2754/tests/slow/ParallelRestoreNewBackupCorrectnessAtomicOp.txt" RandomSeed="814560244" BuggifyEnabled="1" DeterminismCheck="0" OldBinary=""><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /><NoTraceFileGenerated Severity="30" Plugin="" MachineName="11c1fb3ab7a8" /></Test>\n'
Ensemble stopped

I have some questions:

  1. Is it normal behaviour, ore something went wrong?
  2. I see that 10 tests failed. How can I know how many tests passed?
  3. I’d like to rerun a test
bin/fdbserver -r simulation -f tests/fast/VersionStamp.txt

How can I force the same RandomSeed and BuggifyEnabled to the test?

  1. I tried to run ./joshua_test from correctness-6.3.9.tar manually (without joshua machine). But receive an error:
[oleg@oleg2 correctness-6.3.9]$ ./joshua_test 
<Test TestUID="f74d39fc-8e10-4e1f-967c-ca2b085373a7" Severity="40" Passed="0" Failed="-1" PeakMemory="619892736" OK="false"><SummarizationError Severity="40" ErrorMessage="Failed to parse &lt;Event Severity=&quot;10&quot; Time=&quot;0.000000&quot; Type=&quot;BuggifySection&quot; Activated=&quot;0&quot; File=&quot;/home/oleg/work/fdb/FoundationDb/flow/Knobs.cpp&quot; Line=&quot;43&quot; Machine=&quot;0.0.0.0:0&quot; LogGroup=&quot;default&quot; /&gt;" /><TestUnexpectedlyNotFinished /></Test>

what am I missing?

  1. There are lots of differnent tests in test subdirectory of the fdb source tree. Why only few of them are run with joshua?

I think something is wrong here. Usually, we don’t run joshua_test manually. It is intended to be ran by joshua_agent, which is part of the GitHub - FoundationDB/fdb-joshua: FoundationDB Correctness service repo.

The typical setup is:

  • Have a FoundationDB cluster running.
  • Build and run the docker image from this script, make sure the docker image uses the same FDB version as the cluster in the first step (current master branch uses the latest FDB version, which can be changed with this patch). Go inside the docker image and make sure it can talk to the cluster.
  • Submit correctness package to the cluster, and check its output.

For your questions:

  1. Something is wrong.
  2. Use --xml --errors argument in your joshua tail command.
  3. Simulation needs two additional arguments, i.e., -s 123 -b on (123 is an integer seed you can change, “on” can be replaced with “off” to turn on/off buggification).
  4. Probably the environment is not the same as Joshua agent’s docker image.
  5. Many are unit tests not included in correctness package, and some tests are outdated.

Note it’s important to use the Docker image, because it contains necessary old FDB binaries (and old TLS libraries) for upgrade tests. Without them, upgrade tests will fail (like the first one in your test needs 6.2.24 binary).

I think most people outside of Apple/Snowflake use ctest to run tests.

It does appear to be a huge gap in documentation that there’s no “how to run tests” guide in the readme, nor contributing guide, nor anywhere in the wiki. :confused:

1 Like

Note it’s important to use the Docker image, because it contains necessary old FDB binaries (and old TLS libraries) for upgrade tests. Without them, upgrade tests will fail (like the first one in your test needs 6.2.24 binary).

I use the docker image for running tests with joshua. But after joshua shows that tests failed, I need a step-by step investigation process for the failed tests. First of all I have to run failed tests manually, without joshua.

  1. Use --xml --errors argument in your joshua tail command.

I can not see any passed tests with python3 -m joshua.joshua -C ../devops/clusters/joshua/fdb.cluster tail 20210414-154515-oleg-9a72672d246b2754 --xml --errors, only the errors

I’d like to receive a list like this:

  • Test xxx - passed
  • Test xxy - passed
  • Test xyx - failed
  • Test xyy - passed

Is it possible with joshua client?

Many are unit tests not included in correctness package, and some tests are outdated.

But why did joshua run only 10 tests from thats are in the corectness package?

Is there any step-by-step guide, how to run the corectness tests with ctest?

1 Like