Update FreeBSD port

dch · January 22, 2020, 2:29pm

There is already an existing port but it uses the older Makefile build, and therefore doesn’t build C or other bindings. Using the newer cmake structure will provide this. So, I have taken Jesse’s amazing work and prepped it into a port for FreeBSD, so it can simply be pkg install -r FreeBSD databases/foundationdb-devel.

plan

do a port is “usable” at this point so commit it
run the test suite & report what failed
rebase and repeat
hopefully rebase successfully & pass all the tests

fails

The failures reported follow, I’ve attached the traces to the gist if these are useful, as tar.xz and also the ninja check build report. I don’t intend to address any of these failures as it makes more sense to focus on hitting 7.0.0.

97% tests passed, 3 tests failed out of 103

Label Time Summary:
fast          = 921.60 sec*proc (38 tests)
from_6.2.0    = 7365.74 sec*proc (4 tests)
rare          = 347.28 sec*proc (14 tests)
restarting    =  82.59 sec*proc (3 tests)
slow          = 1089.42 sec*proc (31 tests)
status        =  13.68 sec*proc (12 tests)

Total Test time (real) = 10218.50 sec

The following tests FAILED:
	  1 - SimpleExternalTest (Failed)
	 58 - restarting/from_6.2.0/SnapTestSimpleRestart-1 (Timeout)
	 60 - restarting/from_6.2.0/SnapCycleRestart-1 (Timeout)

questions

the trace logs are a bit intimidating, are there any tools or scripts that help diagnose these?
what’s the requirements for upstreaming Jesse’s work into the main repo, test failures notwithstanding?

alexmiller · January 22, 2020, 11:12pm

These are… a bit strange of tests. SimpleExternalTest loads a .so file that has a workload in it. I think snapshotting tests were added after Jesse did most of his work, and added additional platform-specific code, so I’m guessing a FreeBSD version of that is missing.

I’d be fine with merging a PR where these tests still fail. They’re unlikely to impact anyone’s current use of FDB on FreeBSD anyway, and fixing them up later should be fine. Grepping for Severity="40" should tell you what in a test actually failed, and anything beyond that, I’ll go debug.

Can you go re-run ctest with --output-on-failure set? The trace logs for the snapshotting tests don’t actually seem to show a failure, and I’m hoping something was printed to stdout/stderr instead?

SimpleExternalTest fails with:

<Event Severity="40" Time="52.377900" Type="TesterFailed" Machine="3.4.3.4:1" ID="64c49ab5c79cd33d" Error="platform_error" ErrorDescription="Platform error" ErrorCode="1500" Reason="Error" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x2244551 0x2244631 0x162b092 0x153addf 0x1539a9d 0xef63e8 0x214655d 0x21463d6 0x21460ad 0x2146402 0x2146a6b 0xef63e8 0x22172a6 0x221255d 0x2138551 0x102f72b 0xb4a10f 0x802415008" LogGroup="default" Roles="TS" />

Could you symbolicate that? I’d assume it should point towards the failure.

But either way, if we reach a point where master builds, or your port is landed so that there’s a simple one-step build command, then it’s much easier for me to just do the running and debugging on freebsd myself.

dch · January 23, 2020, 12:10pm

I have no idea what symbolicate means but here’s the first test. This is being run in a ramdisk on a rather well spec’d machine, if that should make a difference.

I trimmed the time in the test as I don’t want to wait an actual hour for each failure - is this correct? It’s not clear to me if timeout & testDuration are simulated or actual clock time.

diff --git tests/SimpleExternalTest.txt tests/SimpleExternalTest.txt
index 9d63299f..48d4a577 100644
--- tests/SimpleExternalTest.txt
+++ tests/SimpleExternalTest.txt
@@ -2,3 +2,6 @@ testTitle=SimpleExternalTest
 testName=External
 libraryName=c_workloads
 workloadName=SimpleWorkload
+testDuration=30000
+timeout=30000

ctest --output-on-failure -v
Test project /tmp/wolfspider/build
        Start   1: SimpleExternalTest
  1/103 Test   #1: SimpleExternalTest ..............................***Failed  416.48 sec
2020-01-23 10:45:23,261 - 33485 - INFO - COMMAND: /tmp/wolfspider/build/bin/fdbserver -r simulation -s 1272856103 -b on --trace_format xml -s 1272856103 -f /tmp/wolfspider/tests/SimpleExternalTest.txt
Random seed is 1272856103...
Datacenter 0: 3/12 machines, 1/1 coordinators
Datacenter 1: 3/12 machines, 0/1 coordinators
Datacenter 2: 3/12 machines, 0/1 coordinators
Datacenter 3: 3/12 machines, 0/1 coordinators
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
startingConfiguration:new log_spill:=2 triple regions=[{"datacenters":[{"id":"0","priority":2},{"id":"2","priority":1,"satellite":1}]},{"datacenters":[{"id":"1","priority":1},{"id":"3","priority":1,"satellite":1}]}] remote_double resolvers:=5 ssd-2 usable_regions:=1 start
setting up test (ChangeConfig)...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
running test (ChangeConfig)...
ChangeConfig complete
ChangeConfig complete
ChangeConfig complete
ChangeConfig complete
ChangeConfig complete
ChangeConfig complete
checking test (ChangeConfig)...
fetching metrics (ChangeConfig)...
SimulatedFDBDTerminated: Platform error
SimulatedFDBDTerminated: Platform error
SimulatedFDBDTerminated: Platform error
SimulatedFDBDTerminated: Platform error
SimulatedFDBDTerminated: Platform error
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
ERROR: Test timed out after 30000 seconds.
0 test clients passed; 5 test clients failed
Unseed: 62347
Elapsed: 36001.000000 simsec, 333.358084 real seconds
TestSystem 1.1.1.1:1: 0.000 Mclocks
Server 2.1.1.0:1: 0.000 Mclocks
Server 3.4.3.2:1: 0.000 Mclocks
Server 3.4.3.4:1: 0.000 Mclocks
Server 3.4.3.5:1: 0.000 Mclocks
Server 2.0.1.1:1: 0.000 Mclocks
Server 2.3.1.1:1: 0.000 Mclocks
Server 2.1.1.1:1: 0.000 Mclocks
Server 2.3.1.2:1: 0.000 Mclocks
Server 3.4.3.1:1: 0.000 Mclocks
Server 2.2.1.2:1: 0.000 Mclocks
Server 3.4.3.3:1: 0.000 Mclocks
Server 2.1.1.2:1: 0.000 Mclocks
Server 2.2.1.1:1: 0.000 Mclocks
Server 2.0.1.2:1: 0.000 Mclocks
Server 2.2.1.0:1: 0.000 Mclocks
Server 2.0.1.0:1: 0.000 Mclocks
Server 2.3.1.0:1: 0.000 Mclocks
22 SevError events logged
Delete /tmp/wolfspider/build/test_runs/2020_01_23__10_45_23/test_SimpleExternalTest/simfdb

I will submit the port today, it should be available 2-3 days later on the next package build run.

dch · January 23, 2020, 2:24pm

And the last 2 fails, where I didn’t (yet) alter the timeouts:

 58/103 Test  #58: restarting/from_6.2.0/SnapTestSimpleRestart-1 ...***Timeout 3600.21 sec
2020-01-23 11:14:20,532 - 88279 - INFO - Run old binary at /usr/local/bin/fdbserver
2020-01-23 11:14:20,532 - 88279 - INFO - COMMAND: /usr/local/bin/fdbserver -r simulation -s 1272856164 -b on --trace_format xml -s 1272856164 -f /tmp/wolfspider/tests/restarting/from_6.2.0/SnapTestSimpleRestart-1.txt
Random seed is 1272856164...
Datacenter 0: 8/8 machines, 1/1 coordinators
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
startingConfiguration:new log_spill:=2 log_version:=4 triple resolvers:=5 memory-2 usable_regions:=1 start
FDBD joined cluster.
setting up test (ChangeConfig)...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
running test (ChangeConfig)...
ChangeConfig complete
ChangeConfig complete
ChangeConfig complete
ChangeConfig complete
ChangeConfig complete
checking test (ChangeConfig)...
fetching metrics (ChangeConfig)...
setting up test (SnapSimplePre)...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
running test (SnapSimplePre)...
SnapTest complete
SnapTest complete
SnapTest complete
SnapTest complete
SnapSimplePre complete
checking test (SnapSimplePre)...
fetching metrics (SnapSimplePre)...
setting up test (ConsistencyCheck)...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
running test (ConsistencyCheck)...
ConsistencyCheck complete
ConsistencyCheck complete
ConsistencyCheck complete
ConsistencyCheck complete
ConsistencyCheck complete
checking test (ConsistencyCheck)...
fetching metrics (ConsistencyCheck)...
4 test clients passed; 0 test clients failed
setting up test (SnapSimpleTakeSnap)...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
running test (SnapSimpleTakeSnap)...
SnapTest complete
SnapTest complete
SnapTest complete

        Start  59: restarting/from_6.2.0/SnapTestRestart-1
 59/103 Test  #59: restarting/from_6.2.0/SnapTestRestart-1 .........   Passed   62.13 sec
        Start  60: restarting/from_6.2.0/SnapCycleRestart-1
 60/103 Test  #60: restarting/from_6.2.0/SnapCycleRestart-1 ........***Timeout 3600.06 sec
2020-01-23 12:15:22,875 - 73795 - INFO - Run old binary at /usr/local/bin/fdbserver
2020-01-23 12:15:22,875 - 73795 - INFO - COMMAND: /usr/local/bin/fdbserver -r simulation -s 1272856168 -b on --trace_format xml -s 1272856168 -f /tmp/wolfspider/tests/restarting/from_6.2.0/SnapCycleRestart-1.txt
Random seed is 1272856168...
Datacenter 0: 8/8 machines, 1/1 coordinators
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
FDBD joined cluster.
startingConfiguration:new log_spill:=2 logs:=2 proxies:=5 single ssd-2 usable_regions:=1 start
setting up test (ChangeConfig)...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
running test (ChangeConfig)...
ChangeConfig complete
ChangeConfig complete
ChangeConfig complete
ChangeConfig complete
ChangeConfig complete
ChangeConfig complete
checking test (ChangeConfig)...
fetching metrics (ChangeConfig)...
setting up test (SnapCyclePre)...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
Test received trigger for setup...
running test (SnapCyclePre)...
CycleWorkload;SnapTest complete
CycleWorkload;SnapTest complete
CycleWorkload;SnapTest complete
CycleWorkload;SnapTest complete

I’ll upload the detailed logs, and hopefully you can explain what “symbolicate” means.

ajbeamon · January 23, 2020, 2:52pm

That means to convert the backtrace addresses that were logged in the trace event Alex posted into symbols, usually by running the command provided. That way we can see where in the code things are going wrong. In your case, you could get it by running:

addr2line -e fdbserver.debug -p -C -f -i 0x2244551 0x2244631 0x162b092 0x153addf 0x1539a9d 0xef63e8 0x214655d 0x21463d6 0x21460ad 0x2146402 0x2146a6b 0xef63e8 0x22172a6 0x221255d 0x2138551 0x102f72b 0xb4a10f 0x802415008

dch · January 23, 2020, 4:52pm

OK good progress today on the squash & rebase. The git part has been done, but I don’t expect this yet to build cleanly on any platform yet - there’s still a few changes I need to look at more closely.

That said, it’s close enough for somebody with more expertise to look at the commits and require changes.

My branch contains 2 commits:

many trivial changes of ifdefs & mechanical CMake
a few changes that require more review and further clean-up

I’ll keep working on this locally, to get it to build, and just let me know if this is ready for actual PR, even if it doesn’t quite build yet. If you can check it out on linux or whatever else is used, let me know what breaks during build. Thanks!

alexmiller · January 23, 2020, 11:47pm

I’ll try to take a look sometime over the next few days, and just go poke at the failures myself.

If you make a draft PR to apple/foundationdb, we can flip the CI on for your PR to get automatic linux/windows/macos testing.

dch · February 1, 2020, 11:18pm

PR is here https://github.com/apple/foundationdb/pull/2634 the branch can be squashed prior, but it’s much cleaner to review now.

The fast tests pass \o/ and I’ll look into the full test suite next week.

                                           
-- /repos/fdb /tmp/.build
-- Has aligned_alloc: 1
-- Could NOT find Virtualenv (missing: VIRTUALENV_EXE) 
-- Current git version 0f1a7e1fa7080659b55f7ecc44ec4e9c2e7d519c
-- FDB version is 7.0.0
-- FDB package name is 7.0
-- Check versions.h on flow_actors
-- Check versions.h on fdbrpc_actors
-- Check versions.h on fdbclient_actors
-- Check versions.h on fdbserver_actors
-- Check versions.h on fdbcli_actors
-- Check versions.h on fdbmonitor
-- Check versions.h on fdb_flow_actors
-- Check versions.h on fdb_flow_tester_actors
-- Check versions.h on fdbbackup_actors
-- Use old fdb at /tmp/.build/bin/fdbserver
-- ADDING SIMULATOR TEST 0 SimpleExternalTest
-- ADDING SIMULATOR TEST 1 fast/AtomicBackupCorrectness
-- ADDING SIMULATOR TEST 2 fast/AtomicBackupToDBCorrectness
-- ADDING SIMULATOR TEST 3 fast/AtomicOps
-- ADDING SIMULATOR TEST 4 fast/AtomicOpsApiCorrectness
-- ADDING SIMULATOR TEST 5 fast/BackupCorrectness
-- ADDING SIMULATOR TEST 6 fast/BackupCorrectnessClean
-- ADDING SIMULATOR TEST 7 fast/BackupToDBCorrectness
-- ADDING SIMULATOR TEST 8 fast/BackupToDBCorrectnessClean
-- ADDING SIMULATOR TEST 9 fast/CloggedSideband
-- ADDING SIMULATOR TEST 10 fast/ConstrainedRandomSelector
-- ADDING SIMULATOR TEST 11 fast/CycleAndLock
-- ADDING SIMULATOR TEST 12 fast/CycleTest
-- ADDING SIMULATOR TEST 13 fast/FuzzApiCorrectness
-- ADDING SIMULATOR TEST 14 fast/FuzzApiCorrectnessClean
-- ADDING SIMULATOR TEST 15 fast/IncrementTest
-- ADDING SIMULATOR TEST 16 fast/InventoryTestAlmostReadOnly
-- ADDING SIMULATOR TEST 17 fast/InventoryTestSomeWrites
-- ADDING SIMULATOR TEST 18 fast/KillRegionCycle
-- ADDING SIMULATOR TEST 19 fast/LocalRatekeeper
-- ADDING SIMULATOR TEST 20 fast/LongStackWriteDuringRead
-- ADDING SIMULATOR TEST 21 fast/LowLatency
-- ADDING SIMULATOR TEST 22 fast/MemoryLifetime
-- ADDING SIMULATOR TEST 23 fast/MoveKeysCycle
-- ADDING SIMULATOR TEST 24 fast/RandomSelector
-- ADDING SIMULATOR TEST 25 fast/RandomUnitTests
-- ADDING SIMULATOR TEST 26 fast/SelectorCorrectness
-- ADDING SIMULATOR TEST 27 fast/Sideband
-- ADDING SIMULATOR TEST 28 fast/SidebandWithStatus
-- ADDING SIMULATOR TEST 29 fast/SwizzledRollbackSideband
-- ADDING SIMULATOR TEST 30 fast/SystemRebootTestCycle
-- ADDING SIMULATOR TEST 31 fast/TaskBucketCorrectness
-- ADDING SIMULATOR TEST 32 fast/TimeKeeperCorrectness
-- ADDING SIMULATOR TEST 33 fast/TxnStateStoreCycleTest
-- ADDING SIMULATOR TEST 34 fast/Unreadable
-- ADDING SIMULATOR TEST 35 fast/VersionStamp
-- ADDING SIMULATOR TEST 36 fast/Watches
-- ADDING SIMULATOR TEST 37 fast/WriteDuringRead
-- ADDING SIMULATOR TEST 38 fast/WriteDuringReadClean
-- ADDING SIMULATOR TEST 39 rare/CheckRelocation
-- ADDING SIMULATOR TEST 40 rare/ClogUnclog
-- ADDING SIMULATOR TEST 41 rare/CloggedCycleWithKills
-- ADDING SIMULATOR TEST 42 rare/ConflictRangeCheck
-- ADDING SIMULATOR TEST 43 rare/ConflictRangeRYOWCheck
-- ADDING SIMULATOR TEST 44 rare/CycleRollbackClogged
-- ADDING SIMULATOR TEST 45 rare/CycleWithKills
-- ADDING SIMULATOR TEST 46 rare/FuzzTest
-- ADDING SIMULATOR TEST 47 rare/InventoryTestHeavyWrites
-- ADDING SIMULATOR TEST 48 rare/LargeApiCorrectness
-- ADDING SIMULATOR TEST 49 rare/LargeApiCorrectnessStatus
-- ADDING SIMULATOR TEST 50 rare/RYWDisable
-- ADDING SIMULATOR TEST 51 rare/RandomReadWriteTest
-- ADDING SIMULATOR TEST 52 rare/SwizzledLargeApiCorrectness
-- ADDING SIMULATOR TEST 53 rare/RedwoodCorrectnessBTree
-- ADDING SIMULATOR TEST 54 restarting/from_7.0.0/ConfigureTestRestart-1
-- ADDING SIMULATOR TEST 56 restarting/CycleTestRestart-1
-- ADDING SIMULATOR TEST 58 restarting/StorefrontTestRestart-1
-- ADDING SIMULATOR TEST 60 restarting/from_6.2.0/SnapTestAttrition-1
-- ADDING SIMULATOR TEST 62 restarting/from_6.2.0/SnapTestSimpleRestart-1
-- ADDING SIMULATOR TEST 64 restarting/from_6.2.0/SnapTestRestart-1
-- ADDING SIMULATOR TEST 66 restarting/from_6.2.0/SnapCycleRestart-1
-- ADDING SIMULATOR TEST 68 restarting/from_5.1.7/DrUpgradeRestart-1
-- ADDING SIMULATOR TEST 70 restarting/from_5.2.0/ClientTransactionProfilingCorrectness-1
-- ADDING SIMULATOR TEST 72 slow/ApiCorrectness
-- ADDING SIMULATOR TEST 73 slow/ApiCorrectnessAtomicRestore
-- ADDING SIMULATOR TEST 74 slow/ApiCorrectnessSwitchover
-- ADDING SIMULATOR TEST 75 slow/ClogWithRollbacks
-- ADDING SIMULATOR TEST 76 slow/CloggedCycleTest
-- ADDING SIMULATOR TEST 77 slow/CloggedStorefront
-- ADDING SIMULATOR TEST 78 slow/CommitBug
-- ADDING SIMULATOR TEST 79 slow/ConfigureTest
-- ADDING SIMULATOR TEST 80 slow/CycleRollbackPlain
-- ADDING SIMULATOR TEST 81 slow/DDBalanceAndRemove
-- ADDING SIMULATOR TEST 82 slow/DDBalanceAndRemoveStatus
-- ADDING SIMULATOR TEST 83 slow/DifferentClustersSameRV
-- ADDING SIMULATOR TEST 84 slow/FastTriggeredWatches
-- ADDING SIMULATOR TEST 85 slow/LowLatencyWithFailures
-- ADDING SIMULATOR TEST 86 slow/MoveKeysClean
-- ADDING SIMULATOR TEST 87 slow/MoveKeysSideband
-- ADDING SIMULATOR TEST 88 slow/RyowCorrectness
-- ADDING SIMULATOR TEST 89 slow/Serializability
-- ADDING SIMULATOR TEST 90 slow/SharedBackupCorrectness
-- ADDING SIMULATOR TEST 91 slow/SharedBackupToDBCorrectness
-- ADDING SIMULATOR TEST 92 slow/StorefrontTest
-- ADDING SIMULATOR TEST 93 slow/SwizzledApiCorrectness
-- ADDING SIMULATOR TEST 94 slow/SwizzledCycleTest
-- ADDING SIMULATOR TEST 95 slow/SwizzledDdBalance
-- ADDING SIMULATOR TEST 96 slow/SwizzledRollbackTimeLapse
-- ADDING SIMULATOR TEST 97 slow/SwizzledRollbackTimeLapseIncrement
-- ADDING SIMULATOR TEST 98 slow/VersionStampBackupToDB
-- ADDING SIMULATOR TEST 99 slow/VersionStampSwitchover
-- ADDING SIMULATOR TEST 100 slow/WriteDuringReadAtomicRestore
-- ADDING SIMULATOR TEST 101 slow/WriteDuringReadSwitchover
-- ADDING SIMULATOR TEST 102 slow/ddbalance
-- ADDING SIMULATOR TEST 103 slow/ParallelRestoreCorrectnessAtomicOpTinyData
-- ADDING SIMULATOR TEST 104 slow/ParallelRestoreCorrectnessCycle
-- ADDING SIMULATOR TEST 105 status/invalid_proc_addresses
-- ADDING SIMULATOR TEST 106 status/local_6_machine_no_replicas_remain
-- ADDING SIMULATOR TEST 107 status/separate_1_of_3_coordinators_remain
-- ADDING SIMULATOR TEST 108 status/separate_2_of_3_coordinators_remain
-- ADDING SIMULATOR TEST 109 status/separate_cannot_write_cluster_file
-- ADDING SIMULATOR TEST 110 status/separate_idle
-- ADDING SIMULATOR TEST 111 status/separate_initializing
-- ADDING SIMULATOR TEST 112 status/separate_no_coordinators
-- ADDING SIMULATOR TEST 113 status/separate_no_database
-- ADDING SIMULATOR TEST 114 status/separate_no_servers
-- ADDING SIMULATOR TEST 115 status/separate_not_enough_servers
-- ADDING SIMULATOR TEST 116 status/single_process_too_many_config_params
-- =========================================
--    Components Build Overview 
-- =========================================
-- Build Java Bindings:                  OFF
-- Build with TLS support:               OFF
-- Build Go bindings:                    ON
-- Build Ruby bindings:                  ON
-- Build Python sdist (make package):    ON
-- Build Documentation (make html):      OFF
-- =========================================
-- CPACK_COMPONENTS_ALL clients-deb;clients-el6;clients-el7;clients-pm;clients-tgz;server-deb;server-el6;server-el7;server-pm;server-tgz
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/.build
[32/304] Linking CXX executable bin/actor_flamegraph
c++: warning: argument unused during compilation: '-static-libstdc++' [-Wunused-command-line-argument]
c++: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument]
[267/304] Linking CXX executable bin/fdbmonitor
c++: warning: argument unused during compilation: '-static-libstdc++' [-Wunused-command-line-argument]
c++: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument]
[280/304] Linking CXX shared library lib/libfdb_c.so
c++: warning: argument unused during compilation: '-static-libstdc++' [-Wunused-command-line-argument]
c++: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument]
[281/304] Linking CXX executable bin/fdb_c_performance_test
c++: warning: argument unused during compilation: '-static-libstdc++' [-Wunused-command-line-argument]
c++: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument]
[282/304] Linking CXX shared library share/foundationdb/libc_workloads.so
c++: warning: argument unused during compilation: '-static-libstdc++' [-Wunused-command-line-argument]
c++: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument]
[283/304] Linking CXX executable bin/mako
c++: warning: argument unused during compilation: '-static-libstdc++' [-Wunused-command-line-argument]
c++: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument]
[284/304] Linking CXX executable bin/fdb_c_txn_size_test
c++: warning: argument unused during compilation: '-static-libstdc++' [-Wunused-command-line-argument]
c++: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument]
[285/304] Linking CXX executable bin/fdb_c_ryw_benchmark
c++: warning: argument unused during compilation: '-static-libstdc++' [-Wunused-command-line-argument]
c++: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument]
[289/304] Linking CXX executable bin/fdb_flow_tester
c++: warning: argument unused during compilation: '-static-libstdc++' [-Wunused-command-line-argument]
c++: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument]
[298/304] Linking CXX executable bin/fdbbackup
c++: warning: argument unused during compilation: '-static-libstdc++' [-Wunused-command-line-argument]
c++: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument]
[301/304] Linking CXX executable bin/fdbcli
c++: warning: argument unused during compilation: '-static-libstdc++' [-Wunused-command-line-argument]
c++: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument]
[304/304] Linking CXX executable bin/fdbserver
c++: warning: argument unused during compilation: '-static-libstdc++' [-Wunused-command-line-argument]
c++: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument]
Test project /tmp/.build
      Start  2: fast/AtomicBackupCorrectness
 1/38 Test  #2: fast/AtomicBackupCorrectness .......   Passed  166.66 sec
      Start  3: fast/AtomicBackupToDBCorrectness
 2/38 Test  #3: fast/AtomicBackupToDBCorrectness ...   Passed  107.58 sec
      Start  4: fast/AtomicOps
 3/38 Test  #4: fast/AtomicOps .....................   Passed   13.47 sec
      Start  5: fast/AtomicOpsApiCorrectness
 4/38 Test  #5: fast/AtomicOpsApiCorrectness .......   Passed    4.37 sec
      Start  6: fast/BackupCorrectness
 5/38 Test  #6: fast/BackupCorrectness .............   Passed   26.83 sec
      Start  7: fast/BackupCorrectnessClean
 6/38 Test  #7: fast/BackupCorrectnessClean ........   Passed   39.59 sec
      Start  8: fast/BackupToDBCorrectness
 7/38 Test  #8: fast/BackupToDBCorrectness .........   Passed  148.39 sec
      Start  9: fast/BackupToDBCorrectnessClean
 8/38 Test  #9: fast/BackupToDBCorrectnessClean ....   Passed   59.89 sec
      Start 10: fast/CloggedSideband
 9/38 Test #10: fast/CloggedSideband ...............   Passed    8.96 sec
      Start 11: fast/ConstrainedRandomSelector
10/38 Test #11: fast/ConstrainedRandomSelector .....   Passed    4.60 sec
      Start 12: fast/CycleAndLock
11/38 Test #12: fast/CycleAndLock ..................   Passed  235.82 sec
      Start 13: fast/CycleTest
12/38 Test #13: fast/CycleTest .....................   Passed   29.55 sec
      Start 14: fast/FuzzApiCorrectness
13/38 Test #14: fast/FuzzApiCorrectness ............   Passed   42.50 sec
      Start 15: fast/FuzzApiCorrectnessClean
14/38 Test #15: fast/FuzzApiCorrectnessClean .......   Passed   80.18 sec
      Start 16: fast/IncrementTest
15/38 Test #16: fast/IncrementTest .................   Passed   18.56 sec
      Start 17: fast/InventoryTestAlmostReadOnly
16/38 Test #17: fast/InventoryTestAlmostReadOnly ...   Passed    2.77 sec
      Start 18: fast/InventoryTestSomeWrites
17/38 Test #18: fast/InventoryTestSomeWrites .......   Passed    9.96 sec
      Start 19: fast/KillRegionCycle
18/38 Test #19: fast/KillRegionCycle ...............   Passed   26.41 sec
      Start 20: fast/LocalRatekeeper
19/38 Test #20: fast/LocalRatekeeper ...............   Passed   24.07 sec
      Start 21: fast/LongStackWriteDuringRead
20/38 Test #21: fast/LongStackWriteDuringRead ......   Passed    5.36 sec
      Start 22: fast/LowLatency
21/38 Test #22: fast/LowLatency ....................   Passed   34.69 sec
      Start 23: fast/MemoryLifetime
22/38 Test #23: fast/MemoryLifetime ................   Passed   64.43 sec
      Start 24: fast/MoveKeysCycle
23/38 Test #24: fast/MoveKeysCycle .................   Passed    6.24 sec
      Start 25: fast/RandomSelector
24/38 Test #25: fast/RandomSelector ................   Passed    5.52 sec
      Start 26: fast/RandomUnitTests
25/38 Test #26: fast/RandomUnitTests ...............   Passed    0.36 sec
      Start 27: fast/SelectorCorrectness
26/38 Test #27: fast/SelectorCorrectness ...........   Passed    3.82 sec
      Start 28: fast/Sideband
27/38 Test #28: fast/Sideband ......................   Passed   11.25 sec
      Start 29: fast/SidebandWithStatus
28/38 Test #29: fast/SidebandWithStatus ............   Passed    7.48 sec
      Start 30: fast/SwizzledRollbackSideband
29/38 Test #30: fast/SwizzledRollbackSideband ......   Passed   22.03 sec
      Start 31: fast/SystemRebootTestCycle
30/38 Test #31: fast/SystemRebootTestCycle .........   Passed   27.36 sec
      Start 32: fast/TaskBucketCorrectness
31/38 Test #32: fast/TaskBucketCorrectness .........   Passed    3.21 sec
      Start 33: fast/TimeKeeperCorrectness
32/38 Test #33: fast/TimeKeeperCorrectness .........   Passed   21.25 sec
      Start 34: fast/TxnStateStoreCycleTest
33/38 Test #34: fast/TxnStateStoreCycleTest ........   Passed   15.13 sec
      Start 35: fast/Unreadable
34/38 Test #35: fast/Unreadable ....................   Passed    5.00 sec
      Start 36: fast/VersionStamp
35/38 Test #36: fast/VersionStamp ..................   Passed   12.18 sec
      Start 37: fast/Watches
36/38 Test #37: fast/Watches .......................   Passed   78.46 sec
      Start 38: fast/WriteDuringRead
37/38 Test #38: fast/WriteDuringRead ...............   Passed   13.99 sec
      Start 39: fast/WriteDuringReadClean
38/38 Test #39: fast/WriteDuringReadClean ..........   Passed   18.52 sec

100% tests passed, 0 tests failed out of 38

Label Time Summary:
fast    = 1406.44 sec*proc (38 tests)

Total Test time (real) = 1406.53 sec

dch · April 8, 2020, 1:41pm

TLDR I think we are good to go now. I’ve address last round of Alex’s feedback (many thanks for your review), rebased the PR to master, and it:

Apple CI & PR

passes the CMake test variants in the foundationdb CI
compiles with latest default clang on 12.1 & 13.0-CURRENT
passes the “fast” test suite 100%
builds against either OpenSSL or LibreSSL from ports for TLS support
builds in FreeBSD package build cluster pass latest clang

For the FreeBSD port

I’ll add the remaining 2 items to the port once the main PR lands:

build as many of the bindings as possible, as optional components
make SSL usable, ideally as a static build so we dont have a runtime dependency on it

dch · April 30, 2020, 12:28pm

@alexmiller is there anything you’re waiting on that stops this getting committed?

AFAICT the only remaining issues mentioned in the review are:

boost libs dependencies. FreeBSD ports now ships with 1.72, so perhaps I can remove that. Note to self see https://github.com/apple/foundationdb/issues/1312
add github issue for JVM dependencies on FreeBSD as mentioned I will do these once the first PR goes in, so I’m not working off a moving target

I will rebase the FreeBSD PR https://github.com/apple/foundationdb/pull/2634 to master again today as there are a few new merge conflicts creeping in.

alexmiller · May 1, 2020, 6:54am

I had been waiting on the couple issues I flagged during review to be resolved, but given the maintenance overhead of keeping the PR outstanding, I’ve merged it now, and we can clean up the couple unresolved issues post-merge.

Congrats, and thanks for all the work!

dch · May 1, 2020, 11:23am

Thanks for your review time, it was really appreciated. I’d like to add a FreeBSD CI image to the build process, whether it runs locally with one of my systems, or it’s something that would be set up in/at apple, is that possible? I’m happy to do the leg work obviously.

dch · October 31, 2020, 8:41am

@alexmiller bump wrt CI options

alexmiller · November 2, 2020, 10:14pm

Sorry!

All of the existing CI checks are driven off of an Apple-managed Jenkins instance. If we were to make FreeBSD a required check, I think we’d need a centrally managed FreeBSD image/build environment and a story for how we’d update it with new versions/dependencies/repos over time.

I think we’d be able to set up a webhook to notify a jenkins/etc. setup of yours of new PRs and give permissions to a github account to be able to report PR status checks. As we wouldn’t be able to fix anything that goes wrong on the infra side, I think we’d need to leave a check done this way as non-required.

@alvin is the owner of the build infrastructure though, and would have the final say either way.

dch · November 3, 2020, 9:39pm

Fair enough. I’m happy if we can get a webhook on commit, to start with. Maybe @alvin can contact me via email?