Run 7.2.0 fdbserver multitest has segmentation fault

Run multitest but failed:

# gdb --args fdbserver -r multitest -f /root/test.txt

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/fdbserver...Reading symbols from /usr/sbin/fdbserver...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Missing separate debuginfos, use: debuginfo-install foundationdb-server-7.2.0-1.x86_64
(gdb) r
Starting program: /usr/sbin/fdbserver -r multitest -f /root/test.txt
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff67ff700 (LWP 1221031)]
[New Thread 0x7ffff57e9700 (LWP 1221032)]
startingConfiguration: start
useDB: true

Program received signal SIGSEGV, Segmentation fault.
0x00000000026489de in Reference<IReplicationPolicy>::operator=(Reference<IReplicationPolicy> const&) [clone .isra.2917] ()
(gdb) bt
#0  0x00000000026489de in Reference<IReplicationPolicy>::operator=(Reference<IReplicationPolicy> const&) [clone .isra.2917] ()
#1  0x00000000026a9b21 in (anonymous namespace)::RunTestsActorState<(anonymous namespace)::RunTestsActor>::a_body1cont1loopBody1cont2(Standalone<RangeResultRef> const&, int) ()
#2  0x00000000026aa91b in ActorCallback<(anonymous namespace)::RunTestsActor, 2, Standalone<RangeResultRef> >::fire(Standalone<RangeResultRef> const&) ()
#3  0x00000000037d0038 in SAV<Standalone<RangeResultRef> >::finishSendAndDelPromiseRef() [clone .constprop.25895] ()
#4  0x00000000038f8533 in (anonymous namespace)::GetRangeActorState<GetKeyValuesRequest, GetKeyValuesReply, Standalone<RangeResultRef>, (anonymous namespace)::GetRangeActor<GetKeyValuesRequest, GetKeyValuesReply, Standalone<RangeResultRef> > >::a_body1cont3loopBody1cont9(int) ()
#5  0x00000000039138e7 in ActorCallback<(anonymous namespace)::GetRangeActor<GetKeyValuesRequest, GetKeyValuesReply, Standalone<RangeResultRef> >, 2, GetKeyValuesReply>::fire(GetKeyValuesReply const&) ()
#6  0x0000000003858040 in _ZN13ActorCallbackIN12_GLOBAL__N_19FmapActorIZNS0_11loadBalanceI22StorageServerInterface19GetKeyValuesRequestLb1EEE6FutureIDTcldtcldtcl15getReplyPromisecl7declvalIT0_EEE9getFutureE8getValueEEEP15DatabaseContext9ReferenceI12LocationInfoEMT_13RequestStreamIS6_XT1_EERKS6_12TaskPriority10AtMostOnceP10QueueModelEUlRKSE_E_17GetKeyValuesReplyEELi0ESR_E4fireERKSR_ ()
#7  0x00000000038f4c60 in (anonymous namespace)::LoadBalanceActorState<StorageServerInterface, GetKeyValuesRequest, ReferencedInterface<StorageServerInterface>, true, (anonymous namespace)::LoadBalanceActor<StorageServerInterface, GetKeyValuesRequest, ReferencedInterface<StorageServerInterface>, true> >::a_body1loopBody1loopBody2when1(ErrorOr<GetKeyValuesReply> const&, int) ()
#8  0x00000000038f4dae in ActorCallback<(anonymous namespace)::LoadBalanceActor<StorageServerInterface, GetKeyValuesRequest, ReferencedInterface<StorageServerInterface>, true>, 5, ErrorOr<GetKeyValuesReply> >::fire(ErrorOr<GetKeyValuesReply> const&) ()
#9  0x0000000001604958 in ActorCallback<(anonymous namespace)::WaitValueOrSignalActor<GetKeyValuesReply>, 0, GetKeyValuesReply>::fire(GetKeyValuesReply const&) ()
#10 0x0000000000d62960 in NetSAV<GetKeyValuesReply>::receive(ArenaObjectReader&) ()
#11 0x00000000041e2c57 in (anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_body1cont1(int) [clone .isra.1235] ()
#12 0x00000000041e331b in ActorCallback<(anonymous namespace)::DeliverActor, 0, Void>::fire(Void const&) ()
#13 0x00000000015a3040 in void SAV<Void>::send<Void>(Void&&) ()
#14 0x000000000438bb32 in N2::Net2::run() ()
#15 0x0000000000c82e36 in main ()

Using 7.2.0:

# fdbserver --version
FoundationDB 7.2 (v7.2.0)
source version 5eae3be195ee5c1302878459d3d1d34282b1ee60
protocol fdb00b072000000

# addr2line -e fdbserver.debug -p -C -f -i 0x7f277dd67630 0x26a9b21 0x26aa91b 0x37d0038 0x38f8533 0x39138e7 0x3858040 0x38f4c60 0x38f4dae 0x1604958 0xd62960 0x41e2c57 0x41e331b 0x15a3040 0x438bb32 0xc82e36 0x7f277d9ac555
?? ??:0
a_body1cont17 at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/tester.actor.cpp:1785
 (inlined by) a_body1cont1break1 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/tester.actor.g.cpp:10167
 (inlined by) a_body1cont1loopBody1cont2 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/tester.actor.g.cpp:10213
a_callback_fire at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/tester.actor.g.cpp:10258
 (inlined by) fire at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:1313
SAV<Standalone<RangeResultRef> >::finishSendAndDelPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:688
(anonymous namespace)::GetRangeActorState<GetKeyValuesRequest, GetKeyValuesReply, Standalone<RangeResultRef>, (anonymous namespace)::GetRangeActor<GetKeyValuesRequest, GetKeyValuesReply, Standalone<RangeResultRef> > >::a_body1cont3loopBody1cont9(int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbclient/NativeAPI.actor.g.cpp:20233
a_callback_fire at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbclient/NativeAPI.actor.g.cpp:20364
 (inlined by) fire at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:1313
SAV<GetKeyValuesReply>::finishSendAndDelPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:688
 (inlined by) a_body1cont1 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/include/flow/genericactors.actor.g.h:21757
 (inlined by) a_body1when1 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/include/flow/genericactors.actor.g.h:21776
 (inlined by) a_callback_fire at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/include/flow/genericactors.actor.g.h:21797
 (inlined by) fire at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:1313
SAV<GetKeyValuesReply>::finishSendAndDelPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:688
 (inlined by) a_body1loopBody1loopBody2when1 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/include/fdbrpc/LoadBalance.actor.g.h:2197
 (inlined by) a_body1loopBody1loopBody2when1 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/include/fdbrpc/LoadBalance.actor.g.h:2176
a_callback_fire at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/include/fdbrpc/LoadBalance.actor.g.h:2295
 (inlined by) fire at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:1313
SAV<ErrorOr<GetKeyValuesReply> >::finishSendAndDelPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:688
 (inlined by) a_body1loopBody1when1 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/include/fdbrpc/genericactors.actor.g.h:5943
 (inlined by) a_callback_fire at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/include/fdbrpc/genericactors.actor.g.h:5996
 (inlined by) fire at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:1313
SAV<GetKeyValuesReply>::finishSendAndDelPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:688
 (inlined by) void SAV<GetKeyValuesReply>::sendAndDelPromiseRef<GetKeyValuesReply&>(GetKeyValuesReply&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:682
 (inlined by) NetSAV<GetKeyValuesReply>::receive(ArenaObjectReader&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/include/fdbrpc/fdbrpc.h:110
NetworkAddress::NetworkAddress() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/network.h:218
 (inlined by) NetworkAddressList::NetworkAddressList() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/network.h:303
 (inlined by) a_body1cont1 at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/FlowTransport.actor.cpp:1043
a_callback_fire at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/FlowTransport.actor.g.cpp:4373
 (inlined by) fire at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:1313
void SAV<Void>::send<Void>(Void&&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:654
Promise<Void>::~Promise() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:922
 (inlined by) N2::Net2::PromiseTask::~PromiseTask() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Net2.actor.cpp:253
 (inlined by) N2::Net2::PromiseTask::operator()() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Net2.actor.cpp:260
 (inlined by) N2::Net2::run() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Net2.actor.cpp:1492
main at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/fdbserver.actor.cpp:2426 (discriminator 3)
?? ??:0

Looks like it’s a bug. @jzhou

Yes. This bug has been fixed by Fix consistency check crashes [release-7.2] by jzhou77 · Pull Request #9182 · apple/foundationdb · GitHub and will be included in 7.2.4 release.

1 Like

Hope to see the release of this version soon. :grinning: