FoundationDB CLI 6.2 (v6.2.20)
source version 77b5171e81754f2fda8869703d662e59d85b7f23
protocol fdb00b062010001
The status switches between
Unable to communicate with the cluster controller at 10.19.0.5:4650 to get
status.
to
Recruiting new transaction servers.
Need at least 2 log servers across unique zones, 1 proxies and 1 resolvers.
Have 24 non-excluded processes on 8 machines across 8 zones.
Unable to locate the data distributor worker.
Unable to locate the ratekeeper worker.We have tried upsizing the mem capacity of the cc.
Tried include all, but did not help.
addr2line -e fdbserver.debug -p -C -f -i 0x1986185 0x7fdd58a8c890 0x149acaf 0xf01669 0xf01e0c 0xf098ad 0xf09b90 0xef5490 0xef62a7 0x8ff69c 0x18edf16 0x18ee228 0x18ee691 0x18eee98 0x18e946b 0x18ea6bd 0x7f2ba0 0x19d92f0 0x66e06e 0x7fdd58104b97
shows
$ addr2line -e ./fdbserver.debug -p -C -f -i 0x1986185 0x7fdd58a8c890 0x149acaf 0xf01669 0xf01e0c 0xf098ad 0xf09b90 0xef5490 0xef62a7 0x8ff69c 0x18edf16 0x18ee228 0x18ee691 0x18eee98 0x18e9 46b 0x18ea6bd 0x7f2ba0 0x19d92f0 0x66e06e 0x7fdd58104b97 crashHandler(int) at /opt/foundation/foundationdb/flow/Platform.cpp:2780 ?? ??:0 DatabaseConfiguration::toString() const at /opt/boost_1_67_0/boost/variant/detail/initializer.hpp:110 (inlined by) ?? at /opt/boost_1_67_0/boost/variant/variant.hpp:1665 (inlined by) ?? at /opt/boost_1_67_0/boost/variant/variant.hpp:1828 (inlined by) ?? at /opt/foundation/foundationdb/./fdbclient/json_spirit/json_spirit_value.h:279 (inlined by) DatabaseConfiguration::toString() const at /opt/foundation/foundationdb/fdbclient/DatabaseConfiguration.cpp:384 (anonymous namespace)::RecruitEverythingActorState<(anonymous namespace)::RecruitEverythingActor>::a_body1(int) [clone .isra.3749] at /opt/foundation/foundationdb/fdbserver/masterserver.actor.cpp:548 recruitEverything(Reference<MasterData> const&, std::vector<StorageServerInterface, std::allocator<StorageServerInterface> >* const&, Reference<ILogSystem> const&) at /opt/foundation/foundationdb/./flow/flow.h:746 (inlined by) recruitEverything(Reference<MasterData> const&, std::vector<StorageServerInterface, std::allocator<StorageServerInterface> >* const&, Reference<ILogSystem> const&) at /opt/foundation/foundationdb/fdbserver/masterserver.actor.cpp:544 (anonymous namespace)::RecoverFromActorState<(anonymous namespace)::RecoverFromActor>::a_body1cont4loopBody1when2(Standalone<CommitTransactionRef> const&, int) at /opt/foundation/foundationdb/./flow/flow.h:715 (discriminator 1) (inlined by) a_body1cont4loopBody1when2 at /opt/foundation/foundationdb/fdbserver/masterserver.actor.cpp:882 (discriminator 1) ActorCallback<(anonymous namespace)::RecoverFromActor, 4, Standalone<CommitTransactionRef> >::fire(Standalone<CommitTransactionRef> const&) at /opt/foundation/foundationdb/./flow/FastRef.h:114 (inlined by) ?? at /opt/foundation/foundationdb/./flow/Arena.h:92 (inlined by) ?? at /opt/foundation/foundationdb/./flow/Arena.h:340 (inlined by) fire at /opt/foundation/foundationdb/./flow/flow.h:999 (anonymous namespace)::ProvisionalMasterActorState<(anonymous namespace)::ProvisionalMasterActor>::a_body1cont1loopBody1when2(CommitTransactionRequest const&, int) at /opt/foundation/foundationdb/./flow/flow.h:478 (inlined by) a_body1cont1loopBody1when2 at /opt/foundation/foundationdb/.objs/fdbserver/masterserver.actor.g.cpp:2897 ActorSingleCallback<(anonymous namespace)::ProvisionalMasterActor, 2, CommitTransactionRequest>::fire(CommitTransactionRequest const&) at /opt/foundation/foundationdb/./fdbrpc/fdbrpc.h:135 (inlined by) ?? at /opt/foundation/foundationdb/./fdbclient/MasterProxyInterface.h:117 (inlined by) fire at /opt/foundation/foundationdb/./flow/flow.h:1009 NetNotifiedQueue<CommitTransactionRequest>::receive(ArenaObjectReader&) at /opt/foundation/foundationdb/./flow/flow.h:598 (inlined by) ?? at /opt/foundation/foundationdb/./flow/flow.h:594 (inlined by) NetNotifiedQueue<CommitTransactionRequest>::receive(ArenaObjectReader&) at /opt/foundation/foundationdb/./fdbrpc/fdbrpc.h:253 (anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_body1cont1(int) [clone .isra.924] at /opt/foundation/foundationdb/./flow/FastRef.h:114 (inlined by) ?? at /opt/foundation/foundationdb/./flow/Arena.h:92 (inlined by) ?? at /opt/foundation/foundationdb/./flow/ObjectSerializer.h:120 (inlined by) ?? at /opt/foundation/foundationdb/fdbrpc/FlowTransport.actor.cpp:653 deliver(TransportData* const&, Endpoint const&, ArenaReader const&, bool const&) at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:2258 (inlined by) ?? at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:2489 (inlined by) deliver at /opt/foundation/foundationdb/fdbrpc/FlowTransport.actor.cpp:638 scanPackets(TransportData*, unsigned char*&, unsigned char const*, Arena&, NetworkAddress const&, ProtocolVersion) [clone .isra.929] at /opt/foundation/foundationdb/./fdbrpc/FlowTransport.h:32 (inlined by) scanPackets at /opt/foundation/foundationdb/fdbrpc/FlowTransport.actor.cpp:770 (anonymous namespace)::ConnectionReaderActorState<(anonymous namespace)::ConnectionReaderActor>::a_body1loopBody1loopBody1cont4(int) at /opt/foundation/foundationdb/fdbrpc/FlowTransport.actor.cpp:942 (anonymous namespace)::ConnectionReaderActorState<(anonymous namespace)::ConnectionReaderActor>::a_body1loopBody1loopBody1cont1(int) at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:3173 (inlined by) a_body1loopBody1loopBody1cont1 at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:3017 ActorCallback<(anonymous namespace)::ConnectionReaderActor, 4, Void>::fire(Void const&) at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:3069 (inlined by) ?? at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:3050 (inlined by) ?? at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:3030 (inlined by) ?? at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:2807 (inlined by) ?? at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:2762 (inlined by) ?? at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:2739 (inlined by) ?? at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:2730 (inlined by) ?? at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:3327 (inlined by) ?? at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:3333 (inlined by) ?? at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:3331 (inlined by) ?? at /opt/foundation/foundationdb/.objs/fdbrpc/FlowTransport.actor.g.cpp:3347 (inlined by) fire at /opt/foundation/foundationdb/./flow/flow.h:999 void SAV<Void>::send<Void>(Void&&) at /opt/foundation/foundationdb/./flow/flow.h:446 N2::Net2::run() at /opt/foundation/foundationdb/./flow/flow.h:791 (inlined by) ?? at /opt/foundation/foundationdb/flow/Net2.actor.cpp:852 (inlined by) ?? at /opt/foundation/foundationdb/flow/Net2.actor.cpp:859 (inlined by) N2::Net2::run() at /opt/foundation/foundationdb/flow/Net2.actor.cpp:1144 main at /opt/foundation/foundationdb/fdbserver/fdbserver.actor.cpp:1787 ?? ??:0
Adding a new additional node immediately solved the issue somehow.