Hi All,
I am starting to look at porting FoundationDB code to work on Power (ppc64le) and seems that there is a lot of architecture specific code (dependency on x86_64).
Just wanted to check if there is any general guidance on how to approach this, how complex this might be and if it has been ported (or a port attempted) to any other architecture, which can be used as a reference?
Thanks!
Priya
Iām curious about this too given the way things are going Apple will probably replace their laptopsā intel chips with ARM chips soon. And thereās some really nice cheap arm chips on AWS now.
How much asm is there in foundationdb?
As far as I know you will be the first to try porting to a different CPU architecture.
Obvious architecture dependent stuff:
the binary compatibility version dispatching in the client library (bindings/c/generate_asm.py)
the coroutine library (fdbrpc/libcoroutine/), which already has code for several architectures but we havenāt tested it or kept it up to date
use of rdtsc as a timing source
probably various intrinsics scattered around
probably lots of stuff that assumes little endian (but you say you are le)
probably some stuff that assumes unaligned memory access is possible and reasonably performant
Iām sure there is more.
Care to share why, or what your use case is? 
I think @wolfspider has some amount of an ARM port done that heās posted about before.
Heya @alexmiller I can provide some color. Weāve a number of ideas for use of FoundationDB in IBM. Many of those use cases are motivated by the needs of services delivered in IBM Cloud. Those services often have an on premises counterpart with a consistent API to help our clients build portable applications, and many of our clients run their on prem āprivate cloudā environment on Power hardware. So, weāre looking ahead to make sure that we donāt find ourselves scrambling to port something here. I can think of at least two customers where FoundationDB and Power could intersect in the coming year.
Priyaās team has the broader mission to support a robust open source ecosystem on Power, and Iāve asked for their help to figure out what we might be up against here 
If you want to build and maintain a port of FDB to a new architecture, I would recommend budgeting for ongoing compute and human time to run a full spectrum of simulation and other tests on each release. Otherwise the port may be significantly less trustworthy than the official releases.
It would be nice if there was a document somewhere describing a ābaselineā testing routine for FoundationDB porters, etc. I donāt exactly know what integration strategies Apple uses, for example. (For instance, NixOS users canāt use foundationdb.org binaries, so we have to package our own). I ran many of the simulation tests under tests/ with our builds ā but only after figuring out even how to do that by reading forum threads, etc. It would be nice to have a document saying something like, āHere are the tests you need to run manually, and here are tests that can be fully automated by running fdbserver in simulation modeā, and how to do so.
It would also be nice to know about the memory/compute requirements for this. For instance, I can write a test for NixOS that can execute a set of test routes running FoundationDB, including things like simulation runs; this is how we integrate and test other databases, such as CockroachDB, too (this link contains a full end-to-end cluster test). While this might not be enabled by default (it could take hours), if the requirements are known, I can certainly run tests myself regularly, etc.
Perhaps this already exists somewhere in the source code? I still think this would be good information to have in the documentation so itās easier to find.
On a related note, it would probably be nice on top of this if the GitHub repository for FoundationDB could be set up to use a CI system that ran such tests somehow. This is not only very useful for you but also important for contributors, etc. (Right now I would simply have no clue how to test my changes before submitting them for review). Iād guess that a public CI system for GitHub would probably run a smaller set of tests to keep iteration times lower.
(A useful CI system to examine might be Azure Pipelines, since itās got a free version and does actually support Windows/Linux/Mac natively, and itās the only one I can think of that does support all 3 out of the box.)
I donāt see a PowerPC port for libcoroutine that anyone has already done. FWIW, I wouldnāt be opposed to a PR switching from coroutine to Boost.Context instead.
I think this is going to, somewhat by necessity, fall out of the āport FDB to FreeBSDā PR, because part of that involves a discussion of āwhen would this be marked as stableā and the list of requirements that would entail.
Porting to a new architecture but same platform would likely involve a separate list, as trying to verify that the networking stack or file system doesnāt behave detrimentally differently wouldnāt be needed. Iām happy to sit down and think hard and chat with folks if someone shows up with an FDB port to a new arch, like PowerPC.
Thereās a CMake build system thatās been added, that also has CTest support. Iād recommend using that, because a lot of work was put into getting ctest able to run simulation tests locally in parallel. If ctest passes for you locally, thatās a reasonable time to upload a PR. That still only gives you O(100) runs of simulation versus the O(50,000) runs that Iād generally aim for if itās a serious change to the core, but if thereās any concern, your reviewer will pull the PR and kick off more simulation runs in a cluster of machines to close that testing gap.
Iā¦ donāt see that in the README, which is a pretty good point.
It currently can! That that support was added a week ago or something, though. (The Correctness check on this PR, for example.) It runs ctest -L fast, and thereās commands for reviewers to issue to get it to do a more comprehensive testing if requested.
Thanks all for your inputs!!
Sorry for delay in getting back, I have been following the discussion here, but have just about resumed active work on the port and am able to make some progress in getting an initial build going.
Further to initial inputs from @dave, changes are mostly for
ā¢use of rdtsc as a timing source : introduced appropriate code / implementation for Power.
ā¢probably various intrinsics scattered around : Changes were mostly need for putting in appropriate changes in place of _mm_pause and _mm_prefetch
For the first build, I have by-passed
ā¢crc implementation
ā¢the binary compatibility version dispatching in the client library (bindings/c/generate_asm.py)
The build is reaching the 100% success mark and following binaries / libraries are generated
fdbbackup fdbcli fdbmonitor fdbserver
libfdb_sqlite.a libfdbclient.a libfdbrpc.a libflow.a libthirdparty.a
I have started working on the CRC changes now.
On the testing front, I tried to run a few tests, as described here From Build to Running Tests
and they seemed to pass, however wanted to understand if there is an easy way to do a complete test setup and execute all the available tests in one go?
Thanks in advance!
Regards,
Priya
If you set up cmake, with instructions in the README, then running ctest in your build directory will give you the automated test setup and execution of all tests that youāre looking for. If ctest passes for you locally, then thatās a good point at which to post a PR with reasonable confidence.
Thanks @alexmiller I am able to run the tests now on Intel using ctest, my code was missing this change https://github.com/apple/foundationdb/commit/20c3f7f26427dde45bb9d43b21ffff1bc91a2055
causing all tests to fail earlier.
The command
foundationdb/build/bin/fdbserver -r simulation -S random -b on --trace_format xml -s 1761517622 -f /root/foundationdb/tests/fast/AtomicBackupCorrectness.txt
was failing internally with the below error
/cmake-3.12.4/Source/CTest/cmCTestRunTest.cxx:45 1: ERROR: Could not parse object serializer option: `random
Correct that and will try the tests on Power now
I found and made some data type conversion changes that were required. Then I had successfully built FDB.
The test results of ctest command on power with above changes is 55% tests passed, 46 tests failed out of 102.
Following are the most common error that I have found with filter āSeverity=ā40āā from the test failure logs.
Event Severity=ā40ā Time=ā94.846676ā Type=āInternalErrorā Machine=ā2.0.1.2:1ā ID=ā0000000000000000ā Error=āinternal_errorā ErrorDescription=āAn internal error occurredā ErrorCode=ā4100ā File="/root/foundationdb/fdbserver/SkipList.cpp" Line=ā1130ā Backtrace=āaddr2line -e fdbserver.debug -p -C -f -i 0x29c0498 0x29bf738 0x2939168 0xfd8660 0xfd9af4 0xf7add4 0xf7c9d8 0xf7cf60 0xf7d40c 0xf7dd4c 0xf814a4 0x694c88 0x27aa720 0x27aaac0 0x79255c 0x283d7a4 0x2881958 0x2881d68 0x28819cc 0x2882418 0x79255c 0x296cc30 0x2882040 0x460720 0x3fff55da441c 0x3fff55da4618ā LogGroup=ādefaultā Roles=āLR,RV,SS,TLā /
Event Severity=ā40ā Time=ā94.846676ā Type=āSystemErrorā Machine=ā2.0.1.2:1ā ID=ā0000000000000000ā Error=āinternal_errorā ErrorDescription=āAn internal error occurredā ErrorCode=ā4100ā Backtrace=āaddr2line -e fdbserver.debug -p -C -f -i 0x29c0498 0x29bf738 0x2932d64 0x2939180 0xfd8660 0xfd9af4 0xf7add4 0xf7c9d8 0xf7cf60 0xf7d40c 0xf7dd4c 0xf814a4 0x694c88 0x27aa720 0x27aaac0 0x79255c 0x283d7a4 0x2881958 0x2881d68 0x28819cc 0x2882418 0x79255c 0x296cc30 0x2882040 0x460720 0x3fff55da441c 0x3fff55da4618ā LogGroup=ādefaultā Roles=āLR,RV,SS,TLā /
I am currently looking into above issues and any pointer related to above errors would be really helpful.
Thanks!!
You forgot to paste your Severity 40 events, but a ~50% failure rate makes it sound like you didnāt port the stackful coroutine support to power, and thus every ssd test is failing?
@alexmiller Sorry for the mistake in previous post. I had pasted events but due to special characters, they were not visible. I have removed those and now events are visible in that post.
Can you run the provided addr2line commands and give the top 10 stack frames from each?
addr2line -e fdbserver -p -C -f -i 0x29c0498 0x29bf738 0x2939168 0xfd8660 0xfd9af4 0xf7add4 0xf7c9d8 0xf7cf60 0xf7d40c 0xf7dd4c 0xf814a4 0x694c88 0x27aa720 0x27aaac0 0x79255c 0x283d7a4 0x2881958 0x2881d68 0x28819cc 0x2882418 0x79255c 0x296cc30 0x2882040 0x460720 0x3fff55da441c 0x3fff55da4618
TraceEvent::backtrace(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) at /root/foundationdb/flow/Trace.cpp:938
TraceEvent::~TraceEvent() at /usr/include/c++/7/bits/basic_string.h:211
(inlined by) ?? at /usr/include/c++/7/bits/basic_string.h:220
(inlined by) ?? at /usr/include/c++/7/bits/basic_string.h:647
(inlined by) TraceEvent::~TraceEvent() at /root/foundationdb/flow/Trace.cpp:947
internal_error_impl(char const*, int) at /root/foundationdb/flow/Error.cpp:51
ConflictBatch::checkIntraBatchConflicts() at /root/foundationdb/fdbserver/SkipList.cpp:1130
(inlined by) ConflictBatch::checkIntraBatchConflicts() at /root/foundationdb/fdbserver/SkipList.cpp:1157
ConflictBatch::detectConflicts(long, long, std::vector<int, std::allocator >&, std::vector<int, std::allocator >*) at /root/foundationdb/fdbserver/SkipList.cpp:1191
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont5(int) [clone .isra.1527] at /root/foundationdb/fdbserver/Resolver.actor.cpp:152
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1break1(int) [clone .isra.1532] at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:257
(inlined by) ?? at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:302
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:320
(inlined by) a_body1cont1loopBody1 at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:284
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:265
(inlined by) ?? at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:147
addr2line -e fdbserver -p -C -f -i 0x29c0498 0x29bf738 0x2932d64 0x2939180 0xfd8660 0xfd9af4 0xf7add4 0xf7c9d8 0xf7cf60 0xf7d40c 0xf7dd4c 0xf814a4 0x694c88 0x27aa720 0x27aaac0 0x79255c 0x283d7a4 0x2881958 0x2881d68 0x28819cc 0x2882418 0x79255c 0x296cc30 0x2882040 0x460720 0x3fff55da441c 0x3fff55da4618
TraceEvent::backtrace(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) at /root/foundationdb/flow/Trace.cpp:938
TraceEvent::~TraceEvent() at /usr/include/c++/7/bits/basic_string.h:211
(inlined by) ?? at /usr/include/c++/7/bits/basic_string.h:220
(inlined by) ?? at /usr/include/c++/7/bits/basic_string.h:647
(inlined by) TraceEvent::~TraceEvent() at /root/foundationdb/flow/Trace.cpp:947
Error::Error(int) at /root/foundationdb/flow/Error.cpp:66
internal_error_impl(char const*, int) at /root/foundationdb/flow/Error.cpp:58
ConflictBatch::checkIntraBatchConflicts() at /root/foundationdb/fdbserver/SkipList.cpp:1130
(inlined by) ConflictBatch::checkIntraBatchConflicts() at /root/foundationdb/fdbserver/SkipList.cpp:1157
ConflictBatch::detectConflicts(long, long, std::vector<int, std::allocator >&, std::vector<int, std::allocator >*) at /root/foundationdb/fdbserver/SkipList.cpp:1191
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont5(int) [clone .isra.1527] at /root/foundationdb/fdbserver/Resolver.actor.cpp:152
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1break1(int) [clone .isra.1532] at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:257
(inlined by) ?? at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:302
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:320
(inlined by) a_body1cont1loopBody1 at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:284
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:265
(inlined by) ?? at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:147
(inlined by) ?? at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:183
(inlined by) a_body1loopBody1 at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:164
resolveBatch(Reference<(anonymous namespace)::Resolver> const&, ResolveTransactionBatchRequest const&) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:154
Well, at least itās consistent, but my initial guess was wrong.
You might want to run cmake -DCMAKE_BUILD_TYPE=Debug and then recompile to get an -O0 build which should have more accurate backtraces to double check, but Iām guessing youāve managed to hit this assert. Itās calling a lot of code that does some moderately shady bitmath, and then checks it against the correct answer, so Iām going to guess that the shady bitmath now doesnāt do the right thing on ppc.
I have triggered build with debug flag and currently looking into possible char related issues (char is unsigned by default on ppc64le) as it may be one of the possible cause for failure.
@alexmiller To build and execute test setup of FoundationDB on ppc64le, i have made the changes mentioned at https://github.com/ppc64le/build-scripts/blob/master/FoundationDB/Initial_Patch_For_ppc64le.patch.
@alexmiller I had built binary with cmake -DCMAKE_BUILD_TYPE=Debug and got following.
addr2line -e build/bin/fdbserver -p -C -f -i 0x3a24ccc 0x39762b8 0x179e628 0x1790eb4 0x179129c 0x1715878 0x17151c0 0x1714df4 0x1714c84 0x171491c 0x17143f8 0x1713d5c 0x17128fc 0x171074c 0x170ea68 0x170becc 0x1709264 0x1709418 0x1712ad4 0x171ae34 0x171943c 0x872010 0x85961c 0x36dfa90 0x36e1e04 0x36df7f8 0x36eb7e0 0x36e9480 0x50da90 0x707910 0x37f3838 0x383f508 0x38360e8 0x3829f04 0x3818984 0x383f534 0x38360e8 0x3855fd4 0x3851944 0x50da90 0x707910 0x39e0de0 0x39ca4c4 0x3808318 0x37e6c58 0x37e6d9c 0x37e6e4c 0xf63214 0x3fff4c22441c
platform::get_backtraceabi:cxx11 at /root/foundationdb/flow/Platform.cpp:2626
internal_error_impl(char const*, int) at /root/foundationdb/flow/Error.cpp:49
MiniConflictSet::any(int, int) at /root/foundationdb/fdbserver/SkipList.cpp:1130 (discriminator 7)
ConflictBatch::checkIntraBatchConflicts() at /root/foundationdb/fdbserver/SkipList.cpp:1157
ConflictBatch::detectConflicts(long, long, std::vector<int, std::allocator >&, std::vector<int, std::allocator >) at /root/foundationdb/fdbserver/SkipList.cpp:1191
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont5(int) at /root/foundationdb/fdbserver/Resolver.actor.cpp:152
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont3(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:257
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1break1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:302
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopBody1when1(Void const&, int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:320 (discriminator 4)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:284 (discriminator 11)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopHead1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:265 (discriminator 1)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:147
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1break1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:183
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:164 (discriminator 4)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1loopHead1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:154 (discriminator 1)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:116
(anonymous namespace)::ResolveBatchActor::ResolveBatchActor(Reference<(anonymous namespace)::Resolver> const&, ResolveTransactionBatchRequest const&) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:720
resolveBatch(Reference<(anonymous namespace)::Resolver> const&, ResolveTransactionBatchRequest const&) at /root/foundationdb/fdbserver/Resolver.actor.cpp:71 (discriminator 2)
(anonymous namespace)::ResolverCoreActorState<(anonymous namespace)::ResolverCoreActor>::a_body1loopBody1when1(ResolveTransactionBatchRequest const&, int) at /root/foundationdb/fdbserver/Resolver.actor.cpp:273
(anonymous namespace)::ResolverCoreActorState<(anonymous namespace)::ResolverCoreActor>::a_callback_fire(ActorSingleCallback<(anonymous namespace)::ResolverCoreActor, 0, ResolveTransactionBatchRequest>, ResolveTransactionBatchRequest) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:922
ActorSingleCallback<(anonymous namespace)::ResolverCoreActor, 0, ResolveTransactionBatchRequest>::fire(ResolveTransactionBatchRequest const&) at /root/foundationdb/flow/flow.h:904 (discriminator 2)
addr2line -e build/bin/fdbserver -p -C -f -i 0x3a24ccc 0x39762b8 0x179e628 0x1790eb4 0x179129c 0x1715878 0x17151c0 0x1714df4 0x1714c84 0x171491c 0x17143f8 0x1713d5c 0x17128fc 0x171074c 0x170ea68 0x170becc 0x1709264 0x1709418 0x1712ad4 0x171ae34 0x171943c 0x872010 0x85961c 0x36dfa90 0x36e1e04 0x36df7f8 0x36eb7e0 0x36e9480 0x50da90 0x707910 0x37f3838 0x383f508 0x38360e8 0x3855fd4 0x3851944 0x50da90 0x707910 0x39e0de0 0x39ca4c4 0x3808318 0x37e6c58 0x37e6d9c 0x37e6e4c 0xf63214 0x3fff4c22441c 0x3fff4c224618
platform::get_backtraceabi:cxx11 at /root/foundationdb/flow/Platform.cpp:2626
internal_error_impl(char const*, int) at /root/foundationdb/flow/Error.cpp:49
MiniConflictSet::any(int, int) at /root/foundationdb/fdbserver/SkipList.cpp:1130 (discriminator 7)
ConflictBatch::checkIntraBatchConflicts() at /root/foundationdb/fdbserver/SkipList.cpp:1157
ConflictBatch::detectConflicts(long, long, std::vector<int, std::allocator >&, std::vector<int, std::allocator >) at /root/foundationdb/fdbserver/SkipList.cpp:1191
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont5(int) at /root/foundationdb/fdbserver/Resolver.actor.cpp:152
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont3(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:257
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1break1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:302
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopBody1when1(Void const&, int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:320 (discriminator 4)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:284 (discriminator 11)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopHead1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:265 (discriminator 1)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:147
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1break1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:183
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:164 (discriminator 4)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1loopHead1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:154 (discriminator 1)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:116
(anonymous namespace)::ResolveBatchActor::ResolveBatchActor(Reference<(anonymous namespace)::Resolver> const&, ResolveTransactionBatchRequest const&) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:720
resolveBatch(Reference<(anonymous namespace)::Resolver> const&, ResolveTransactionBatchRequest const&) at /root/foundationdb/fdbserver/Resolver.actor.cpp:71 (discriminator 2)
(anonymous namespace)::ResolverCoreActorState<(anonymous namespace)::ResolverCoreActor>::a_body1loopBody1when1(ResolveTransactionBatchRequest const&, int) at /root/foundationdb/fdbserver/Resolver.actor.cpp:273
(anonymous namespace)::ResolverCoreActorState<(anonymous namespace)::ResolverCoreActor>::a_callback_fire(ActorSingleCallback<(anonymous namespace)::ResolverCoreActor, 0, ResolveTransactionBatchRequest>, ResolveTransactionBatchRequest) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:922
ActorSingleCallback<(anonymous namespace)::ResolverCoreActor, 0, ResolveTransactionBatchRequest>::fire(ResolveTransactionBatchRequest const&) at /root/foundationdb/flow/flow.h:904 (discriminator 2)
void NotifiedQueue::send(ResolveTransactionBatchRequest&&) at /root/foundationdb/flow/flow.h:493
NetNotifiedQueue::receive(ArenaReader&) at /root/foundationdb/fdbrpc/fdbrpc.h:246
(anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_body1cont1(int) at /root/foundationdb/fdbrpc/FlowTransport.actor.cpp:663
(anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_body1cont2(Void const&, int) at /root/foundationdb/build/fdbrpc/FlowTransport.actor.g.cpp:2153
(anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_body1when1(Void const&, int) at /root/foundationdb/build/fdbrpc/FlowTransport.actor.g.cpp:2159
(anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_callback_fire(ActorCallback<(anonymous namespace)::DeliverActor, 0, Void>*, Void) at /root/foundationdb/build/fdbrpc/FlowTransport.actor.g.cpp:2181
ActorCallback<(anonymous namespace)::DeliverActor, 0, Void>::fire(Void const&) at /root/foundationdb/flow/flow.h:895
void SAV::send(Void&&) at /root/foundationdb/flow/flow.h:343
void Promise::send(Void&&) const at /root/foundationdb/flow/flow.h:674
I am still trying to identify the point of failure, but till time not able to identify.
If you get time to look into it would be really helpful.