Porting FoundationDB to Power (ppc64le)

seth-priya · February 5, 2019, 10:43am

Hi All,

I am starting to look at porting FoundationDB code to work on Power (ppc64le) and seems that there is a lot of architecture specific code (dependency on x86_64).

Just wanted to check if there is any general guidance on how to approach this, how complex this might be and if it has been ported (or a port attempted) to any other architecture, which can be used as a reference?

Thanks!

Priya

josephg · February 5, 2019, 1:18pm

I’m curious about this too given the way things are going Apple will probably replace their laptops’ intel chips with ARM chips soon. And there’s some really nice cheap arm chips on AWS now.

How much asm is there in foundationdb?

dave · February 5, 2019, 2:01pm

As far as I know you will be the first to try porting to a different CPU architecture.

Obvious architecture dependent stuff:

the binary compatibility version dispatching in the client library (bindings/c/generate_asm.py)
the coroutine library (fdbrpc/libcoroutine/), which already has code for several architectures but we haven’t tested it or kept it up to date
use of rdtsc as a timing source
probably various intrinsics scattered around
probably lots of stuff that assumes little endian (but you say you are le)
probably some stuff that assumes unaligned memory access is possible and reasonably performant

I’m sure there is more.

alexmiller · February 5, 2019, 5:37pm

Care to share why, or what your use case is?

I think @wolfspider has some amount of an ARM port done that he’s posted about before.

kocolosk · February 8, 2019, 5:50pm

Heya @alexmiller I can provide some color. We’ve a number of ideas for use of FoundationDB in IBM. Many of those use cases are motivated by the needs of services delivered in IBM Cloud. Those services often have an on premises counterpart with a consistent API to help our clients build portable applications, and many of our clients run their on prem “private cloud” environment on Power hardware. So, we’re looking ahead to make sure that we don’t find ourselves scrambling to port something here. I can think of at least two customers where FoundationDB and Power could intersect in the coming year.

Priya’s team has the broader mission to support a robust open source ecosystem on Power, and I’ve asked for their help to figure out what we might be up against here

dave · February 8, 2019, 7:14pm

If you want to build and maintain a port of FDB to a new architecture, I would recommend budgeting for ongoing compute and human time to run a full spectrum of simulation and other tests on each release. Otherwise the port may be significantly less trustworthy than the official releases.

aseipp · February 14, 2019, 7:46pm

It would be nice if there was a document somewhere describing a “baseline” testing routine for FoundationDB porters, etc. I don’t exactly know what integration strategies Apple uses, for example. (For instance, NixOS users can’t use foundationdb.org binaries, so we have to package our own). I ran many of the simulation tests under tests/ with our builds – but only after figuring out even how to do that by reading forum threads, etc. It would be nice to have a document saying something like, “Here are the tests you need to run manually, and here are tests that can be fully automated by running fdbserver in simulation mode”, and how to do so.

It would also be nice to know about the memory/compute requirements for this. For instance, I can write a test for NixOS that can execute a set of test routes running FoundationDB, including things like simulation runs; this is how we integrate and test other databases, such as CockroachDB, too (this link contains a full end-to-end cluster test). While this might not be enabled by default (it could take hours), if the requirements are known, I can certainly run tests myself regularly, etc.

Perhaps this already exists somewhere in the source code? I still think this would be good information to have in the documentation so it’s easier to find.

On a related note, it would probably be nice on top of this if the GitHub repository for FoundationDB could be set up to use a CI system that ran such tests somehow. This is not only very useful for you but also important for contributors, etc. (Right now I would simply have no clue how to test my changes before submitting them for review). I’d guess that a public CI system for GitHub would probably run a smaller set of tests to keep iteration times lower.

(A useful CI system to examine might be Azure Pipelines, since it’s got a free version and does actually support Windows/Linux/Mac natively, and it’s the only one I can think of that does support all 3 out of the box.)

alexmiller · February 14, 2019, 8:31pm

I don’t see a PowerPC port for libcoroutine that anyone has already done. FWIW, I wouldn’t be opposed to a PR switching from coroutine to Boost.Context instead.

I think this is going to, somewhat by necessity, fall out of the “port FDB to FreeBSD” PR, because part of that involves a discussion of “when would this be marked as stable” and the list of requirements that would entail.

Porting to a new architecture but same platform would likely involve a separate list, as trying to verify that the networking stack or file system doesn’t behave detrimentally differently wouldn’t be needed. I’m happy to sit down and think hard and chat with folks if someone shows up with an FDB port to a new arch, like PowerPC.

There’s a CMake build system that’s been added, that also has CTest support. I’d recommend using that, because a lot of work was put into getting ctest able to run simulation tests locally in parallel. If ctest passes for you locally, that’s a reasonable time to upload a PR. That still only gives you O(100) runs of simulation versus the O(50,000) runs that I’d generally aim for if it’s a serious change to the core, but if there’s any concern, your reviewer will pull the PR and kick off more simulation runs in a cluster of machines to close that testing gap.

I… don’t see that in the README, which is a pretty good point.

It currently can! That that support was added a week ago or something, though. (The Correctness check on this PR, for example.) It runs ctest -L fast, and there’s commands for reviewers to issue to get it to do a more comprehensive testing if requested.

seth-priya · May 23, 2019, 10:00am

Thanks all for your inputs!!

Sorry for delay in getting back, I have been following the discussion here, but have just about resumed active work on the port and am able to make some progress in getting an initial build going.

Further to initial inputs from @dave, changes are mostly for
•use of rdtsc as a timing source : introduced appropriate code / implementation for Power.
•probably various intrinsics scattered around : Changes were mostly need for putting in appropriate changes in place of _mm_pause and _mm_prefetch

For the first build, I have by-passed
•crc implementation
•the binary compatibility version dispatching in the client library (bindings/c/generate_asm.py)

The build is reaching the 100% success mark and following binaries / libraries are generated
fdbbackup fdbcli fdbmonitor fdbserver
libfdb_sqlite.a libfdbclient.a libfdbrpc.a libflow.a libthirdparty.a

I have started working on the CRC changes now.

On the testing front, I tried to run a few tests, as described here From Build to Running Tests
and they seemed to pass, however wanted to understand if there is an easy way to do a complete test setup and execute all the available tests in one go?

Thanks in advance!

Regards,
Priya

alexmiller · May 24, 2019, 3:06am

If you set up cmake, with instructions in the README, then running ctest in your build directory will give you the automated test setup and execution of all tests that you’re looking for. If ctest passes for you locally, then that’s a good point at which to post a PR with reasonable confidence.

seth-priya · May 24, 2019, 12:45pm

Thanks @alexmiller I am able to run the tests now on Intel using ctest, my code was missing this change https://github.com/apple/foundationdb/commit/20c3f7f26427dde45bb9d43b21ffff1bc91a2055
causing all tests to fail earlier.

The command
foundationdb/build/bin/fdbserver -r simulation -S random -b on --trace_format xml -s 1761517622 -f /root/foundationdb/tests/fast/AtomicBackupCorrectness.txt

was failing internally with the below error
/cmake-3.12.4/Source/CTest/cmCTestRunTest.cxx:45 1: ERROR: Could not parse object serializer option: `random

Correct that and will try the tests on Power now

KPatel · August 6, 2019, 2:28pm

I found and made some data type conversion changes that were required. Then I had successfully built FDB.

The test results of ctest command on power with above changes is 55% tests passed, 46 tests failed out of 102.

Following are the most common error that I have found with filter ‘Severity=“40”’ from the test failure logs.

Event Severity=“40” Time=“94.846676” Type=“InternalError” Machine=“2.0.1.2:1” ID=“0000000000000000” Error=“internal_error” ErrorDescription=“An internal error occurred” ErrorCode=“4100” File="/root/foundationdb/fdbserver/SkipList.cpp" Line=“1130” Backtrace=“addr2line -e fdbserver.debug -p -C -f -i 0x29c0498 0x29bf738 0x2939168 0xfd8660 0xfd9af4 0xf7add4 0xf7c9d8 0xf7cf60 0xf7d40c 0xf7dd4c 0xf814a4 0x694c88 0x27aa720 0x27aaac0 0x79255c 0x283d7a4 0x2881958 0x2881d68 0x28819cc 0x2882418 0x79255c 0x296cc30 0x2882040 0x460720 0x3fff55da441c 0x3fff55da4618” LogGroup=“default” Roles=“LR,RV,SS,TL” /

Event Severity=“40” Time=“94.846676” Type=“SystemError” Machine=“2.0.1.2:1” ID=“0000000000000000” Error=“internal_error” ErrorDescription=“An internal error occurred” ErrorCode=“4100” Backtrace=“addr2line -e fdbserver.debug -p -C -f -i 0x29c0498 0x29bf738 0x2932d64 0x2939180 0xfd8660 0xfd9af4 0xf7add4 0xf7c9d8 0xf7cf60 0xf7d40c 0xf7dd4c 0xf814a4 0x694c88 0x27aa720 0x27aaac0 0x79255c 0x283d7a4 0x2881958 0x2881d68 0x28819cc 0x2882418 0x79255c 0x296cc30 0x2882040 0x460720 0x3fff55da441c 0x3fff55da4618” LogGroup=“default” Roles=“LR,RV,SS,TL” /

I am currently looking into above issues and any pointer related to above errors would be really helpful.

Thanks!!

alexmiller · August 6, 2019, 10:14pm

You forgot to paste your Severity 40 events, but a ~50% failure rate makes it sound like you didn’t port the stackful coroutine support to power, and thus every ssd test is failing?

KPatel · August 7, 2019, 6:17am

@alexmiller Sorry for the mistake in previous post. I had pasted events but due to special characters, they were not visible. I have removed those and now events are visible in that post.

alexmiller · August 7, 2019, 6:19am

Can you run the provided addr2line commands and give the top 10 stack frames from each?

KPatel · August 7, 2019, 6:51am

addr2line -e fdbserver -p -C -f -i 0x29c0498 0x29bf738 0x2939168 0xfd8660 0xfd9af4 0xf7add4 0xf7c9d8 0xf7cf60 0xf7d40c 0xf7dd4c 0xf814a4 0x694c88 0x27aa720 0x27aaac0 0x79255c 0x283d7a4 0x2881958 0x2881d68 0x28819cc 0x2882418 0x79255c 0x296cc30 0x2882040 0x460720 0x3fff55da441c 0x3fff55da4618

TraceEvent::backtrace(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) at /root/foundationdb/flow/Trace.cpp:938
TraceEvent::~TraceEvent() at /usr/include/c++/7/bits/basic_string.h:211
(inlined by) ?? at /usr/include/c++/7/bits/basic_string.h:220
(inlined by) ?? at /usr/include/c++/7/bits/basic_string.h:647
(inlined by) TraceEvent::~TraceEvent() at /root/foundationdb/flow/Trace.cpp:947
internal_error_impl(char const*, int) at /root/foundationdb/flow/Error.cpp:51
ConflictBatch::checkIntraBatchConflicts() at /root/foundationdb/fdbserver/SkipList.cpp:1130
(inlined by) ConflictBatch::checkIntraBatchConflicts() at /root/foundationdb/fdbserver/SkipList.cpp:1157
ConflictBatch::detectConflicts(long, long, std::vector<int, std::allocator >&, std::vector<int, std::allocator >*) at /root/foundationdb/fdbserver/SkipList.cpp:1191
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont5(int) [clone .isra.1527] at /root/foundationdb/fdbserver/Resolver.actor.cpp:152
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1break1(int) [clone .isra.1532] at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:257
(inlined by) ?? at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:302
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:320
(inlined by) a_body1cont1loopBody1 at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:284
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:265
(inlined by) ?? at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:147

addr2line -e fdbserver -p -C -f -i 0x29c0498 0x29bf738 0x2932d64 0x2939180 0xfd8660 0xfd9af4 0xf7add4 0xf7c9d8 0xf7cf60 0xf7d40c 0xf7dd4c 0xf814a4 0x694c88 0x27aa720 0x27aaac0 0x79255c 0x283d7a4 0x2881958 0x2881d68 0x28819cc 0x2882418 0x79255c 0x296cc30 0x2882040 0x460720 0x3fff55da441c 0x3fff55da4618

TraceEvent::backtrace(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) at /root/foundationdb/flow/Trace.cpp:938
TraceEvent::~TraceEvent() at /usr/include/c++/7/bits/basic_string.h:211
(inlined by) ?? at /usr/include/c++/7/bits/basic_string.h:220
(inlined by) ?? at /usr/include/c++/7/bits/basic_string.h:647
(inlined by) TraceEvent::~TraceEvent() at /root/foundationdb/flow/Trace.cpp:947
Error::Error(int) at /root/foundationdb/flow/Error.cpp:66
internal_error_impl(char const*, int) at /root/foundationdb/flow/Error.cpp:58
ConflictBatch::checkIntraBatchConflicts() at /root/foundationdb/fdbserver/SkipList.cpp:1130
(inlined by) ConflictBatch::checkIntraBatchConflicts() at /root/foundationdb/fdbserver/SkipList.cpp:1157
ConflictBatch::detectConflicts(long, long, std::vector<int, std::allocator >&, std::vector<int, std::allocator >*) at /root/foundationdb/fdbserver/SkipList.cpp:1191
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont5(int) [clone .isra.1527] at /root/foundationdb/fdbserver/Resolver.actor.cpp:152
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1break1(int) [clone .isra.1532] at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:257
(inlined by) ?? at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:302
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:320
(inlined by) a_body1cont1loopBody1 at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:284
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:265
(inlined by) ?? at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:147
(inlined by) ?? at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:183
(inlined by) a_body1loopBody1 at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:164
resolveBatch(Reference<(anonymous namespace)::Resolver> const&, ResolveTransactionBatchRequest const&) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:154

alexmiller · August 7, 2019, 7:23am

Well, at least it’s consistent, but my initial guess was wrong.

You might want to run cmake -DCMAKE_BUILD_TYPE=Debug and then recompile to get an -O0 build which should have more accurate backtraces to double check, but I’m guessing you’ve managed to hit this assert. It’s calling a lot of code that does some moderately shady bitmath, and then checks it against the correct answer, so I’m going to guess that the shady bitmath now doesn’t do the right thing on ppc.

KPatel · August 7, 2019, 8:50am

I have triggered build with debug flag and currently looking into possible char related issues (char is unsigned by default on ppc64le) as it may be one of the possible cause for failure.

KPatel · August 12, 2019, 12:06pm

@alexmiller To build and execute test setup of FoundationDB on ppc64le, i have made the changes mentioned at https://github.com/ppc64le/build-scripts/blob/master/FoundationDB/Initial_Patch_For_ppc64le.patch.

KPatel · August 28, 2019, 6:33am

@alexmiller I had built binary with cmake -DCMAKE_BUILD_TYPE=Debug and got following.

addr2line -e build/bin/fdbserver -p -C -f -i 0x3a24ccc 0x39762b8 0x179e628 0x1790eb4 0x179129c 0x1715878 0x17151c0 0x1714df4 0x1714c84 0x171491c 0x17143f8 0x1713d5c 0x17128fc 0x171074c 0x170ea68 0x170becc 0x1709264 0x1709418 0x1712ad4 0x171ae34 0x171943c 0x872010 0x85961c 0x36dfa90 0x36e1e04 0x36df7f8 0x36eb7e0 0x36e9480 0x50da90 0x707910 0x37f3838 0x383f508 0x38360e8 0x3829f04 0x3818984 0x383f534 0x38360e8 0x3855fd4 0x3851944 0x50da90 0x707910 0x39e0de0 0x39ca4c4 0x3808318 0x37e6c58 0x37e6d9c 0x37e6e4c 0xf63214 0x3fff4c22441c

platform::get_backtraceabi:cxx11 at /root/foundationdb/flow/Platform.cpp:2626
internal_error_impl(char const*, int) at /root/foundationdb/flow/Error.cpp:49
MiniConflictSet::any(int, int) at /root/foundationdb/fdbserver/SkipList.cpp:1130 (discriminator 7)
ConflictBatch::checkIntraBatchConflicts() at /root/foundationdb/fdbserver/SkipList.cpp:1157
ConflictBatch::detectConflicts(long, long, std::vector<int, std::allocator >&, std::vector<int, std::allocator >) at /root/foundationdb/fdbserver/SkipList.cpp:1191
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont5(int) at /root/foundationdb/fdbserver/Resolver.actor.cpp:152
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont3(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:257
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1break1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:302
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopBody1when1(Void const&, int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:320 (discriminator 4)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:284 (discriminator 11)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopHead1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:265 (discriminator 1)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:147
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1break1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:183
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:164 (discriminator 4)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1loopHead1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:154 (discriminator 1)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:116
(anonymous namespace)::ResolveBatchActor::ResolveBatchActor(Reference<(anonymous namespace)::Resolver> const&, ResolveTransactionBatchRequest const&) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:720
resolveBatch(Reference<(anonymous namespace)::Resolver> const&, ResolveTransactionBatchRequest const&) at /root/foundationdb/fdbserver/Resolver.actor.cpp:71 (discriminator 2)
(anonymous namespace)::ResolverCoreActorState<(anonymous namespace)::ResolverCoreActor>::a_body1loopBody1when1(ResolveTransactionBatchRequest const&, int) at /root/foundationdb/fdbserver/Resolver.actor.cpp:273
(anonymous namespace)::ResolverCoreActorState<(anonymous namespace)::ResolverCoreActor>::a_callback_fire(ActorSingleCallback<(anonymous namespace)::ResolverCoreActor, 0, ResolveTransactionBatchRequest>, ResolveTransactionBatchRequest) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:922
ActorSingleCallback<(anonymous namespace)::ResolverCoreActor, 0, ResolveTransactionBatchRequest>::fire(ResolveTransactionBatchRequest const&) at /root/foundationdb/flow/flow.h:904 (discriminator 2)

addr2line -e build/bin/fdbserver -p -C -f -i 0x3a24ccc 0x39762b8 0x179e628 0x1790eb4 0x179129c 0x1715878 0x17151c0 0x1714df4 0x1714c84 0x171491c 0x17143f8 0x1713d5c 0x17128fc 0x171074c 0x170ea68 0x170becc 0x1709264 0x1709418 0x1712ad4 0x171ae34 0x171943c 0x872010 0x85961c 0x36dfa90 0x36e1e04 0x36df7f8 0x36eb7e0 0x36e9480 0x50da90 0x707910 0x37f3838 0x383f508 0x38360e8 0x3855fd4 0x3851944 0x50da90 0x707910 0x39e0de0 0x39ca4c4 0x3808318 0x37e6c58 0x37e6d9c 0x37e6e4c 0xf63214 0x3fff4c22441c 0x3fff4c224618

platform::get_backtraceabi:cxx11 at /root/foundationdb/flow/Platform.cpp:2626
internal_error_impl(char const*, int) at /root/foundationdb/flow/Error.cpp:49
MiniConflictSet::any(int, int) at /root/foundationdb/fdbserver/SkipList.cpp:1130 (discriminator 7)
ConflictBatch::checkIntraBatchConflicts() at /root/foundationdb/fdbserver/SkipList.cpp:1157
ConflictBatch::detectConflicts(long, long, std::vector<int, std::allocator >&, std::vector<int, std::allocator >) at /root/foundationdb/fdbserver/SkipList.cpp:1191
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont5(int) at /root/foundationdb/fdbserver/Resolver.actor.cpp:152
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont3(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:257
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1break1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:302
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopBody1when1(Void const&, int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:320 (discriminator 4)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:284 (discriminator 11)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1loopHead1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:265 (discriminator 1)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1cont1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:147
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1break1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:183
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1loopBody1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:164 (discriminator 4)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1loopHead1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:154 (discriminator 1)
(anonymous namespace)::ResolveBatchActorState<(anonymous namespace)::ResolveBatchActor>::a_body1(int) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:116
(anonymous namespace)::ResolveBatchActor::ResolveBatchActor(Reference<(anonymous namespace)::Resolver> const&, ResolveTransactionBatchRequest const&) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:720
resolveBatch(Reference<(anonymous namespace)::Resolver> const&, ResolveTransactionBatchRequest const&) at /root/foundationdb/fdbserver/Resolver.actor.cpp:71 (discriminator 2)
(anonymous namespace)::ResolverCoreActorState<(anonymous namespace)::ResolverCoreActor>::a_body1loopBody1when1(ResolveTransactionBatchRequest const&, int) at /root/foundationdb/fdbserver/Resolver.actor.cpp:273
(anonymous namespace)::ResolverCoreActorState<(anonymous namespace)::ResolverCoreActor>::a_callback_fire(ActorSingleCallback<(anonymous namespace)::ResolverCoreActor, 0, ResolveTransactionBatchRequest>, ResolveTransactionBatchRequest) at /root/foundationdb/build/fdbserver/Resolver.actor.g.cpp:922
ActorSingleCallback<(anonymous namespace)::ResolverCoreActor, 0, ResolveTransactionBatchRequest>::fire(ResolveTransactionBatchRequest const&) at /root/foundationdb/flow/flow.h:904 (discriminator 2)
void NotifiedQueue::send(ResolveTransactionBatchRequest&&) at /root/foundationdb/flow/flow.h:493
NetNotifiedQueue::receive(ArenaReader&) at /root/foundationdb/fdbrpc/fdbrpc.h:246
(anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_body1cont1(int) at /root/foundationdb/fdbrpc/FlowTransport.actor.cpp:663
(anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_body1cont2(Void const&, int) at /root/foundationdb/build/fdbrpc/FlowTransport.actor.g.cpp:2153
(anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_body1when1(Void const&, int) at /root/foundationdb/build/fdbrpc/FlowTransport.actor.g.cpp:2159
(anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_callback_fire(ActorCallback<(anonymous namespace)::DeliverActor, 0, Void>*, Void) at /root/foundationdb/build/fdbrpc/FlowTransport.actor.g.cpp:2181
ActorCallback<(anonymous namespace)::DeliverActor, 0, Void>::fire(Void const&) at /root/foundationdb/flow/flow.h:895
void SAV::send(Void&&) at /root/foundationdb/flow/flow.h:343
void Promise::send(Void&&) const at /root/foundationdb/flow/flow.h:674

I am still trying to identify the point of failure, but till time not able to identify.
If you get time to look into it would be really helpful.