FDB out of memory

Using FoundationDB Running FoundationDB
1

Our cluster run on ec2.
Test env:
storage: total 5 processes, one process per node, 2 cores, 16g memory
log: total 3 processes, one process per node, 2 cores, 8g memory
stateless: total 3 nodes, 2 cores, 8g memory
status details shows OOM happend on storage process:

fdb> status details

Using cluster file `/etc/foundationdb/fdb.cluster'.

Unable to retrieve all status information.

Configuration:
  Redundancy mode        - triple
  Storage engine         - ssd-2
  Coordinators           - 5
  Usable Regions         - 1

Cluster:
  FoundationDB processes - 14 (less 0 excluded; 1 with errors)
  Zones                  - 11
  Machines               - 11
  Memory availability    - 4.8 GB per process on machine with least available
  Retransmissions rate   - 0 Hz
  Fault Tolerance        - 1 machines
  Server time            - 07/20/22 11:21:08

Data:
  Replication health     - HEALING: Only two replicas remain of some data
  Moving data            - 9.246 GB
  Sum of key-value sizes - 22.033 GB
  Disk space used        - 81.638 GB

Operating space:
  Storage server         - 62.8 GB free on most full server
  Log server             - 69.8 GB free on most full server

Workload:
  Read rate              - 216 Hz
  Write rate             - 62 Hz
  Transactions started   - 34 Hz
  Transactions committed - 6 Hz
  Conflict rate          - 0 Hz

Backup and DR:
  Running backups        - 0
  Running DRs            - 0

Process performance details:
  10.3.55.183:4500       (  1% cpu;  1% machine; 0.000 Gbps;  3% disk IO; 2.0 GB / 8.0 GB RAM  )
  10.3.55.184:4500       (  2% cpu;  1% machine; 0.000 Gbps;  4% disk IO; 2.1 GB / 8.0 GB RAM  )
  10.3.55.185:4500       ( 46% cpu; 25% machine; 0.112 Gbps; 37% disk IO;11.5 GB / 12.0 GB RAM  )
  10.3.55.186:4500       (  1% cpu; 29% machine; 0.084 Gbps; 19% disk IO; 0.2 GB / 6.6 GB RAM  )
  10.3.55.187:4500       ( 49% cpu; 26% machine; 0.116 Gbps; 26% disk IO;11.8 GB / 12.0 GB RAM  )
  10.3.55.188:4500       ( 43% cpu; 24% machine; 0.063 Gbps; 26% disk IO;11.9 GB / 12.0 GB RAM  )
  10.3.55.189:4500       ( 43% cpu; 23% machine; 0.093 Gbps; 24% disk IO;11.5 GB / 12.0 GB RAM  )
  10.3.55.190:4500       (  1% cpu;  1% machine; 0.000 Gbps;  3% disk IO; 2.1 GB / 8.0 GB RAM  )
  10.3.55.191:4500       (  3% cpu; 18% machine; 0.182 Gbps; 92% disk IO; 0.8 GB / 4.8 GB RAM  )
  10.3.55.192:4500       ( 40% cpu; 23% machine; 0.122 Gbps; 61% disk IO;11.8 GB / 12.0 GB RAM  )
  10.3.55.193:4500       (  2% cpu; 19% machine; 0.074 Gbps; 67% disk IO; 0.1 GB  )

Coordination servers:
  10.3.55.185:4500  (reachable)
  10.3.55.187:4500  (reachable)
  10.3.55.188:4500  (reachable)
  10.3.55.189:4500  (reachable)
  10.3.55.192:4500  (reachable)

Client time: 07/20/22 11:21:08

storage process config:
memory = 12GiB
others config are default

status details shows the memory already used 11.8GB, but free -g command shows below, just 3g memory used, it is weird…

# free -g
               total        used        free      shared  buff/cache   available
Mem:              14           3          10           0           1          11
Swap:              0           0           0

In trace file, I found OOM.

<Event Severity="40" ErrorKind="Unset" Time="1658254220.795019" DateTime="2022-07-19T18:10:20Z" Type="OutOfMemory" ID="0000000000000000" Message="Out of memory" ThreadID="13112316067727058784" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x3277b7c 0x3276b30 0x3276f1b 0x324288c 0x32428bc 0x3579c7c 0xa33e3b 0xa9558b 0xa95b6f 0x309898e 0x30a3e65 0x1a58504 0x19fd285 0x19fe41e 0x19b2860 0x19bd908 0x19bddc5 0x12ef108 0xfebfa0 0x3213852 0x9c2d1a 0x7f5b4e621d90" Machine="10.3.55.185:4500" LogGroup="default" Roles="CD,SS" />

I tried run addr2line -e fdbserver.debug, but there is no fdbserver.debug file…
Also I tried to reboot server, after rebooting server, OOM happend again.

I have three doubts, please help me, thanks.

  1. why is the memory used different between “free -g” and “status details”?
  2. where is the file fdbserver.debug? I install the fdb by foundationdb-server_6.3.24-1_amd64.deb, should I compile fdb with debug mode?
  3. how to locate OOM issue?
2
  1. In 6.3, status json reports process virtual memory usage in “memory::used_bytes”, while free -g is reporting unused memory (MemFree and SwapFree in /proc/meminfo), i.e., physical memory usage. In 7.1, status json reports resident memory usage in “memory::rss_bytes”.
  2. fdbserver.debug file is here: https://github.com/apple/foundationdb/releases/download/6.3.24/fdbserver.debug.x86_64
  3. OOM issue is not obvious, because memory could be allocated fairly quickly. The place triggers the OOM may not be the one shown in the backtrace, but could be a good hint. You need to collect other signals to be sure:
3

Thanks

Is the storage server doing a lot of reads, or writes, or fetching data due to data movement?

It is not too much,
Operations: Read requests 455 ops/s, Reads 455 ops/s, Writes 81.8K ops/s
Transactions: Transactions started 124ops/s, Transactions committed 60.8 ops/s

图片
图片771×226 12.8 KB

图片
图片1215×233 40.3 KB

As I understand, Transactions per Second Limit becomes low means the cluster performance is limited in some way by ratekeeper. The reason maybe is OOM.

MemoryMetrics has fast allocator memory usage by block sizes.

I found MemoryMetrics in trace log, seems like it shows memory increased.

<Event Severity="10" Time="1658322645.043668" DateTime="2022-07-20T13:10:45Z" Type="MemoryMetrics" ID="0000000000000000" 
TotalMemory16="131072" ApproximateUnusedMemory16="0" ActiveThreads16="1" 
TotalMemory32="131072" ApproximateUnusedMemory32="0" ActiveThreads32="1" 
TotalMemory64="41811968" ApproximateUnusedMemory64="1441792" ActiveThreads64="4" 
TotalMemory96="360622080" ApproximateUnusedMemory96="262211040" ActiveThreads96="1" 
TotalMemory128="917504" ApproximateUnusedMemory128="524288" ActiveThreads128="1" 
TotalMemory256="340525056" ApproximateUnusedMemory256="157548544" ActiveThreads256="2" 
TotalMemory512="4325376" ApproximateUnusedMemory512="3407872" ActiveThreads512="1" 
TotalMemory1024="2621440" ApproximateUnusedMemory1024="2359296" ActiveThreads1024="1" 
TotalMemory2048="3932160" ApproximateUnusedMemory2048="3670016" ActiveThreads2048="1" 
TotalMemory4096="7208960" ApproximateUnusedMemory4096="6946816" ActiveThreads4096="1" 
TotalMemory8192="388497408" ApproximateUnusedMemory8192="385351680" ActiveThreads8192="1" 
HugeArenaMemory="14100799" DCID="[not set]" ZoneID="e8fc756089026d7b5db5a940dc0d41dc" 
MachineID="e8fc756089026d7b5db5a940dc0d41dc" ThreadID="17012268515172095333" 
Machine="10.3.55.189:4500" LogGroup="default" Roles="CD,SS" />

Also I tracked HugeArenaSample and GetMagazineSample:
Backtrace of HugeArenaSample

# addr2line -e fdbserver.debug -p -C -f -i 0x1957f87 0x69c0ec 0x18deb12 0x19d8d88 0x663015 0x7fbe95472505
actor_cancelled() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Error.h:94
 (inlined by) ReadWriteWorkload::RandomReadWriteClientActor<Transaction>::cancel() at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/workloads/ReadWrite.actor.g.cpp:4606
 (inlined by) ReadWriteWorkload::RandomReadWriteClientActor<Transaction>::cancel() at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/workloads/ReadWrite.actor.g.cpp:4601
ActorCallback<(anonymous namespace)::TPCC::EmulatedUserActor, 6, Void>::fire(Void const&) [clone .cold.1065] at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/workloads/TPCC.actor.g.cpp:5902
 (inlined by) fire at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:1016
std::_Vector_base<double, std::allocator<double> >::_M_deallocate(double*, unsigned long) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_vector.h:303
 (inlined by) std::_Vector_base<double, std::allocator<double> >::~_Vector_base() at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_vector.h:285
 (inlined by) std::vector<double, std::allocator<double> >::~vector() at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_vector.h:570
 (inlined by) ContinuousSample<double>::~ContinuousSample() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/ContinuousSample.h:32
 (inlined by) QueuePushWorkload::~QueuePushWorkload() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/workloads/QueuePush.actor.cpp:30
Standalone<KeySelectorRef>::operator=(Standalone<KeySelectorRef> const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Arena.h:364
 (inlined by) SerializabilityWorkload::GetKeyOperation::operator=(SerializabilityWorkload::GetKeyOperation&&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/workloads/Serializability.actor.cpp:47
 (inlined by) std::_Optional_payload<SerializabilityWorkload::GetKeyOperation, false, false, false>::operator=(std::_Optional_payload<SerializabilityWorkload::GetKeyOperation, false, false, false>&&) at /opt/rh/devtoolset-8/root/usr/include/c++/8/optional:176
 (inlined by) std::_Optional_payload<SerializabilityWorkload::GetKeyOperation, false, false, false>::operator=(std::_Optional_payload<SerializabilityWorkload::GetKeyOperation, false, false, false>&&) at /opt/rh/devtoolset-8/root/usr/include/c++/8/optional:171
 (inlined by) std::_Optional_base<SerializabilityWorkload::GetKeyOperation, false, false>::operator=(std::_Optional_base<SerializabilityWorkload::GetKeyOperation, false, false>&&) at /opt/rh/devtoolset-8/root/usr/include/c++/8/optional:661
 (inlined by) std::optional<SerializabilityWorkload::GetKeyOperation>::operator=(std::optional<SerializabilityWorkload::GetKeyOperation>&&) at /opt/rh/devtoolset-8/root/usr/include/c++/8/optional:943
 (inlined by) Optional<SerializabilityWorkload::GetKeyOperation>::operator=(Optional<SerializabilityWorkload::GetKeyOperation>&&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Arena.h:205
 (inlined by) SerializabilityWorkload::randomTransaction() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/workloads/Serializability.actor.cpp:142
(anonymous namespace)::NetworkSenderActorState<Standalone<VectorRef<UID, (VecSerStrategy)0> >, (anonymous namespace)::NetworkSenderActor<Standalone<VectorRef<UID, (VecSerStrategy)0> > > >::a_body1cont2(Standalone<VectorRef<UID, (VecSerStrategy)0> > const&, int) [clone .isra.4996] [clone .cold.16458] at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/networksender.actor.g.h:206
 (inlined by) a_body1cont2 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/networksender.actor.g.h:125
?? ??:0

Backtrace of GetMagazineSample:

# addr2line -e fdbserver.debug -p -C -f -i 0x199945c 0x195f588 0x195f7d1 0x18bada4 0x18b2e32 0x18b5bb2 0x711564 0x746640 0x756378 0x7564e6 0x7565db 0x75668c 0xb5badf 0xb5d4fa 0xb5e10b 0xa38856 0xa353e9 0x189da86 0x7fbe95498140
 Callback<DataDistributorInterface>::remove() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:389
 (inlined by) a_exitChoose1 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/networksender.actor.g.h:153
 (inlined by) a_callback_error at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/networksender.actor.g.h:189
 (inlined by) error at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:1017
SAV<Optional<Void> >::sendErrorAndDelPromiseRef(Error) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:515
 (inlined by) a_body1Catch1 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/genericactors.actor.g.h:2738
 (inlined by) a_callback_error at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/genericactors.actor.g.h:2833
 (inlined by) error at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:1017
Callback<Void>::remove() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:390
 (inlined by) a_exitChoose1 at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/genericactors.actor.g.h:2794
 (inlined by) a_callback_error at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/genericactors.actor.g.h:2831
 (inlined by) cancel at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/genericactors.actor.g.h:2922
 (inlined by) cancel at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/genericactors.actor.g.h:2917
 SAV<Optional<Standalone<StringRef> > >::delFutureRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:546
 (inlined by) SAV<Optional<Standalone<StringRef> > >::delFutureRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:543
 (inlined by) Future<Optional<Standalone<StringRef> > >::~Future() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:707
 (inlined by) StrictFuture<Optional<Standalone<StringRef> > >::~StrictFuture() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:769
 (inlined by) PingWorkload::PersistInterfaceActorState<PingWorkload::PersistInterfaceActor>::a_body1loopBody1(int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:769
Standalone<VectorRef<VectorRef<KeyValueRef, (VecSerStrategy)0>, (VecSerStrategy)0> >::Standalone(Standalone<VectorRef<VectorRef<KeyValueRef, (VecSerStrategy)0>, (VecSerStrategy)0> > const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Arena.h:361
 (inlined by) PerformanceWorkload::_setupActorState<PerformanceWorkload::_setupActor>::a_body1(int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/workloads/Performance.actor.cpp:131
ActorCallback<(anonymous namespace)::SuccessActor<LoadedReply>, 0, LoadedReply>::fire(LoadedReply const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:1016
trackBoundedStorageMetrics(Standalone<KeyRangeRef> const&, Reference<MultiInterface<ReferencedInterface<StorageServerInterface> > > const&, StorageMetrics const&, StorageMetrics const&, PromiseStream<StorageMetrics> const&) [clone .cold.12221] at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/FastAlloc.h:203 (discriminator 1)
 (inlined by) trackBoundedStorageMetrics(Standalone<KeyRangeRef> const&, Reference<MultiInterface<ReferencedInterface<StorageServerInterface> > > const&, StorageMetrics const&, StorageMetrics const&, PromiseStream<StorageMetrics> const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/NativeAPI.actor.cpp:4402 (discriminator 1)
ActorCallback<(anonymous namespace)::DoOnMainThreadActor<Optional<Standalone<StringRef> >, ThreadSafeTransaction::get(StringRef const&, bool)::{lambda()#1}>, 0, Void>::fire(Void const&) [clone .cold.1243] at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/ThreadHelper.actor.g.h:731
 (inlined by) fire at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:1016
rocksdb::InternalStats::DumpCFMapStats(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >*) [clone .cold.583] at internal_stats.cc:?
rocksdb::InternalStats::DumpCFStatsNoFileHistogram(std::string*) [clone .cold.584] at internal_stats.cc:?
rocksdb::InternalStats::HandleBlobStats(std::string*, rocksdb::Slice) [clone .cold.590] at internal_stats.cc:?
__static_initialization_and_destruction_0(int, int) [clone .constprop.527] [clone .cold.591] at internal_stats.cc:?
 RangeResultRef::operator=(RangeResultRef const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/FDBTypes.h:656
 (inlined by) Standalone<RangeResultRef>::operator=(Standalone<RangeResultRef> const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Arena.h:364
 (inlined by) a_body1loopBody1when1 at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/DataDistribution.actor.cpp:4833
 (inlined by) a_callback_fire at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/DataDistribution.actor.g.cpp:23441
 (inlined by) fire at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:1016
ActorCallback<(anonymous namespace)::NetworkSenderActor<GetStorageMetricsReply>, 0, GetStorageMetricsReply>::fire(GetStorageMetricsReply const&) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/networksender.actor.g.h:158
 (inlined by) fire at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:1016
ErrorOr<Standalone<VectorRef<DDMetricsRef, (VecSerStrategy)0> > >::get() const at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:193
 (inlined by) a_body1cont1 at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/DataDistribution.actor.cpp:5391
  (anonymous namespace)::MonitorClientTxnInfoConfigsActorState<(anonymous namespace)::MonitorClientTxnInfoConfigsActor>::a_body1loopBody1loopBody1cont4(Void const&, int) [clone .isra.4483] at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/ClusterController.actor.g.cpp:11859
(anonymous namespace)::UpdatedChangedDatacentersActorState<(anonymous namespace)::UpdatedChangedDatacentersActor>::a_body1loopBody1(int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/ClusterController.actor.g.cpp:12788 (discriminator 4)
Deque<DistributorExclusionSafetyCheckRequest>::cleanup() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Deque.h:213
 (inlined by) Deque<DistributorExclusionSafetyCheckRequest>::~Deque() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Deque.h:100
 (inlined by) std::queue<DistributorExclusionSafetyCheckRequest, Deque<DistributorExclusionSafetyCheckRequest> >::~queue() at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_queue.h:96
 (inlined by) NotifiedQueue<DistributorExclusionSafetyCheckRequest>::~NotifiedQueue() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:585
 (inlined by) NetNotifiedQueue<DistributorExclusionSafetyCheckRequest>::~NetNotifiedQueue() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/fdbrpc.h:245
 (inlined by) NetNotifiedQueue<DistributorExclusionSafetyCheckRequest>::destroy() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/fdbrpc.h:253
 (inlined by) NotifiedQueue<DistributorExclusionSafetyCheckRequest>::delPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:639
?? ??:0

After I did the testing again, OOM happened, but after few hours, the memory used still high, seems like the memory doesnot released…

  10.3.55.185:4500       (  1% cpu;  1% machine; 0.000 Gbps;  1% disk IO;12.0 GB / 12.0 GB RAM  )
  10.3.55.187:4500       (  1% cpu;  1% machine; 0.000 Gbps;  1% disk IO;11.8 GB / 12.0 GB RAM  )
4

I see you have installed the 6.3.24 version from the debian package. Did you download the package from the github releases page?

If yes, then you might be in fact running an unstable 7.1 version instead of 6.3.24 as detailed in this issue: Client 6.3.24 points to 7.1 for Debian · Issue #6872 · apple/foundationdb · GitHub Can you double check with ‘fdbcli --verion’?

5

Version is 7.1 :joy:
FoundationDB CLI 7.1 (v7.1.0)
source version 32f2731773010e5c8a6f831af85199c0b8b76dd1
protocol fdb00b071010000

6

I also got burned by this :slight_smile: I suggest you manually build the DEB packages as shown in my post in this thread: Creating our own / custom foundation db builds and retry your benchmark.