Memory leak when use C API normaly

Using FoundationDB
1

Environment:

  1. fdb-version 7.1.9
  2. os 5.15.0-50-generic #56~20.04.1-Ubuntu
  3. fdb configuration : one single fdbserver on local machine with below status
Configuration:
  Redundancy mode        - single
  Storage engine         - ssd-2
  Coordinators           - 1
  Usable Regions         - 1

Cluster:
  FoundationDB processes - 1
  Zones                  - 1
  Machines               - 1
  Memory availability    - 32.0 GB per process on machine with least available
  Fault Tolerance        - 0 machines
  Server time            - 10/13/22 16:34:23

Data:
  Replication health     - Healthy
  Moving data            - 0.000 GB
  Sum of key-value sizes - 181 MB
  Disk space used        - 4.919 GB

Operating space:
  Storage server         - 24.1 GB free on most full server
  Log server             - 1510.8 GB free on most full server

Workload:
  Read rate              - 17 Hz
  Write rate             - 0 Hz
  Transactions started   - 5 Hz
  Transactions committed - 0 Hz
  Conflict rate          - 0 Hz

Backup and DR:
  Running backups        - 0
  Running DRs            - 0

Process performance details:
  127.0.0.1:4500         (  1% cpu;  6% machine; 0.000 Gbps;  3% disk IO; 2.0 GB / 32.0 GB RAM  )

Coordination servers:
  127.0.0.1:4500  (reachable)

Problem:

When i running our application on this fdbserver with C API for one hour (maybe more time), the heap memory is continuing growing and i catch whole process with valgrind + massif on my application, memory allocate stack is like this :upside_down_face:

│ 14713 --------------------------------------------------------------------------------
│ 14714 ▏ n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
│ 14715 --------------------------------------------------------------------------------
│ 14716  93 627,237,678,279      348,705,720      340,253,768     8,451,952            0
│ 14717  94 629,821,773,623      349,505,544      341,041,669     8,463,875            0
│ 14718  95 630,995,566,124      356,079,448      347,602,122     8,477,326            0
│ 14719 97.62% (347,602,122B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
│ 14720 ->32.78% (116,736,000B) 0x536E479: ArenaBlock::create(int, Reference<ArenaBlock>&) (Arena.cpp:359)
│ 14721 | ->32.78% (116,736,000B) 0x536E715: ArenaBlock::allocate(Reference<ArenaBlock>&, int) (Arena.cpp:293)
│ 14722 | | ->32.77% (116,695,040B) 0x52F0581: operator new [] (Arena.h:202)
│ 14723 | | | ->32.77% (116,695,040B) 0x52F0581: a_body1loopBody1loopBody1 (FlowTransport.actor.cpp:1173)
│ 14724 | | |   ->32.77% (116,695,040B) 0x52F0581: a_body1loopBody1loopHead1 (FlowTransport.actor.g.cpp:4716)
│ 14725 | | |     ->32.77% (116,695,040B) 0x52F0581: a_body1loopBody1 (FlowTransport.actor.g.cpp:4693)
│ 14726 | | |       ->32.77% (116,695,040B) 0x52F0581: a_body1loopHead1 (FlowTransport.actor.g.cpp:4684)
│ 14727 | | |         ->32.77% (116,695,040B) 0x52F0581: a_body1loopBody1cont3 (FlowTransport.actor.g.cpp:5431)
│ 14728 | | |           ->32.77% (116,695,040B) 0x52F0581: a_body1loopBody1cont2when1 (FlowTransport.actor.g.cpp:5443)
│ 14729 | | |             ->32.77% (116,695,040B) 0x52F0581: a_body1loopBody1cont2when1 (FlowTransport.actor.g.cpp:5441)
│ 14730 | | |               ->32.77% (116,695,040B) 0x52F0581: a_callback_fire (FlowTransport.actor.g.cpp:5464)
│ 14731 | | |                 ->32.77% (116,695,040B) 0x52F0581: ActorCallback<(anonymous namespace)::ConnectionReaderActor, 4, Void>::fire(Void const&) (flow.h:1318)
│ 14732 | | |                   ->32.77% (116,695,040B) 0x53AFC47: send<Void> (flow.h:660)
│ 14733 | | |                     ->32.77% (116,695,040B) 0x53AFC47: send<Void> (flow.h:906)
│ 14734 | | |                       ->32.77% (116,695,040B) 0x53AFC47: operator() (Net2.actor.cpp:1201)
│ 14735 | | |                         ->32.77% (116,695,040B) 0x53AFC47: N2::Net2::run() (Net2.actor.cpp:1548)
│ 14736 | | |                           ->32.77% (116,695,040B) 0x4BD442A: runNetwork() (NativeAPI.actor.cpp:2538)
│ 14737 | | |                             ->32.77% (116,695,040B) 0x521BE3E: ThreadSafeApi::runNetwork() (ThreadSafeTransaction.cpp:564)
│ 14738 | | |                               ->32.77% (116,695,040B) 0x4B7B6C1: MultiVersionApi::runNetwork() (MultiVersionTransaction.actor.cpp:2295)
│ 14739 | | |                                 ->32.77% (116,695,040B) 0x4B51C3D: fdb_run_network (fdb_c.cpp:161)
│ 14740 | | |                                   ->32.77% (116,695,040B) 0xC055C0: runFdbNetwork (fdb_access.c:154)
│ 14741 | | |                                     ->32.77% (116,695,040B) 0x625A608: start_thread (pthread_create.c:477)
│ 14742 | | |                                       ->32.77% (116,695,040B) 0x76CC132: clone (clone.S:95)

It’s confusing that why a client global network consuming so much memory. Seems this network-thread
is scheduling client-requests normally, but why not releasing the allocated memory.

So any one who could take a simple look?

2

That call site is allocating memory to store incoming network messages to be processed. It’s possible that something could be keeping it alive unexpectedly long. Is it possible that you’re not calling e.g. fdb_transaction_destroy on your transactions? Not calling fdb_future_destroy on futures returned from reads within a transaction could also explain this.

3

Thanks for your reply. You’re right, that’s our problem when using C-API.