Why many clientThreads will cause fdbserver(stateless & grv_proxy) CPU too high?

Using FoundationDB
,
1

In my production environment, I am using FoundationDB version 7.2.5.

We have developed an application which use github.com/apple/foundationdb/bindings/go to operate FoundationDB：

networkOptions := fdb.Options()
err := networkOptions.SetExternalClientLibrary(cfg.FDBLibPath)
if err != nil {
	panic(err)
}
err = networkOptions.SetClientThreadsPerVersion(int64(cfg.NetworkThreadCount))
if err != nil {
	panic(err)
}

we deployed the application on 5 hosts(that is 5 standalone instances) ，and made a Stress test(just simple read transaction)，then we observed the following:

  • the CPU usage of each foundationDB grv_proxy reached 95%

  • the CPU usage of each foundationDB stateless(Transaction log server) 60%

then we increased the grv_proxy from 13 to 30, and we observed the following:

  • the CPU usage of each foundationDB grv_proxy reached 80%

  • the CPU usage of each foundationDB stateless(transaction Log) reached 85%

I have tried many methods but still can’t decrease the CPU usage, but in one day I suddenly found that decrease ‘ClientThreadsPerVersion’ from 10 to 5 can significantly reduce CPU usage from 95% to 58%, so my question is:

  1. Why increase grv_proxy number can not significantly reduce CPU usage of foundationdb grv_proxy server, and has a side effect that cause increase the CPU usage of transaction log server?

  2. Why decrease ‘ClientThreadsPerVersion’ can significantly reduce CPU usage?