Understand FDB read/write with disk IOPS/throughtput/Blocksize

Hi all,
I setup FDB cluster with fdb-kubernetes-operator with 8 instances m6a.2xlarge (8 vcore, 32GB RAM each).
I have 8 storage pods, each pod run 3 storage processes.

apiVersion: apps.foundationdb.org/v1beta2
kind: FoundationDBCluster
metadata:
  labels:
    cluster-group: foundationdb-cluster
  name: foundationdb-cluster
spec:
  version: 7.1.27
  faultDomain:
    key: foundationdb.org/none
  storageServersPerPod: 3
  processCounts:
    cluster_controller: 1
    stateless: 20
    log: 8
    storage: 8
    coordinator: 3
    test: 2
  databaseConfiguration:
    redundancy_mode: "double"
    commit_proxies: 6
    grv_proxies: 2
    logs: 8
    resolvers: 4
  processes:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: "100G"
      podTemplate:
        spec:
          containers:
            - name: foundationdb
              resources:
                requests:
                  cpu: 500m
                  memory: 1Gi
                limits:
                  cpu: 3000m
                  memory: 24Gi

Then I run RandomReadWriteTest

testTitle=RandomReadWriteTest
    testName=ReadWrite
    testDuration=60.0
    transactionsPerSecond=1000000
    readsPerTransactionA=1
    rangeReads=true
    valueBytes=2000
    nodeCount=500000000
    discardEdgeMeasurements=false

This always make storage queue reach to limit (1.4GB) then cluster performance drop.
I read almost question on FDB forums and get some advice from link below:

https://forums.foundationdb.org/t/reasons-for-not-co-locating-tlog-and-ss-io-characteristics-of-ss/
https://forums.foundationdb.org/t/how-to-troubleshoot-throughput-performance-degrade

I change some configuration like:

  • storageServerPerPod
  • knob_max_outstanding (from 64 to 128)
  • change EBS gp3 to maximum IOPS (16000) and throughput (1000MB)
  • add some more TLog, storage pod …

But storage queue problem is still there.
Then I change only one config for EBS gp3 type and this affect, queue doesn’t have problem anymore

  • blockSize=64 (this is maximum block size for EBS gp3, default is 4KB)

Can anyone who understand explain for me why it make cluster load data for test workload faster?
Does it consume more space on disk?
Is this configuration good for cluster?

Another question, I read a lots topic about write in FDB but I still not understand IOPS on disk and IO queue depth of FDB write, what is knob_max_outstanding parameter…