Understand FDB read/write with disk IOPS/throughtput/Blocksize

[mason]

Hi all,
I setup FDB cluster with fdb-kubernetes-operator with 8 instances m6a.2xlarge (8 vcore, 32GB RAM each).
I have 8 storage pods, each pod run 3 storage processes.

apiVersion: apps.foundationdb.org/v1beta2
kind: FoundationDBCluster
metadata:
  labels:
    cluster-group: foundationdb-cluster
  name: foundationdb-cluster
spec:
  version: 7.1.27
  faultDomain:
    key: foundationdb.org/none
  storageServersPerPod: 3
  processCounts:
    cluster_controller: 1
    stateless: 20
    log: 8
    storage: 8
    coordinator: 3
    test: 2
  databaseConfiguration:
    redundancy_mode: "double"
    commit_proxies: 6
    grv_proxies: 2
    logs: 8
    resolvers: 4
  processes:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: "100G"
      podTemplate:
        spec:
          containers:
            - name: foundationdb
              resources:
                requests:
                  cpu: 500m
                  memory: 1Gi
                limits:
                  cpu: 3000m
                  memory: 24Gi

Then I run RandomReadWriteTest

testTitle=RandomReadWriteTest
    testName=ReadWrite
    testDuration=60.0
    transactionsPerSecond=1000000
    readsPerTransactionA=1
    rangeReads=true
    valueBytes=2000
    nodeCount=500000000
    discardEdgeMeasurements=false

This always make storage queue reach to limit (1.4GB) then cluster performance drop.
I read almost question on FDB forums and get some advice from link below:

Storage Server CPU bottleneck - Growing data lag

You seem to be running a very write-heavy workload. As the disk doesn’t seem to be saturated, my guess is that the write-queue within FDB is the problem. The disk interface uses AIO with O_DIRECT, and we only allow to queue 64 operations at a time and reads will get a higher priority. This means that the max throughput is somewhere around 4KB * 64 * disk-latency - or in other words: your performance might be limited by disk latency instead of disk throughput… There are a few things you can do: …

https://forums.foundationdb.org/t/reasons-for-not-co-locating-tlog-and-ss-io-characteristics-of-ss/
https://forums.foundationdb.org/t/how-to-troubleshoot-throughput-performance-degrade

I change some configuration like:

• storageServerPerPod
• knob_max_outstanding (from 64 to 128)
• change EBS gp3 to maximum IOPS (16000) and throughput (1000MB)
• add some more TLog, storage pod …

But storage queue problem is still there.
Then I change only one config for EBS gp3 type and this affect, queue doesn’t have problem anymore

• blockSize=64 (this is maximum block size for EBS gp3, default is 4KB)

Can anyone who understand explain for me why it make cluster load data for test workload faster?
Does it consume more space on disk?
Is this configuration good for cluster?

Another question, I read a lots topic about write in FDB but I still not understand IOPS on disk and IO queue depth of FDB write, what is knob_max_outstanding parameter…

[SteavedHams (Steve Atherton)]

You will have to post more details about what you have done.

The test spec you posted is going to first build a 500M * 2000 byte data set, which is 2 TB, before it executes the test workload. During this time, given your log and proxy configuration the Storage Queue will be full because the storage servers persisting the loaded data to disk will be the bottleneck.

After the build step, this workload will likely be IO bound at a low enough transaction rate rate that the Storage Queue will be smaller. The storage queue holds 5s worth of mutations + a very large amount of overhead normally larger than the KV records changed in the 5s window.

The SQLite engine is using 4k pages by default. Redwood uses 8k. These are the page sizes that FDB is writing, so there is no reason I can think of that making your GP3 volume block size larger would increase performance. If that is truly occurring, I would be curious to know why.