I setup FDB cluster with fdb-kubernetes-operator with 8 instances m6a.2xlarge (8 vcore, 32GB RAM each).
I have 8 storage pods, each pod run 3 storage processes.
apiVersion: apps.foundationdb.org/v1beta2 kind: FoundationDBCluster metadata: labels: cluster-group: foundationdb-cluster name: foundationdb-cluster spec: version: 7.1.27 faultDomain: key: foundationdb.org/none storageServersPerPod: 3 processCounts: cluster_controller: 1 stateless: 20 log: 8 storage: 8 coordinator: 3 test: 2 databaseConfiguration: redundancy_mode: "double" commit_proxies: 6 grv_proxies: 2 logs: 8 resolvers: 4 processes: volumeClaimTemplate: spec: resources: requests: storage: "100G" podTemplate: spec: containers: - name: foundationdb resources: requests: cpu: 500m memory: 1Gi limits: cpu: 3000m memory: 24Gi
Then I run RandomReadWriteTest
testTitle=RandomReadWriteTest testName=ReadWrite testDuration=60.0 transactionsPerSecond=1000000 readsPerTransactionA=1 rangeReads=true valueBytes=2000 nodeCount=500000000 discardEdgeMeasurements=false
This always make storage queue reach to limit (1.4GB) then cluster performance drop.
I read almost question on FDB forums and get some advice from link below:
I change some configuration like:
- knob_max_outstanding (from 64 to 128)
- change EBS gp3 to maximum IOPS (16000) and throughput (1000MB)
- add some more TLog, storage pod …
But storage queue problem is still there.
Then I change only one config for EBS gp3 type and this affect, queue doesn’t have problem anymore
- blockSize=64 (this is maximum block size for EBS gp3, default is 4KB)
Can anyone who understand explain for me why it make cluster load data for test workload faster?
Does it consume more space on disk?
Is this configuration good for cluster?
Another question, I read a lots topic about write in FDB but I still not understand IOPS on disk and IO queue depth of FDB write, what is knob_max_outstanding parameter…