How to pass parameter to fdbserver process running inside operator

Development Kubernetes Operator Development
1

Hello,
I am trying to setup FDB Operator on K8s cluster. The issue I have hit looks similar to this

I see there is a property --knob_use_o_direct=0 that be used by fdbserver.
Is there a way to pass this through operator configuration file ?
Thanks

2

Does https://github.com/FoundationDB/fdb-kubernetes-operator/blob/master/docs/manual/operations.md help?

apiVersion: apps.foundationdb.org/v1beta1
kind: FoundationDBCluster
metadata:
  name: sample-cluster
spec:
  version: 6.2.30
  customParameters:
    - "knob_always_causal_read_risky=1"
1 Like
3

Thanks @panghy , that helped to pass the parameter!
Although unfortunately didn’t fix my problem :confused:
root 10 9 0 10:01 ? 00:00:01 /usr/bin/fdbserver --class cluster_controller --cluster_file /var/fdb/data/fdb.cluster --datadir /var/fdb/data --knob_use_o_direct 0 --listen_address 172.30.238.45:4500:tls --locality_inst

Time="1632391633.025984" Severity="40" LogGroup="kg-cluster-tls" Process="fdbserver.1": WARNING: Unrecognized knob option 'use_o_direct'
Time="1632391633.143311" Severity="40" LogGroup="kg-cluster-tls" Process="fdbserver.1": ERROR: error creating or opening process id file `/var/fdb/data/processId'.
Time="1632391633.143662" Severity="40" LogGroup="kg-cluster-tls" Process="fdbserver.1": Fatal Error: Disk i/o operation failed
Time="1632391633.216439" Severity="40" LogGroup="kg-cluster-tls" Process="fdbserver.1": Process 46 exited 1, restarting in 61 seconds

so I need to troubleshoot more what else can be causing this.
Thanks!

4

Another error I see is following

Type="UnableToLockFile" ID="0000000000000000" Filename="/var/fdb/data/processId" UnixErrorCode="25" UnixError="No locks available"

and it could be related to the fact that I am using storage class defined on top of NFS and the way locks are handled. Investigation in progress …

5

So it looks like it’s a problem with NFS/locking mechanism in FDB. I will try to isolate this to not K8S environment when I find more time and look at the code which flag has a problem. It could be also related to some services missing in Ubuntu image where operator runs.
Anyway, @panghy definitely the example you provided helps to pass this custom knob, thanks!

1 Like
6

Just as a heads up the customParameters on the top level spec is deprecated and you should be using the setting on the process level (see: fdb-kubernetes-operator/cluster.yaml at master · FoundationDB/fdb-kubernetes-operator · GitHub) I’ll create an issue in the GitHub repository to update the example(s).

PS.: Contribution are welcome :slight_smile: