Cannot create FDB cluster over managed-nfs storage class

Using FoundationDB Kubernetes Operator
1

Hi,
We are running FDB on a OCP cluster which run in a managed-nfs which is based on vSphere (VMWare).
All the pods that created PVC failed. In the trace log, I see this:

We can login to the pod and create file under the data directory without issue but fdbserver failed to do that. We see the processID.part with a size of 0 meaning the fdbserver can only create a file name but failed to create the file at all. Any idea what is needed for NFS to set? We have tried setting (no_root_squash) and that does not help. fdbserver is running as a local ID and we have logged in with that id and write the file in the dataDir without any issue.
We are seeing these in the trace log:

<Event Severity=“10” Time=“1688582799.174804” DateTime=“2023-07-05T18:46:39Z” Type=“AsyncFileEIORenameError” ID=“0000000000000000” Filename=“/var/fdb/data/processId” UnixErrorCode=“10” UnixError=“Device or resource busy”

It seems the AsyncFIleEIOREnameError means it can’t rename the file while it is being written? The file in question is the processID, we can see it as processID.part and have a size of zero. Normally, it should contain a small binary number.
Any help or suggestion are much appreciated

2

Have you tried to move a file inside the container to see if you’re able to do this too? Have you tried to inspect the logs of the NFS service to see if there is anything obvious?

It’s a common pattern to create a file as a “temporary file” and once the file is written it will be moved to the desired destination, to make sure the application is not observing a partially written file.

3

Yes, we have tried that and it work.
Just the fdbserver cannot run the AsyncOpenandLock and getting a device/resource busy error.

4

As the storage is provided by NFS, what’s the requirement for the NFS in terms of privilege? Things like “no_root_squash” and other stuff?

5

I don’t have a test cluster at hand to run this setup over NFS, and I’m not sure if anyone has done this. If you figure out what settings on the NFS side are required, I would appreciate if you could document this in the operator repository.

6

it turns out the particular NFS that we use (v4) is not good. We went back to V3 and the issue disappeared. It is related to this particular version of NFS v4 not working well with Async Lock that FDB perform.