We are getting this error in production:
LogGroup="default" Process="fdbserver.4505": ERROR: error creating or opening process id file /mnt/foundationdb/sdc/data/4505/processId'.
When this happens we get an I/O error on all 4 drives we have, if we try for example ls when this occurs. Our disks with SMART metrics look fine and are not close to the limits.
If we reboot the box everything comes back, but we would like to understand what could be wrong.
Each FDB server has 4x8TB SSDs, Samsung 870 EVO drives.
Each server has 128GB RAM (non-ECC, DDR5).
Any guidance you can provide will be very helpful. We are OK to upgrade the hardware if required.