I’m trying to find out why fdbserver exits immediately with “Fatal Error: File could not be written”. It doesn’t tell me where it tried to write, it gets started by fdbserver so stracing it isn’t obvious, and perusal of the fine source code didn’t get me very far.
I checked and the following locations are definitely readable and writable by the foundationdb user:
fdb.cluster and the directory it’s in
datadir
logdir
According to the documentation, these should be the only places where fdbserver tries to write.
Replying to myself, because I think this might be useful for others in the future, and it took me a while to find the problem.
“Fatal Error: File could not be written” was caused by a directory two levels up from datadir not having the ‘x’ permission for ‘others’. This is why fdbserver wasn’t able to write: it could not access its data directory.
I do believe that the log message could be improved by telling users what could not be written, but the fault here is entirely mine.
Hey there,
When troubleshooting the error “Fatal Error: File could not be written” in fdbserver, there are a few additional places and steps you might want to check beyond the basic documentation. Here are some hints and possible steps to follow:
Check Configuration Paths: Ensure datadir, logdir, and fdb.cluster paths in foundationdb.conf are correct.
Verify Permissions: Confirm the foundationdb user has read/write permissions for these directories and files.
Ensure Disk Space: Make sure there’s enough disk space available.
Check Quotas: Verify no filesystem quotas are restricting writing.
SELinux/AppArmor: Ensure these security measures aren’t blocking writes.
Environment Variables: Make sure no environment variables override the paths.
Use strace: Trace file operations to find where the failure occurs.
Review Logs: Check FoundationDB logs for additional errors.
Check Filesystem Health: Use filesystem tools to diagnose issues.
Run as Root: Temporarily run fdbserver as root to identify permission issues.