we have multiple storage nodes in fdb cluster. In one of the node the disk utilization reached to 70% but other nodes are having only 45% disk usage.
Upon troubleshooting further, we noticed that the impacted node was rebooted few days back and created seperate data file in the fdb data directory as shown below. Is this expected behaviour?
How frequently we rotate data directory files in fdb?
what is the reason for the impacted node having high disk usage after reboot?
-rw---S--- 1 foundationdb foundationdb 4.0K May 3 13:44 processId
-rw-r--r-- 1 foundationdb foundationdb 11 May 3 13:50 fitness
-rw-r-Sr-- 1 foundationdb foundationdb 114G May 22 21:12 storage-xxxxxxxx.redwood-v1
-rw-r-Sr-- 1 foundationdb foundationdb 61G May 31 03:51 storage-xxxxxxxx.redwood-v1
A storage server process shouldn’t have multiple storage roles (and their associated data files) recruited on it, but I think in some scenarios it’s possible that a storage roles data files get leaked. https://github.com/apple/foundationdb/pull/9794 may be relevant.
I’m pretty confident that restarting the fdbserver process should cause any leaked data files to get cleaned up though. An fdb cluster should tolerate restarting an fdbserver process pretty well so it’s not very risky. Feel free to follow up if restarting the process doesn’t delete the leaked data files.