We are noticing (starting in 5.2.x) that fdb trace logs are showing Net2SlowTaskTrace truncate() in AsyncFileKAIO.actor.h which points to
AsyncFileKAIO::truncate(long) at /home/ubuntu/foundationdb/fdbrpc/AsyncFileKAIO.actor.h:263 AsyncFileKAIO::launch() at /home/ubuntu/foundationdb/./flow/flow.h:636 (inlined by) AsyncFileKAIO::launch() at /home/ubuntu/foundationdb/fdbrpc/AsyncFileKAIO.actor.h:381 N2::Net2::run() at /home/ubuntu/foundationdb/flow/Net2.actor.cpp:576 main at /home/ubuntu/foundationdb/fdbserver/fdbserver.actor.cpp:1617 (discriminator 2)
The specific line that’s slow is fallocate(). In 3.x, the code does a simple ftruncate(fd, size) while in 5.2.x it attempts a fallocate() if it’s supported (which it is on our system, the code would shut that off anyways and do ftruncate() anyways). Just throwing it out there to see if that’s something that’s known to happen (and if it’s an issue with the file system being inefficient with fallocate() vs ftruncate()). It does show up as an increase in CPU usage though. Will probably attempt a custom build to test the differences between the two.