We see performance problems when clearing large ranges in foundationdb3. We see many storage nodes get pegged at 100% cpu, and perf top shows they’re spending their time in walMerge in sqlite. We’re running foundationdb3 on EBS.
A couple questions
Is this improved in foundationdb6?
Would it make sense expose a throttled clear range in the API? We’re imagining something like issuing a special clear range with a budget for how much work it can do per storage node. This would make sense in the directory layer, where clients would know that a directory was removed and the actual work of clearing the contents of the directory can be deferred.
I have been using fdb6, and have cleared almost half a terabyte in a single range clear (on multiple occasions), without seeing the cpu problems you mentioned.
Just putting in another vote for FDB 6 here: I’ve done 400GB range clears extremely quickly, in < 500ms or so (IIRC). The important bit is that the actual on-disk storage will take longer to clean up in the background, but the key-value ranges themselves are cleared extremely quickly either way.
I had the very same issue with latest FDB on a big setup (3Tb raw data, 9Tb disk used) when clearing half of the database during the load: I had bunch of timeouts for short period of time (seconds) and spike in disk usage up to 100% for long period of time (had to decrease db load to get back from 100% usage)
Before you performed the clear operation, what is the average and maximum disk usage?
The default storage engine uses sqlite, which will truncate files when the disk usage is above a threshold. This background job can temporarily increase disk usage further (which is the reason why disk usage spikes during cleanup) and slowdown the storage server by eat up a lot of CPU (which is the reason why you may see timeout).
I am clearing some keys with a big range… approximately 1000 gb. The clear operation for every key is taking around 15 milliseconds. which is quite high because I have 170 million keys and it takes around 2 weeks to clear all the keys. Any help on how to make it faster?
It sounds like you’re clearing one key per transaction, and you have few commits in flight at a time. Try clearing multiple keys per transaction (sharing the cost of starting and committing transactions), and also have multiple commits in flight to increase throughput. See throughput requires concurrency