Large clear range performance

Hello,

We see performance problems when clearing large ranges in foundationdb3. We see many storage nodes get pegged at 100% cpu, and perf top shows they’re spending their time in walMerge in sqlite. We’re running foundationdb3 on EBS.

A couple questions

  • Is this improved in foundationdb6?
  • Would it make sense expose a throttled clear range in the API? We’re imagining something like issuing a special clear range with a budget for how much work it can do per storage node. This would make sense in the directory layer, where clients would know that a directory was removed and the actual work of clearing the contents of the directory can be deferred.

I have been using fdb6, and have cleared almost half a terabyte in a single range clear (on multiple occasions), without seeing the cpu problems you mentioned.

Just putting in another vote for FDB 6 here: I’ve done 400GB range clears extremely quickly, in < 500ms or so (IIRC). The important bit is that the actual on-disk storage will take longer to clean up in the background, but the key-value ranges themselves are cleared extremely quickly either way.

@aseipp @gaurav Are you running on EBS or bare metal?

I have tried this on EBS (gp2 ssd - with around 3K iops provisioned); as well as on ESX virtualized storage env. (vmware).

2 Likes

I had the very same issue with latest FDB on a big setup (3Tb raw data, 9Tb disk used) when clearing half of the database during the load: I had bunch of timeouts for short period of time (seconds) and spike in disk usage up to 100% for long period of time (had to decrease db load to get back from 100% usage)

Before you performed the clear operation, what is the average and maximum disk usage?

The default storage engine uses sqlite, which will truncate files when the disk usage is above a threshold. This background job can temporarily increase disk usage further (which is the reason why disk usage spikes during cleanup) and slowdown the storage server by eat up a lot of CPU (which is the reason why you may see timeout).