Large clear range performance

#1

Hello,

We see performance problems when clearing large ranges in foundationdb3. We see many storage nodes get pegged at 100% cpu, and perf top shows they’re spending their time in walMerge in sqlite. We’re running foundationdb3 on EBS.

A couple questions

  • Is this improved in foundationdb6?
  • Would it make sense expose a throttled clear range in the API? We’re imagining something like issuing a special clear range with a budget for how much work it can do per storage node. This would make sense in the directory layer, where clients would know that a directory was removed and the actual work of clearing the contents of the directory can be deferred.
(gaurav) #2

I have been using fdb6, and have cleared almost half a terabyte in a single range clear (on multiple occasions), without seeing the cpu problems you mentioned.

(Austin Seipp) #3

Just putting in another vote for FDB 6 here: I’ve done 400GB range clears extremely quickly, in < 500ms or so (IIRC). The important bit is that the actual on-disk storage will take longer to clean up in the background, but the key-value ranges themselves are cleared extremely quickly either way.

(Meng Xu) #4

@aseipp @gaurav Are you running on EBS or bare metal?

(gaurav) #5

I have tried this on EBS (gp2 ssd - with around 3K iops provisioned); as well as on ESX virtualized storage env. (vmware).

2 Likes
(Daniil Gitelson) #6

I had the very same issue with latest FDB on a big setup (3Tb raw data, 9Tb disk used) when clearing half of the database during the load: I had bunch of timeouts for short period of time (seconds) and spike in disk usage up to 100% for long period of time (had to decrease db load to get back from 100% usage)