# Cannot clear all keys by "clearrange \\x00 \\xff" **URL:** \ **Category:** Using FoundationDB\ **Created:** [August 10, 2019, 5:44am UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578 "2019-08-10T05:44:48Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Sraw](https://avatars.discourse-cdn.com/v4/letter/s/a88e57/32.png) [@Sraw](https://forums.foundationdb.org/u/Sraw)\ **Post date:** [August 10, 2019, 5:44am UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/1 "2019-08-10T05:44:48Z")
We have a huge database for testing purpose and now we want to re-test it. So we need to clear the whole database. I’ve searched a lot and all of them point to the same answer: `clearrange \x00 \xff`. But after running this command, there are still a lot of data inside the database: ```auto Sum of key-value sizes - 2.030 TB Disk space used - 10.812 TB ``` So I have two questions: 1. Is this the correct way to clear the whole database? 2. How could I free the whole disk space usage? Thanks in advance! ---
**Author:** ![ryanworl](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ryanworl/32/440_2.png) [@ryanworl](https://forums.foundationdb.org/u/ryanworl)\ **Post date:** [August 10, 2019, 10:31am UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/2 "2019-08-10T10:31:26Z")
I’m assuming you meant to type `clearrange \x00 \xff` and not `\ff`. Past that, deletes do not take effect immediately on disk. There is a background process which periodically frees old disk pages. It was changed relatively recently to run more often in this PR: [https://github.com/apple/foundationdb/pull/1485/files](https://github.com/apple/foundationdb/pull/1485/files) There are some knobs you can fiddle with listed in that PR if your workload will involve a lot of deletes and you want to dedicate more IO and CPU to them to run more frequently and perform more work per run. Definitely test these out on a non-prod cluster first! ---
**Author:** ![Sraw](https://avatars.discourse-cdn.com/v4/letter/s/a88e57/32.png) [@Sraw](https://forums.foundationdb.org/u/Sraw)\ **Post date:** [August 10, 2019, 10:46am UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/3 "2019-08-10T10:46:17Z")
Hi, you are correct I mean to type `clearrange \x00 \xff`. I think I have waited a relatively long time - one day. But there is no difference at all. It looks like there are really still 2TB key-value sizes in the database. But I just don’t know where are they and how to delete them. ---
**Author:** ![ryanworl](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ryanworl/32/440_2.png) [@ryanworl](https://forums.foundationdb.org/u/ryanworl)\ **Post date:** [August 10, 2019, 10:49am UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/4 "2019-08-10T10:49:04Z")
There hasn’t been any progress at all during that day? What did you start off with? The only thing that command wouldn’t delete AFAIK is data in the system keyspace, and there wouldn’t be 2TB of data in there unless someone misconfigured their application to write into it. ---
**Author:** ![Sraw](https://avatars.discourse-cdn.com/v4/letter/s/a88e57/32.png) [@Sraw](https://forums.foundationdb.org/u/Sraw)\ **Post date:** [August 10, 2019, 10:51am UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/5 "2019-08-10T10:51:17Z")
I actually use go-ycsb to benchmark the cluster. It is possible that the implementation of go-ycsb’s driver is misconfigured. But as it uses the client, will client write data into it? And yes there isn’t any progress at all during the day. As it is just a test environment. ---
**Author:** ![gaurav](https://avatars.discourse-cdn.com/v4/letter/g/b487fb/32.png) [@gaurav](https://forums.foundationdb.org/u/gaurav)\ **Post date:** [August 10, 2019, 12:07pm UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/6 "2019-08-10T12:07:24Z")
I am almost sure that there can be keys before `\x00` too. In java bindings, I usually create a range like `new byte[]{}, new byte[]{(byte) 0xFF}` to capture the `entire` key space for deletion etc. Is it possible that you have keys written in keyspace that sort before `\x00`? I do not know off-hand how to list or delete those from CLI; can you write a small code snippet to test if there is any data that you are getting with the range I mentioned above? ```auto db.run(tx -> { tx.clear(new byte[]{}, new byte[]{(byte) 0xFF}); return null; }); ``` Also, I’ve observed that FDB instantly reduces the `key-value sizes` when keys are cleared, but `Disk space used` gradually comes down as the background vacuuming progresses. In your case, it seems like there are a lot of `live` keys in the DB. ---
**Author:** ![Sraw](https://avatars.discourse-cdn.com/v4/letter/s/a88e57/32.png) [@Sraw](https://forums.foundationdb.org/u/Sraw)\ **Post date:** [August 10, 2019, 12:23pm UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/7 "2019-08-10T12:23:58Z")
Yes, I observed the same behavior as you say. The key-value sizes go down instantly while the disk usage goes down slowly. In my case, I think there are still a lot of live keys ☹ I will try your advice, thanks a lot. And hope we can get a way to check all keys including hidden ones through CLI. ---
**Author:** ![ajbeamon](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ajbeamon/32/13_2.png) [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)\ **Post date:** [August 10, 2019, 1:45pm UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/8 "2019-08-10T13:45:51Z")
There is only one key before `\x00`, which is the empty key. This is why I typically recommend using the following command in `fdbcli` to clear the whole database (note you must use double quotes `"` instead of single quotes `'`): `clearrange "" \xff` When you run a clear range like this, you should see the sum of key value sizes drop quickly, but the disk space may take a while to recover. The empty key won’t hold 2 TB, though, so that wouldn’t explain your issue. What happens if you run a range read on the main key space? It should return no keys: `getrange "" \xff` If that is empty, the next thing to check would be the `\xff` key space. The most likely way of accumulating data there would be if you turned on backup or DR but didn’t have any agents to do the associated work. An effective way to check this is to use the locality API to get shard boundaries. If you really have a lot of data somewhere, there should be a lot of shard boundaries that indicate where. In the [Python bindings](https://apple.github.io/foundationdb/api-python.html#module-fdb.locality), for example, you would use: `fdb.locality.get_boundary_keys(db, b"", b"\xff\xff")` If that doesn’t turn up anything, or if turns up a bunch of shards which are in fact empty, then there may be something going wrong. ---
**Author:** ![Sraw](https://avatars.discourse-cdn.com/v4/letter/s/a88e57/32.png) [@Sraw](https://forums.foundationdb.org/u/Sraw)\ **Post date:** [August 11, 2019, 6:46am UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/9 "2019-08-11T06:46:02Z")
Thanks for your reply! As I have to keep the test going, I eventually reinitialized the whole cluster. But I think it will happen again during the rest of the test. I will try your advice later. Some additional information: Because it is a test, I frequently change the settings of the cluster, so it might cause some problem… Or not? ---
**Author:** ![gaurav](https://avatars.discourse-cdn.com/v4/letter/g/b487fb/32.png) [@gaurav](https://forums.foundationdb.org/u/gaurav)\ **Post date:** [August 11, 2019, 11:25am UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/10 "2019-08-11T11:25:06Z")
> [@Sraw](#): > > I frequently change the settings of the cluster What settings are those? ---
**Author:** ![Sraw](https://avatars.discourse-cdn.com/v4/letter/s/a88e57/32.png) [@Sraw](https://forums.foundationdb.org/u/Sraw)\ **Post date:** [August 11, 2019, 2:54pm UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/11 "2019-08-11T14:54:22Z")
The class of processes. To figure out the best combination and to understand it better. ---
**Author:** ![markus.pilman](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/markus.pilman/32/379_2.png) [@markus.pilman](https://forums.foundationdb.org/u/markus.pilman)\ **Post date:** [August 11, 2019, 3:29pm UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/12 "2019-08-11T15:29:55Z")
That typically shouldn’t cause any issues. However, I am not sure how well we test those scenarios. If you want to be on the super safe side (might not be the case here - but just in case you want to do this in production), it is usually better to first exclude the process and then change its class. ---
**Author:** ![osamarin](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/osamarin/32/905_2.png) [@osamarin](https://forums.foundationdb.org/u/osamarin)\ **Post date:** [April 6, 2022, 3:44pm UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/15 "2022-04-06T15:44:18Z")
Hi, All. I’m tuning an application that is running against fdb. The problem is that after clearing a large range foundationdb starts performing some background disk i/o activity that influances the application performance. 1. Are there any knobs that to control the agressivity of the i/o activity? 2. Are there any capability of temporary enabling/disabling this background activity? ---
**Author:** ![markus.pilman](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/markus.pilman/32/379_2.png) [@markus.pilman](https://forums.foundationdb.org/u/markus.pilman)\ **Post date:** [June 21, 2022, 10:03pm UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/16 "2022-06-21T22:03:08Z")
I think the knobs are optimized for this use-case already. I don’t believe this will ever get better with the sqlite storage engine (and I would expect this getting much worse with RocksDB). I would recommend you try Redwood in FDB 7.1 which should at this point be pretty stable. We don’t yet recommend using Redwood for production but if you can reproduce your issue in a testing environment you could verify whether redwood solve this problem for you. We hopefully will declare redwood as being stable very soon ---
**Author:** ![osamarin](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/osamarin/32/905_2.png) [@osamarin](https://forums.foundationdb.org/u/osamarin)\ **Post date:** [June 22, 2022, 2:06pm UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/17 "2022-06-22T14:06:35Z")
> [@markus.pilman](#): > > I think the knobs are optimized for this use-case already. I don’t believe this will ever get better with the sqlite storage engine But the user should have a choice between more fast cleaning and better application performance. ---
**Author:** ![ajbeamon](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ajbeamon/32/13_2.png) [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)\ **Post date:** [July 3, 2022, 10:20pm UTC](https://forums.foundationdb.org/t/cannot-clear-all-keys-by-clearrange-x00-xff/1578/18 "2022-07-03T22:20:16Z")
There are a number of knobs that relate to this cleaning function that you can tune: > One knob in particular, `SPRING_CLEANING_LAZY_DELETE_INTERVAL`, used to be set at 1.0 and was reduced to 0.1 to cause lazy deletion to run 10x more frequently. You could try increasing it to slow it back down, but one potential consequence of this is that a cluster may not be able to reclaim space from data movement, etc. very effectively. You could also try increasing `SPRING_CLEANING_VACUUM_INTERVAL`. This is already fairly slow, but the process is not usually necessary unless since FDB can reuse unvacuumed space. If you run multiple processes on the same disk or share it with non-FDB things, then vacuuming may still be helpful though to return unused space to the OS.