Thanks for confirming that this is not an internal space leak.
Based on your workload description I should have realized earlier what was going on here, but it didn’t click for me until you pointed out that the category count affects the slack.
Here’s what is happening and I think I have a configuration fix for you.
First, I should explain that Redwood does not merge pages. This was intentionally omitted (for now) because in the context of a large cluster FDB Data Movement between Storage Servers will remove all slack space from a shard during its move by clearing it at the source and generating maximally filled pages at the destination via large sequential insertions. However, since some shards may not move for long periods of time, as a fallback FDB has a feature called the Perpetual Storage Wiggle which will safely exclude and re-add Storage Servers, one at a time, once they reach some maximum age which defaults to 3 weeks. Of course, for a cluster with 1 Storage Server there is no Data Movement so everything stays in place.
As pages fill up, they must eventually must split. When a page splits (whether due to adding 1 byte or millions) the process is to build sequential pages with REDWOOD_PAGE_REBUILD_MAX_SLACK in slack space and then try to balance space between the last two pages in the split set. In the case of gradually filling a page, the split set is just 2 pages.
What is going wrong here is that the balancing logic will shift records from the second to last page of the set to the last page of the set so long as the last page slack is > REDWOOD_PAGE_REBUILD_MAX_SLACK. With the default value of 33%, this is actually shifting more records to the right-hand side of a 2-way split, leaving the left side with up to 66% slack.
If within some key range, such as under a Category prefix, your only updates are either to insert new highest key(s) or to remove lowest key(s), then the left side page of the 2-way split on insertion remains under-filled and untouched until it is eventually completely cleared. Each new right side of a 2-way split eventually becomes the left side of the next 2-way split under that prefix range, which leads to the slack space steady state you observe.
I think the reason that lowering your Category prefix count by 10x reduces the slack is that now you are inserting enough data under a single Category in one Storage Server commit cycle (which is every 250ms or 10MB, whichever comes first, and it doesn’t matter how many different clients/threads/transactions produced the sets/clears) that you are doing >2-way splits so you are leaving behind some very filled pages which mitigates the effect of the under-filled second to last pages.
The reason all of this was not apparent from the stats is that that only pages that are touched during update are reported on, so an under-filled untouched page is not.
Redwood’s split logic could be better here. The balancing part was added because not balancing between the last two pages could have a worse outcome, where you keep making small inserts on the left and generating many single-record pages on the right. But the balance % should be a different setting and should default to 50%. This of course still won’t be ideal for your workload. I do plan to add some kind of page merging in the future but I want to avoid anything that would allow certain update patterns to cause split/merge/split/merge thrashing.
Since your records are known to be far smaller than a page, the page-upsizing logic I mentioned previously won’t be active, so the knob is only affecting the balance between the last two post-split pages. Therefore, you should see lower slack with a higher value of this setting (counter-intuitive, I realize) because it will leave more slack on the right side of 2-way splits. So please try adding
knob_redwood_page_rebuild_max_slack = .90
to your foundationdb.conf file. You could also try a value of 1.0 which will leave all slack on the right side of a 2-way split.