Hi,
I manually complied the master code branch which seems to include the fill-factor features since there are already have some REDWOOD_PAGE_REBUILD_FILL_FACTOR in the master source code like below shown:
fdbserver/VersionedBTree.actor.cpp: state int pageFillTarget = pageSize * SERVER_KNOBS->REDWOOD_PAGE_REBUILD_FILL_FACTOR;
fdbserver/VersionedBTree.actor.cpp: pageFillTarget = pageSize * SERVER_KNOBS->REDWOOD_PAGE_REBUILD_FILL_FACTOR;
fdbserver/Knobs.h: double REDWOOD_PAGE_REBUILD_FILL_FACTOR; // When rebuilding pages, start a new page after this capacity
fdbserver/Knobs.cpp: init( REDWOOD_PAGE_REBUILD_FILL_FACTOR, 0.66 )
so, is it possible to change REDWOOD_PAGE_REBUILD_FILL_FACTOR from foundationdb.conf? and how we can set the customized fill-factor?
I generally only push changes to my branch that pass correctness testing and what is there now is likely to be what goes into 6.3.0.
The most interesting very recent addition is the RedwoodMetrics trace event. It gives a great deal of insight into what Redwood is doing. If you have any questions about the performance you are seeing then those events will be the first thing I ask for.
many thanks for your so much detail explanation, that’s very valuable.
I used master code branch because it passed the compilation but release-6.3 didn’t basing the latest docker image.
I am testing basing on my built out rpm package on master code branch, and later I will use your latest Redwood source code to try.
for those released FDB versions that don’t have knob_redwood_page_rebuild_fill_factor, every page will be filled fully or some other size?
when using redwood engine, FDB use direct-io or buffer-io by default? can we choose direct-io or buffer-io from certain parameters?
using latest master branch code which including the feature of setting knob_redwood_page_rebuild_fill_factor, is the storage engine still named ssd-redwood-experimental or some else?
let’s say: reducing knob_redwood_page_rebuild_fill_factor from default 0.66 to 0.50, in my test result, the ops/sec performance of 100-update-workloads will drop a lot (0.66:9321 vs 0.50:5009), do you think it is reasonable?
The hard-coded value has been 0.66 for a while, but going back far enough it used to be 1.0.
It should be using non-buffered IO. Since there used to be a bug here with the file open flags, I just checked again and it should be using non-buffered, using our linux KAIO file class. To verify that this is what it is doing, please look for AsyncFileKAIOOpen events in your log files, you should see one for each .redwood file opened.
Yes, it will still be called ssd-redwood-experimental until FDB 7.0
Remember that filling pages less full means writing more pages which means touching more unique pages during commits, as each page covers a smaller range of keys than if the pages were more full.
I can tell you more about what your workload is doing if you can post some RedwoodMetrics trace events from your logs from somewhere in the middle of your test. Builds from master would not yet have these events, but as of a few minutes ago release-6.3 branch in the main repo does and my personal repo on GitHub has had them for a few days, so if you don’t have them in your logs you will need to update to one of those.
@SteavedHams
Your explanation are so helpful! Thanks very much!
yes,my test build is basing on the master code branch (I complied it and make the rpm), so there could have no any RedwoodMetrics trace.
I will try to make the compilation pass successfully with release-6.3 or your personal repo on github.