Apple and Snowflake folk have been coordinating on the roadmap for the next FDB release (planned to be 7.0), and the document for this has been hiding on the wiki for a little while now. This is neither a promise that the work will make it into the release, nor the only fixes or improvements that will be included, but should outline most of the larger planned features.
Anyone see something important to them missing? Excited for anything in particular?
@alexmiller I think the big win will be the Redwood Storage Engine particularly the prefix-compression and all the benefits that comes with it. It would be great if Steve could describe some of the ‘significant performance improvements for a variety of workloads’ that you have measured - or maybe you are saving it for the summit. Either way sounds good!
Hi Alex, are there some preliminary details of what kinds Predicate pushdown operators would be available? I want to check the possibility of including a few user-defined server-side operations on version-stamps when forming the row-key/row-values, as discussed here?
This request is similar to the usage pattern by the FDB backup agent, as described here.
(PS: 7.0 roadmap looks quite exciting: I started to mention specific items from the list, but figured that almost everything on the list will be very useful )
It definitely is part of the content slated for @SteavedHams’s summit talk. I’ll let him decide if he’d like to leak content or not.
My current knowledge of @mbhaskar’s plans for the first version of this is to allow selection filters for tuple-encoded keys to be pushed down. e.g. "get_range(start, end), but also only return all kv-pairs that match key==0xF00 && key=="bar". But he might be able to share more light than I, or promise a future design doc
Yeah, I am in the process of writing down a design doc. Will share it very soon.
@gaurav User defined functions on version stamp operations, you were asking for, are in the write path (in Proxy). Scope of Pushdown project we are looking at is pushdowns to storage server only, just in the read path. We are looking into pushdowns of different kinds of user defined functions - Predicate pushdowns, Aggregate pushdowns and Index lookup pushdowns.
For the first version of this, we are leaning more towards a well defined language to express predicates or operations. This language gives constructs to build complex expression trees which can be pushed down to storage servers. Hope is this can help many use-cases without compromising security, reliability or portability.