Extending the Record layer query and planner

Using Layers Record Layer
1

Hi,

I’ve looked into extending the Record layer for other use cases and I found fdb-record-layer/Extending.md at main · FoundationDB/fdb-record-layer · GitHub

I was wondering if there are any examples of extending the functionality of the system. For example, let’s say I want to enable a query like this to work:

    RecordQuery query = RecordQuery.newBuilder()
        .setRecordType("myRecord")
        .setFilter(Query.and(
            Query.field("str_field").equalsValue("hello!"),
            AComplexMathFunc(Query.field("num_field")) > 10 ))
        .setSort(Key.Expressions.field("rec_no"))
        .build();

What are the steps that I have to take to achieve that?

2

For that, you may want to look at the FunctionKeyExpression class: fdb-record-layer/FunctionKeyExpression.java at main · FoundationDB/fdb-record-layer · GitHub

That key expression allows you to register an arbitrary function, and then it will evaluate your function as a part of indexing or querying. Registering the function is a little bit of work, but the easiest way is to register an associated Factory so that the service loader can pick up your custom implementation. For example, that’s what we do with a few FunctionKeyExpressions we use internally, like this one which supports string collation: fdb-record-layer/CollateFunctionKeyExpressionFactoryICU.java at main · FoundationDB/fdb-record-layer · GitHub (Note that the factory class is annotated with @AutoService.)

To get the function to be used in a query, you can have your custom key expression definition implement QueryableKeyExpression: fdb-record-layer/QueryableKeyExpression.java at main · FoundationDB/fdb-record-layer · GitHub It can then be used as predicate in a query in a manner like this:

    RecordQuery query = RecordQuery.newBuilder()
        .setRecordType("myRecord")
        .setFilter(Query.and(
            Query.field("str_field").equalsValue("hello!"),
            Query.keyExpression(Key.Expressions.function("a_complex_math_func", Key.Expressions.field("num_field")).greaterThan(10)))
        .setSort(Key.Expressions.field("rec_no"))
        .build();

That being said, there are some limitations. The first is that, to my knowledge, the planner can only use this expression to match against an index. So, for example, you could satisfy the above query (if you removed the sort) with an index on (str_field, a_complex_math_func(num_field)), but it wouldn’t be able to apply the predicate by scanning an index on, say, (str_field, rec_no) and then filtering out records where the function predicate wasn’t matched. In theory, that is a valid plan, but the planner isn’t quite smart enough to handle that yet.

The planner itself is relatively difficult to extend without submitting a PR with planner changes yourself because it’s not particularly modular. You could, potentially, craft the plan yourself if there is a query you can’t get the planner to plan adequately, though that’s a bit extreme. The other option would be to use the new Cascades-based planner, which is supposed to be more extensible with things like custom rule sets, though that planner is currently experimental/in active development, and so neither the plans nor the codebase is guaranteed to be stable.

2 Likes