Creating secondary index on data within a Map

Using Layers Record Layer
1

Hi,

I am completely new to Foundation DB and Record layer. I was trying to understand secondary indexes through the existing documentation. I am curious to know how would be index a data which is unknown before hand.

For example… imagine a data model below…

message Outer {
	optional Inner static_events = 1;
         map<String, Inner> dynamic_events = 2;	
}

message Inner {
       optional Event known_a = 1;
       optional Event known_b = 2;
       optional Event known_c = 3;

}

message Event {
	optional string name = 1;
     ......
}

Creating secondary indexes for any of the properties (eg name) for the “Event” type nested within the static_events is understood.
Can we do the same for the the same type within the dynamic_event map ?
Can we write some sort of FunctionKeyExpression implementation to do that ?

Thanks,

Samit

2

In Protobuf, a map is actually implemented as a repeated nested message with two fields, one called “key” and one called “value”. The record layer allows you to then address them as such, and there are some constants and utilities included to make that somewhat easier.

I’m not exactly sure which indexes you’re interested in, but if you wanted to index just the “name” of all known_a dynamic events, you could do something like:

Key.Expressions.field("dynamic_events", FanType.FanOut).nest(
   Key.Expressions.field(Key.Expressions.MAP_VALUE_FIELD).nest(
       Key.Expressions.field("known_a").nest(
           Key.Expressions.field("name")
       )
   )
)

And getting a little more advanced, if you wanted to have an index on, say, (dynamic_event.key, known_b.name), you could do that with something like:

Key.Expressions.field("dynamic_events", FanType.FanOut).nest(
   Key.Expressions.concat(
       Key.Expressions.field(Key.Expressions.MAP_KEY_FIELD),
       Key.Expressions.field(Key.Expressions.MAP_VALUE_FIELD).nest(
           Key.Expressions.field("known_b").nest(
               Key.Expressions.field("name")
           )
       )
   )
)

Note that the known_b.name is indexed here in a way that associates it with the dynamic_event.key that produced it specifically.

There are also utility methods for defining key expressions on either the keys, the values, the (key, value) pairs, or the (value, key) pairs of a map. (Those last two produce the same index entries except with the order flipped, which means that the former can answer queries of the form key = x, the latter can answer queries of the form value = y, and both can answer queries of the form key = x AND value = y.) For example, if you just wanted to index the keys of all dynamic events, you can do that with:

Key.Expressions.mapKeys("dynamic_events")

Which is essentially syntactic sugar for:

Key.Expressions.field("dyanmic_events", FanType.FanOut).nest(
    Key.Expressions.field(Key.Expressions.MAP_KEY_FIELD)
)

Unfortunately, none of those static utility methods play great with map types where the value is itself a nested message, so it’ll have to be done the long way if you need to nest into the value.

See: our javadoc on Key.Expressions, in particular all of the things with “map” in the name: https://javadoc.io/static/org.foundationdb/fdb-record-layer-core/2.8.97.0/com/apple/foundationdb/record/metadata/Key.Expressions.html

3

Hi Alec,

Thanks for the detailed reply.

I made a mistake in my earlier data model. The dynamic_events has “Event” as the type instead of “Inner”.

message Outer {
	optional Inner static_events = 1;
         map<String, **Event**> dynamic_events = 2;	
}

message Inner {
       optional Event known_a = 1;
       optional Event known_b = 2;
       optional Event known_c = 3;

}

message Event {
	optional string name = 1;
     ......
}

But from you reply it looks like it would still be possible won’t it ?

Key.Expressions.field("dynamic_events", FanType.FanOut).nest(
   Key.Expressions.field(Key.Expressions.MAP_VALUE_FIELD).nest(
       Key.Expressions.field("name")
       )
   )
4

Yeah, exactly. That will produce an index on all names from all dynamic events, linking the name back to the Outer record from which it came.