How to generate record schemas dynamically?

Using Layers Record Layer
1

Hi all,

I am evaluating FDB record layer for a new project. One key feature is that users can register data schemas at runtime and then store data for those schemas.

I have read the original record layer paper and it looks like this is something CloudKit is already doing. The app developers can define the schemas for their data and CloudKit stores this in the metadata store. But I don’t understand how they do this. All the examples show how the schemas are defined in .proto files and the record layer process needs to know the schema definitions at compile time.

They hint that: “CloudKit translates the application schema into a Record Layer metadata definition and stores it in a metadata store (depicted in Figure 3).”

And later go on with: “We currently use descriptor objects to generate new metadata to be stored in the metadata store.”

Are they really running the protoc compiler in a separate process “protoc --include_imports --descriptor_set_out=descriptorfilename.desc path/to/protofile/abc.proto” and build the metadata from the output?

Thanks for any hints or explanations :slight_smile:

1 Like
2

Protobuf descriptors themselves have a Protobuf representation.

Take a DescriptorProtos.FileDescriptorProto.Builder, populate it, build to get a DescriptorProtos.FileDescriptorProto. Give that to Descriptors.FileDescriptor.buildFrom.

1 Like
3

Ok I see. So I could build the proto file dynamically via FileDescriptorProto and save it in the meta data store. And then build dynamic messages from the type descriptors.

Is there any performance penalty from doing it this way?