Record Layer Design Questions

[rajivr (Rajiv M Ranganath)]

Thanks for the detailed reply @MMcM. I really appreciate it

@MMcM

I don’t know what you are thinking about for your runtime execution.

Currently there are no plans for a query execution engine. Please allow me to share with you my current plans.

My first objective for Rust Record Layer is to have something like DynamoDB API (but on steroids) along with queuing system API. In addition, I want to have the APIs to handle non-idempotent transactions.

My second objective is to figure out the operational issues around running a production cluster on AWS: Access Control, Audit, Logging, Metrics, Alerting, Upgrades, DR + Multi-region replication, Failover, Load testing, etc. We will also need to lay the groundwork for regulatory compliance.

In case of logging, our business case will not be able to pay for current log management solutions. Here I am planning on using Quickwit. I also plan on adding FoundationDB/Rust RecordLayer support for Quickwit Metastore. When it is done, I think this will be helpful for others in the FoundationDB community as well.

Hopefully we’ll be able meet the above two objectives in 2023.

For my use-case, I will need to figure out a way to do

• Analytical queries. For this, I am thinking of using DuckDB and its extensions mechanism.

• Graph Queries. There is a new project in this space called KuzuDB. I am planning on experimenting with it to see if I can make it work. If it works, it will greatly simplify graph side of the architecture. Being able to do graph queries is very important for us.

• Tantivy would be used for full text search.

My current thinking is that I’ll use the Rust Record Layer queuing system to integrate the above three features.

@MMcM

As far as I know, the Rust FDB bindings do not have serde-style serialization and deserialization of tuples into corresponding structs. But it could, I think.

That is correct. There is no serde style serialization within Rust FDB bindings. I’ll explore this further and see if the support can be introduced.

@MMcM

Along with prost-generated protobuf.

Few months back I did a survey around the potential serialization formats that I could use for serializing the values. The primary contenders were Protobuf and Avro.

In the Rust ecosystem, for Protobuf there are two libraries: prost and rust-protobuf. Unfortunately, prost does not support Protobuf reflection. rust-protobuf has support Protobuf reflection and message descriptors but the current maintainer is looking for a new maintainer. As you would know there is no official Protobuf spec. Currently there is no “blessed” Google Rust implementation of Protobuf in the open either.

Even though Protobuf would have given me lot more capability, I did not want to put myself in a situation where I have to debug a Protobuf seralization issue in production. prost is primarily being used to support gRPC usecase and not the data storage usecase.

On the other hand, Avro has much less capability. However, it is properly specified and its backward and forward compability semantics is well understood. The specification and code base is simple enough for me to be comfortable in case of any potential issues with the “official” Apache Avro crate. So currently I am leaning more towards using rsgen-avro and apache-avro crate. Additionally upcoming arrow2 and parquet2 crates have built-in support for Avro, so that would make things a bit easier when building data pipelines.

[rajivr (Rajiv M Ranganath)]

Hi @MMcM @alloc

Thanks once again for the earlier feedback. I wanted to share an update on the progress of my Rust Record Layer work. Since the last update, few things have happened.

1. I’ve now moved from Avro to using Protobuf. Initially, I encountered some problems when trying to compose Avro structs. For example, I wanted to compose KeyValue cursor continuation with a RawRecord cursor continuation. I was not at all satisfied with the API ergonomics and that led me to revisit my earlier decision of using Avro.

   I then discovered prost_reflect crate, which is a thin wrapper on top of prost crate. It also provides DynamicMessage capability.

   With reflection and dynamic messages available, I can take a similar approach as what Java RecordLayer does and store the schema information in the database itself. This is something I would have had to build using Avro. With Protobuf I can basically adopt a proven pattern from Java RecordLayer.

   I was also happy with the API and the generated code, so I decided to make the switch. You can see the proto definitions for KeyValueContinuation and RawRecordContinuation here. The only limitation right now is unlike proto2, proto3 does not have required field semantics. That requires me to have work around in Rust by defining additional types (relevant links 1, 2).

2. In the split helper, the generated record header now includes additional metadata. The design is documented here. In the initial implementation of split helper, the record header was a tuple of the form (incarnation, versionstamp). Now the record header is of the form (header_version, data_splits, incarnation, versionstamp).

   Initially I attempted to implement of RawRecord cursor by hand. While implementing it I got overwhelmed with the potential edge cases that could occur from splitting records and error conditions from KeyValueCursor. Eventually, I ended up designing a statechart, based on which both the code and unit tests was written (relevant links 1, 2, 3, 4).

   Turns out having data_splits in the record header is very useful for ensuring correctness. Additionally, with statechart approach, RawRecordCursor does not exceed out-of-band limits.

3. One design decision in my Record Layer is that, within a record store, all record types must conform to the same primary key schema. I have documented the details here.

Please let me know your thoughts! I would really appreciate any feedback!

My next step is to implement key expressions related code.