# Record Layer Design Questions **URL:** \ **Category:** FoundationDB Layers\ **Created:** [July 27, 2022, 5:43am UTC](https://forums.foundationdb.org/t/record-layer-design-questions/3468 "2022-07-27T05:43:33Z")\ **Posts on this page:** 2\ **Page:** 2
**Author:** ![rajivr](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/rajivr/32/1100_2.png) [@rajivr](https://forums.foundationdb.org/u/rajivr)\ **Post date:** [February 12, 2023, 6:23am UTC](https://forums.foundationdb.org/t/record-layer-design-questions/3468/21 "2023-02-12T06:23:17Z")
Thanks for the detailed reply @MMcM. I really appreciate it 🙂 > [@MMcM](#): > > I don’t know what you are thinking about for your runtime execution. Currently there are no plans for a query execution engine. Please allow me to share with you my current plans. My first objective for Rust Record Layer is to have something like DynamoDB API (but on steroids) along with queuing system API. In addition, I want to have the [APIs](https://docs.google.com/document/d/17MwwGVObca3Y_KBDNM3Gkz-8znI7C_ZLdYT3gflfo0M/) to handle non-idempotent transactions. My second objective is to figure out the operational issues around running a production cluster on AWS: Access Control, Audit, Logging, Metrics, Alerting, Upgrades, DR + Multi-region replication, Failover, Load testing, etc. We will also need to lay the groundwork for regulatory compliance. In case of logging, our business case will not be able to pay for current log management solutions. Here I am planning on using Quickwit. I also plan on adding FoundationDB/Rust RecordLayer support for [Quickwit Metastore](https://github.com/quickwit-oss/quickwit/tree/v0.4.0/quickwit/quickwit-metastore). When it is done, I think this will be helpful for others in the FoundationDB community as well. Hopefully we’ll be able meet the above two objectives in 2023. For my use-case, I will need to figure out a way to do - Analytical queries. For this, I am thinking of using DuckDB and its [extensions](https://duckdb.org/docs/archive/0.6.1/extensions/overview) mechanism. - Graph Queries. There is a new project in this space called [KuzuDB](https://kuzudb.com/). I am planning on experimenting with it to see if I can make it work. If it works, it will greatly simplify graph side of the architecture. Being able to do graph queries is very important for us. - [Tantivy](https://github.com/quickwit-oss/tantivy) would be used for full text search. My current thinking is that I’ll use the Rust Record Layer queuing system to integrate the above three features. > [@MMcM](#): > > As far as I know, the Rust FDB bindings do not have `serde`-style serialization and deserialization of tuples into corresponding `struct`s. But it could, I think. That is correct. There is no serde style serialization within Rust FDB bindings. I’ll explore this further and see if the support can be introduced. > [@MMcM](#): > > Along with `prost`-generated protobuf. Few months back I did a survey around the potential serialization formats that I could use for serializing the values. The primary contenders were Protobuf and Avro. In the Rust ecosystem, for Protobuf there are two libraries: `prost` and [`rust-protobuf`](https://github.com/stepancheg/rust-protobuf). Unfortunately, `prost` does not support Protobuf reflection. `rust-protobuf` has support Protobuf reflection and message descriptors but the current maintainer is [looking](https://github.com/stepancheg/rust-protobuf#help) for a new maintainer. As you would know there is no official Protobuf spec. Currently there is no “blessed” Google Rust implementation of Protobuf in the open either. Even though Protobuf would have given me lot more capability, I did not want to put myself in a situation where I have to debug a Protobuf seralization issue in production. `prost` is primarily being used to support [gRPC](https://github.com/hyperium/tonic#overview) usecase and not the data storage usecase. On the other hand, Avro has much less capability. However, it is properly specified and its backward and forward compability semantics is well understood. The specification and code base is simple enough for me to be comfortable in case of any potential issues with the “official” Apache Avro crate. So currently I am leaning more towards using [rsgen-avro](https://github.com/lerouxrgd/rsgen-avro) and [apache-avro](https://github.com/apache/avro/tree/release-1.11.1/lang/rust) crate. Additionally upcoming [arrow2](https://github.com/jorgecarleitao/arrow2#features) and [parquet2](https://github.com/jorgecarleitao/parquet2) crates have built-in support for Avro, so that would make things a bit easier when building data pipelines. ---
**Author:** ![rajivr](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/rajivr/32/1100_2.png) [@rajivr](https://forums.foundationdb.org/u/rajivr)\ **Post date:** [July 20, 2023, 2:19pm UTC](https://forums.foundationdb.org/t/record-layer-design-questions/3468/22 "2023-07-20T14:19:37Z")
Hi @MMcM @alloc Thanks once again for the earlier feedback. I wanted to share an update on the progress of my Rust Record Layer work. Since the last update, few things have happened. 1. I’ve now moved from Avro to using Protobuf. Initially, I encountered some problems when trying to compose Avro structs. For example, I wanted to compose KeyValue cursor continuation with a RawRecord cursor continuation. I was not at all satisfied with the API ergonomics and that led me to revisit my earlier decision of using Avro. 2. In the split helper, the generated record header now includes additional metadata. The design is documented [here](https://rajivr.github.io/fdb-rl-wip/fdb_rl/split_helper/index.html). In the initial implementation of split helper, the record header was a tuple of the form `(incarnation, versionstamp)`. Now the record header is of the form `(header_version, data_splits, incarnation, versionstamp)`. 3. One design decision in my Record Layer is that, within a record store, all record types must conform to the same primary key schema. I have documented the details [here](https://rajivr.github.io/fdb-rl-wip/fdb_rl/raw_record/struct.RawRecord.html#primary-key-schema-constraint). Please let me know your thoughts! I would really appreciate any feedback! 🙂 My next step is to implement key expressions related code. [Previous page](https://forums.foundationdb.org/t/record-layer-design-questions/3468.md?page=1)