Split long record causes conflict with other record

Using Layers Record Layer
1

I have two records

    message SampleEntityName {
        string entity_id = 1;
        string name = 2;
    }

    message SampleEntityNumber {
        string entity_id = 1;
        int32 number = 2;
    }

The SampleEntityName has entity_id is primary key and SampleEntityNumber has (entity_id, number) as primary key.

When i try to get some SampleEntityName records, it throws com.apple.foundationdb.record.RecordCoreException: Expected only a single key extension for split record.. After some investigation, i guess it’s because of the splitLongRecords (my model is bigger than 100kb so i need split long records), since it try to add another long to primary key.

Is there any solution to fix this, currently i split the SampleEntityNumber to another store, but it’s a bit boilerplate. I’m wondering is there any solution that i can fit these two records on the same store?

2

Yeah, unfortunately, that’s true. We really should document this better, but (1) the Record Layer doesn’t like it if you have two records with the same primary key but one is a prefix of another but (2) it doesn’t warn you, it just errors at inconvenient times. We’ve discussed ways in which we could make it better, but I think they are:

  1. Enforce this rule, and check for other keys with the same prefix at write time
  2. Enforce this rule at meta-data validation time, i.e., look at the primary key definitions and validate that you don’t have two entries where the types match, but one is a prefix of another
  3. Change our data model on disk so that we use nested tuples, which would solve this problem. I’d also have to think if we’d need to do that for index entries (I think not, but I could be wrong), and there probably wouldn’t be an upgrade path for existing record stores

In your case, assuming your still developing, probably the base courses of action would be to:

  1. Change each primary key to be prefixed by its type name, e.g., something like: concat(recordType(), field("entity_id")) and concat(recordType(), field("entity_id"), field("number")). This has a number of advantages anyway, so you might want to do this for its own sake.
  2. If, for some reason, you want all of your records of all types to have primary keys prefixed by “entity_id” (because, e.g., you will sometimes query for all of the things for a given entity, and want the name and number records together), you could change the primary key for SampleEntityName to something like concat(field(entity), new LiteralKeyExpression<Long>(null)), which will add a suffix to that key to avoid this problem.

I hope that helps. This is definitely a thorny pit that exists in the Record Layer that probably we should do a better job either removing or at least publicizing.

2 Likes
3

Great, thanks!
The error is a bit of implicit to me. I think it should be on the compile time or something that i can notice earlier.
And i think adding Record Type as prefix of all primary key is a good suggestion. But in the future when a primary key can’t be a prefix of another primary key, will it be an issue? Since i think the recordType is just a string.

4

Yeah. There are probably ways we could adjust this error to be more upfront and let you know what happened. Filed: One record cannot have a primary key that is a prefix of another · Issue #959 · FoundationDB/fdb-record-layer · GitHub

So, it’s a little subtle, but it’s actually only a problem if the primary key tuple is an element-wise prefix of another primary key. So, for example, if you had a single string field as your primary key, then ("foo") and ("food") could coexist without issue. The problem is if you had something like one record with primary key ("foo") and another with a key like ("foo", "bar").

It’s also possibly worth noting that the record type key expression isn’t (generally) a string, but a configurable item that the meta-data requires is unique per record type. By default, it’s actually an integer (the same integer used for the field in the union message in the protobuf definition).

1 Like