Confusing error when using FDBMetaDataStore

I have a basic FDBRecordStore that is instantiated like so:

var metadata: RecordMetaDataBuilder = RecordMetaData
    .newBuilder()
    .setRecords(fileDescriptor)
val recordType = session.metadata.getRecordType("User")
recordType.setPrimaryKey(
    Key.Expressions
    .concat(Key.Expressions.recordType(), Key.Expressions.field("id"))
)
context = db.openContext()
val store = FDBRecordStore.newBuilder()
    .setMetaDataProvider(session.metadata)
    .setContext(context)
    .setKeySpacePath(session.getPath())
    .createOrOpen()
db.close()

I tried to add a FDBMetaDataStore like so:

var metadata: RecordMetaDataBuilder = RecordMetaData
    .newBuilder()
    .setRecords(fileDescriptor)

val recordType = metadata.getRecordType("User")
recordType.setPrimaryKey(
    Key.Expressions
    .concat(Key.Expressions.recordType(), Key.Expressions.field("id"))
)

context = db.openContext()
val mdStore = new FDBMetaDataStore(context, session.getPath())
mdStore.saveRecordMetaData(metadata)
mdStore.setLocalFileDescriptor(fileDescriptor)
context.commit()
context.close()

context = db.openContext()
val store = FDBRecordStore.newBuilder()
    .setMetaDataProvider(metadata.build(true))
    .setMetaDataStore(mdStore)
    .setContext(context)
    .setKeySpacePath(session.getPath())
    .createOrOpen()
context.close()
db.close()

And now the createOrOpen call fails with RecordStoreNoInfoAndNotEmptyException. I’ve tried setting this up a variety of different ways and I keep getting the same error. In looking at the Record Layer tests all the asserts on that exception seem to be very simple cases where no metadata is provided.

Any thoughts?

Is it possible that at some time in the past you wrote data to this subspace without using the createOrOpen builder methods?

Those methods maintain a header on the record store that contains the latest meta-data version at which the store was accessed. This in turn drives schema evolution operations, such as building new indexes. If there are records written to the record store before the meta-data version is recorded in the header, it is impossible to know what meta-data was used to write them and so what schema evolution operations they might be missing. That is what that error message is about.

If that is the case, look at StoreExistenceCheck for other ways to open the record store once to get the current meta-data version recorded so that you are back on track going forward.

Another possibility is that you are accidentally using (part of) the record store subspace for something else. The key in the exception’s logInfo will tell you want it found and perhaps that will indicate this.

Which can be inspected with something like:

KeyValueLogMessage msg = KeyValueLogMessage.build("some message")
    .addKeysAndValues(((LoggableException)err).getLogInfo());
LOGGER.error(msg.toString()); // or equivalent

We should probably add a utility method to add those to log messages, etc… Might be good to document that that’s what we do with our exceptions somewhere, though where is non-obvious…

Thanks @alloc and @MMcM for the responses and the tips.

I’ve been wiping the database between test runs so I think that might disqualify some of these possibilities?

I ran KeyValueLogMessage and got the following output: some message key="(null, 0)" key_space_path="/hi:'hi'"

Going to poke around the values in the database a bit more to see if I’m missing anything.

Interesting. That’s not a key that a record store should ever write.

How are you wiping the database? With a clear_range(\x00, \xff)? Or by calling, say, FDBRecordStore::deleteAllRecords?

Are you using the /"hi"/null path, by any chance? That kind of looks like one is trying to store a record store at /"hi"/null but then trying to access it just from /"hi". (Or that there are two stores, one at /"hi"/null and another at /"hi", and then they aren’t playing nicely.)

Either setting the db up from scratch in docker or with the following:

val metaDataSubspace = session.getPath().toSubspace(context)
context.ensureActive.clear(Range.startsWith(metaDataSubspace.pack))
context.commit()

I don’t think I’m using a path with null in it. The path used in this setup passes the following: assert(path.toString() == "/hi:\"hi\"") (not sure if the escaped quotes are expected)

I’m building my keyspace/path as follows:

val ks = new KeySpace(
  new KeySpaceDirectory(
    this.keySpace,
    KeySpaceDirectory.KeyType.STRING,
    this.keySpace
  )
)
return ks.path(this.keySpace)

this.keySpace is a String

Right before the createOrOpen call there is just one key (I used this to grab all keys: Range-reading all key-values) in the database, it is:

[{\x02hi\x00\x00\x14 [10 123 10 12 116 97 98 108 101 115 46 112 114 111 116 111 34 44 10 4 85 115 101 114 18 14 10 2 105 100 24 1 32 1 40 9 82 2 105 100 18 20 10 5 101 109 97 105 108 24 2 32 1 40 9 82 5 101 109 97 105 108 34 45 10 15 82 101 99 111 114 100 84 121 112 101 85 110 105 111 110 18 26 10 5 95 85 115 101 114 24 1 32 1 40 11 50 5 46 85 115 101 114 82 4 85 115 101 114 66 6 90 4 109 97 105 110 98 6 112 114 111 116 111 51 26 26 10 4 85 115 101 114 18 18 10 16 10 2 90 0 10 10 26 8 10 2 105 100 16 1 24 1 32 0 40 2 64 0]}]

Oh, wait, and then you’re using the same key space for your meta-data and your data?

Then I think I see what’s going on here. The meta-data store will store it’s meta-data in a key prefixed with a (tuple packed) null, which then conflicts with what the record store is trying to do with it.

Something like:

new KeySpace(
   new KeySpaceDirectory("hi", KeyType.STRING, "hi")
      .addSubdirectory(new KeySpaceDirectory("meta_data", KeyType.STRING, "m"))
      .addSubdirectory(new KeySpaceDirectory("data", KeyType.STRING, "d"))
)

I think will do it? Then you use ks.path("hi").add("meta_data") for the meta-data store and then ks.path("hi").add("data") for the data store.

And then you can also do fancy things if you have multiple data stores with the same meta-data, but this will be the simplest to get you to something that works. (You could also choose to use integers or null instead of strings, but again, not necessary; just a possible storage optimization as (small) integers are cheaper to store than (all but the smallest) strings).

1 Like

oh man, this is so obvious in retrospect. it works now, thank you!