Change data capture

Using FoundationDB
1

Hi there,

I am new to FoundationDB. We are evaluating foundionDb as our core transactional store.
And then using change log to build near-realtime read optimized view of our data. I am sure this topic is covered somewhere but i couldn’t find documentation on recommended way to capture the mutations.

Could someone point me to relevant documentation?

Thanks!

2

You can build that into your layer yourself. The basic idea is you perform whatever operations within your transactions you’d like, but also store the before and after values of each logical entity and write them out into a log structure using set_versionedstamped_key.

3

@alloc gave a talk on building an efficient index for clients to sync changes to data in Avoiding Conflicts in Highly-Concurrent Applications, which might be interesting for you?

4

thanks a lot ryanworl!!

Any reference implementation for adding capabilities to layer?

5

I think the record layer has the ability to do this. Here is the relevant core of the code:

But this is a part of a much larger and more complicated system, so it may not be the best example. Other than this I don’t know of anything publicly available.

6

We also have docs on how to use the version index: https://foundationdb.github.io/fdb-record-layer/Overview.html#indexing-by-version

One perhaps non-obvious thing about the semantics of this index is that if a record is updated multiple times, only its most recent update will be included in the version index. This is fine for some applications (like sync–except that you also need to include tombstones to sync deletes), but not others.

It sounds like the index included in the record layer would work for you, but it depends.