Document expiry in FoundationDB

Using FoundationDB Running FoundationDB
1

Hi,

I am trying to find a way to setup TTL in foundationdb to auto expire the documents. Can someone please help if you have implemented this feature. Appreciate your help.

Thanks

2

FWIW, I also needed this, and I’m doing this manually.

3

I would do it this way:

  • Include the expiry date in every document
  • Create an index structure for the documents based on the expiry date: (expires_at, document_id) = empty byte array
  • When reading a document, check if it has been expired, if it has, return null, but also delete the document (lazy cleanup)
  • Periodically run a process that range reads document ids in batches from the index from 0 to current time and deletes those items (eager cleanup)

If you have a very large number of documents, instead of the index (expires_at, document_id) you can create a (logical_partition, expires_at, document_id) so that frequent writes of the index will hit different storage nodes and also you can parallelize the background batch process.

1 Like
4

Wouldn’t it be possible to add the possibility to delete data from a key range if said data has a version older than a threshold? This would make it easy to implement a TTL mechanism.

A scan could also benefit from the possibility to only read data within a given version range.