CometDB: A FoundationDB queue manager for Erlang

Using FoundationDB
1

The project link is here.

CometDB

CometDB is an Erlang implementation of queue model over FoundationDB.

About

This project aims to provide you a set of simple interfaces so that you can use FoundationDB as a Distributed, High Concurrency and Low Latency Queue.

Features

  • You can insert data in batch.
  • You can fetch data in batch.
  • You can delete data in range.
  • You can insert data of any size; we overcome the size limitation of FDB.

How to use

We need to setup the FDB environment first. You can find details in FoundationDB official website.

The usage is really simple. Here is an example:

P = comet:start(). 
comet:insert(P, <<"testq">>, <<"testvalue">>). 
comet:batch_insert(P, <<"testq">>, [<<"testvalue_2">>, <<"testvalue_3">>, <<"testvalue_4">>]). comet:fetch(P, <<"testq">>, 100). 
comet:delete(P, <<"testq">>, 1).

TODO

  • type checking
  • stream
  • cursors
  • index
  • priority queue
  • connection management
  • performance
  • tests …
4 Likes
2

Hello fellow BEAM user… :slight_smile: Very cool!

I’m looking to use FoundationDB to implement queues also and I came across this in the documentation:

High-Contention Dequeue Operations

To minimize conflicts during dequeue operations, we can use a staging technique to service the requests. If a dequeue operation doesn’t initially succeed, it registers a dequeue request in a semi-ordered set of such requests. It then enters a retry loop in which it attempts to fulfill outstanding requests.

Did you implement that algorithm? Or how did you deal with high contention consumers?

Do you know how that “staging” algorithm works? Can you help explain it?

Thanks for the help!

3

Hi. Sorry for the late reply. We didn’t use foundationdb in our product environment at last. So I didn’t notice the comments.

I know little about “staging” algorithm… But if you need any help just let me know. Maybe we can build a solid queue model for FoundationDB in erlang together.

4

@wfnuser Hi! I’m interested in knowing more about your staging algorithm. Could you please describe it?

5

I’m not familiar with Erlang, it would be nice if you could describe your implementation of the Distributed, High Concurrency and Low Latency Queue .