QFoundation - Entity layer for Java

[Imperatorx]

Let me introduce my Java entity layer to the community.

Screenshots, documentation, getting started guide, maven coordinates and source code are available at the project website licensed under the Apache License.

The entity layer provides a JPA-like access pattern for your POJO entities, where you have the choice to use mutable or immutable entities. It has a persistence context, so every mutation is automatically applied at commit/flush time. The layer is packaged as a Quarkus extension (with a bunch of build time AOT automatic configuration) and supports native compilation, so you can use it in serverless deployments and regular JVM applications also. The extension consists of multiple layers, most are turned off by default and removed in the build phase if not used.

Entity layer features:

• Use regular @jakarta.transaction.Transactional annotations for an automatic FDB transaction
• The JTA-managed FDB transaction is propagated to forked virtual threads, so you can use Structured Concurrency and do things in parallel inside a single transaction while still using a nice blocking style of programming, no async programming needed.
• Inject a com.apple.foundationdb.Transaction for low level access
• Inject a Persistence object for entity access
• Automatic entity registration at build time
• Retrieval: find (single key get), lazy stream, load (snapshot isolation and transaction splitting)
• Find by exact id, or restrictions on the id
• Find by partial id in case of id is a POJO, make restrictions on the fields of the id
• Find by indexed fields, or restrictions on them. Indexes are POJOs and queries are strongly typed.
• getReference for lazy entity proxies
• Covering index queries (retrieve index POJOs)
• Normal or materialized indices (index size vs n+1 problem)
• Partial indices based on expressions
• Kryo/Smile/Json serialization of entities
• Optional entity compression using LZ4
• Full and automatic GraalVM support

Other layers are built on top of the entity layer and include a distributed micro-matching enabled DAG job processing layer, a distributed, persistent actor layer, a message queue/topic layer and a simple proof-of-concept implementation of Apple’s QuiCK paper. The distributed layers use entities and FDB watches to communicate, coordinate and load balance.

There is an web based admin interface to query/manage the different layers and FoundationDB (UI for entity queries, entity and index level statistics, etc.)

[fuCtor (Alexey Shcherbakov)]

Great job!

But who maintains this layer?
It only has one commit. It looks like some internal solution was published.

[Imperatorx]

Thank you. This is my personal research project, so I’m the sole maintainer. It got to a point where the main features were stable for months so I published it with an initial commit.

I’ll keep it updated to the latest versions of Quarkus LTS releases (coming this September) and fix bugs if found. Bits and pieces of it were/are used in production, it could really accelerate development.

Considered stable, were in production:

• Entity layer
• Distributed primitives (locks, partition allocation, watch processors, etc.)
• Operations layer (job graph, micro batching)
• Time series layer

Were not used in production:

• Messaging layer (queues and topics)
• Actor layer
• QuiCK work queues

[Imperatorx]

The newest stable version of QFoundation was released to Maven Central with over 200 new commits since the initial release.

It includes some notable new features:

• a refactored, stable partitoned Messaging layer, with keys and values for messages (topics, queues, task queues)
• a partitioned Task Queue feature, which combines QuiCK’s pointer requeueing and the Partitioned Queues to provide a fair, partitioned task queue service using message keys as a basis for grouping messages
• Retention based cleanup or Topic compaction based on message keys for topic messages
• a Global KTable-like local rocksdb-backed representation for topics
• explicit watch notification strategies when sending messages to use less writes vs to have lower consumer latency
• Retry logic and error queues for the messaging layer (topic consumers can skip errors, queues and task queues can delete them or send it to the error queue) and the actor layer (actor inboxes)
• Bugfixes and optimizations and tests for the single-cluster QuiCK work queue implementation
• A PartitonedResoruceGroup abstract class to support the easy creation of custom partitioned distributed resources. Instant detection of clean node exits and partition reassignment.
• Observability improvements and dashboard UI improvements
• Improved test coverage for the Messaging module
• As always, all features also function properly when compiled to a native image

Graph of major components:

QuiCK dashboard:

Task queue dashboard:

Topic message retention:

Topic and persistent consumer group dashboard:

Actor dashboard: