# Snowflake Entity Layer

**URL:** <https://forums.foundationdb.org/t/snowflake-entity-layer/3327>\
**Category:** Development\
**Created:** [May 17, 2022, 12:51am UTC](https://forums.foundationdb.org/t/snowflake-entity-layer/3327 "2022-05-17T00:51:10Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![rajivr](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/rajivr/32/1100_2.png) [@rajivr](https://forums.foundationdb.org/u/rajivr)\
**Post date:** [May 17, 2022, 12:51am UTC](https://forums.foundationdb.org/t/snowflake-entity-layer/3327/1 "2022-05-17T00:51:10Z")

</div>

I was wondering if there are any publicly available resources (papers, blog posts, videos) that describes Snowflake’s Entity Layer?

The only information that I could find was a mention at [10:22](https://www.youtube.com/watch?v=KkeyjFMmIf8&t=622s) from FDB Summit Presentation.

Snowflake and Apple have the most operational expertise with FoundationDB.

Before embarking on layer development on top of Tokio/Rust bindings, I wanted to make sure I’ve learned as much as possible from existing prior-art.

With Record Layer, its bit easier. 🙂

The areas where I am looking to gain some insight about Snowflake’s Entity Layer are around: Type System, How schema migration is handled and Index Maintenance.

Also I am very open to design level contributions/suggestions that I can get from existing layer developers.

If you think Tokio/Rust might be something that you might adopt in the future and would like to shape the design for your needs as well, please let me know! 🙂

---

<div class="post-metadata">

**Author:** ![Imperatorx](https://avatars.discourse-cdn.com/v4/letter/i/d2c977/32.png) [@Imperatorx](https://forums.foundationdb.org/u/Imperatorx)\
**Post date:** [July 5, 2022, 4:29pm UTC](https://forums.foundationdb.org/t/snowflake-entity-layer/3327/2 "2022-07-05T16:29:54Z")

</div>

I’ve created an Entity layer for Java, similar to JPA. The key features are:

- Every entity has a primary key
- Entities can be queried via primary key or secondary indices.
- It has async (CompletableFuture) and sync access modes
- Entities can be immutable or mutable
- Mutable entities have dirty checking and auto-save at the end of the transaction
- Retrieval can be instant fetch (exact), lazy stream (iterator) or reactive stream (iterator)
- Entities can have references to other entities
- Immediate query, Lazy reference proxies or Eager reference proxies for related entities
- Session cache and second level cache
- No code generation needed
- Indices are Java POJOs too
- Indices can be queried using method references instead of string field names or statically generated classes
- Covering index queries (returns index POJOs)
- Conditional indices (with WHERE clause)
- Multi range queries are prefetched in the background
- Entity management dashboard with live statistics down to index-level access
- Distributed grid computing engine on top of entities with micro batching and job-embedding support (jobs and tasks can form a directed acyclical graph) (e.g. for background micro batching jobs like interest calculation of different account ranges etc.)

The company I work for created a high performance micropayment engine and an IoT time series collection engine on top of this proprietary framework, and can provide commercial support if someone is interested.

Sample usage:

```auto
// Entity and index definition in POJO style:

@IndexDefinition(shape = IxCustomer.class, type = StoreMode.REFERENCE)
public class Account implements Entity<Long> {
  private Long id;
  private Customer customer; // another entity
  private int someOtherField;
  // getters setters constructors omitted
  
 public static class IxCustomer implements Index<Account, Long> {
    private final Customer customer;
    private final int someOtherField
    // getters setters constructors omitted
  }
}

// new entity with related entity
@Transactional
public void create() {
  var customer = orm.getReference(Customer.class, 100); // this is a lazy proxy, no DB query is performed
  var account = new Account(42L, customer, 100); // new entity
  orm.persist(account); // entity gets persisted to memory only
} // At the end of transaction demarcation, the newly persistend entity gets flushed to FDB

// existing entity gets modified
@Transactional
public void update() {
  var account = orm.get(Account.class, 42L); // this is an eager proxy, the statement immediately returns, but an async query is immediately started in the background to fetch the entity
  account.setBalance(200);
} // At the end of transaction demarcation, the entity gets modified, and the account with the balance of 200 gets written to the DB. All changed indices are updated, if needed.

// query by index
@Transactional
public void index() {

  var customerReference = orm.getReference(Customer.class, 42L);

  var accountsForCustomer = orm.query(Account.class)
    .index(Account.IxCustomer.class, qurery -> query
      .field(Account.IxCustomer::getCustomer, c -> c.equal(customer).ascending())
      .field(Account.IxCustomer::getSomeOtherField, f -> f.between(200,400).greaterThan(1000))
    )
    .stream()
    .take(200)
}

```