# Discussion thread for new storage engine ideas **URL:** \ **Category:** Development\ **Created:** [April 20, 2018, 6:11am UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101 "2018-04-20T06:11:56Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![daverosenthal](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/daverosenthal/32/51_2.png) [@daverosenthal](https://forums.foundationdb.org/u/daverosenthal)\ **Post date:** [April 20, 2018, 6:11am UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/1 "2018-04-20T06:11:56Z")
All, one of the most interesting and potentially easy ways to improve FoundationDB is to integrate new storage engines. Storage engines are simple key-value stores that work on a local machine. They don’t need to handle transactions, etc. at all. All you need to do to implement a new one is follow the IKeyValueAbstration, shown here: [[https://github.com/apple/foundationdb/blob/e0c8175f3ccad92c582a3e70e9bcae58fff53633/fdbserver/IKeyValueStore.h](https://github.com/apple/foundationdb/blob/e0c8175f3ccad92c582a3e70e9bcae58fff53633/fdbserver/IKeyValueStore.h)] The only non-standard/tricky part is that erasing a range is expected to be a fast operation, even for large ranges. Currently there are only two storage engines: The main SQLite-based one, and a “toy” one that works only for datasets that fit in memory. Many interesting ones could be built/integrated, for example, a log-structured engine for spinning disks, a better-optimized b-tree one, or even one that directly used some non-disk remote storage. Discuss 🙂 ---
**Author:** ![sumit](https://avatars.discourse-cdn.com/v4/letter/s/59ef9b/32.png) [@sumit](https://forums.foundationdb.org/u/sumit)\ **Post date:** [April 20, 2018, 10:42am UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/2 "2018-04-20T10:42:44Z")
Definitely it would be needed and a big improvisation over existing storage engine, integration with rocksDB is something can cover couple of use cases which would be a big boost . Would like to thought of others . ---
**Author:** ![nutjob](https://avatars.discourse-cdn.com/v4/letter/n/f1d935/32.png) [@nutjob](https://forums.foundationdb.org/u/nutjob)\ **Post date:** [April 20, 2018, 11:21am UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/3 "2018-04-20T11:21:58Z")
LMDB would seem to be a good fit for this as its a lean and efficient KV store, although it does have some idiocracies: > **[LMDB/lmdb](https://github.com/LMDB/lmdb/tree/mdb.master/libraries/liblmdb)** > > Read-only mirror of official repo on openldap.org. Issues and pull requests here are ignored. Use OpenLDAP ITS for issues. - LMDB/lmdb ---
**Author:** ![SteavedHams](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/steavedhams/32/18_2.png) [@SteavedHams](https://forums.foundationdb.org/u/SteavedHams)\ **Post date:** [April 20, 2018, 7:59pm UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/4 "2018-04-20T19:59:39Z")
Here are some design points of SQLite which a new FDB storage engine may want to avoid: - SQLite uses a BTree rather than a B+Tree so it stores full key/value pairs in internal tree nodes, which harms branching factor when using large key/value pairs. - There is no prefix or other type of compression in SQLite. - Large key/value pairs are stored using a linked list of 4k overflow pages exclusive to that record. This adds additional disk latencies when reading large records and can also increase slack space. - SQLite uses a page-based WAL so every 4k page modified in a commit ends up being written in full twice. ---
**Author:** ![abdullin](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/abdullin/32/83_2.png) [@abdullin](https://forums.foundationdb.org/u/abdullin)\ **Post date:** [April 21, 2018, 4:01am UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/5 "2018-04-21T04:01:08Z")
LMDB is a nice piece of engineering that might be a good fit. We’ve used it in production as in-process app storage for years. Key benefits: - Simple design and small codebase - Predictable and consistent performance - Single writer and parallel readers working with a proper ACID transactions - Uses memory-mapped files and lets OS manage the cache (it is different from SQLite in that sense) - Aligns well with FDB concepts (e.g. FDB could be replaced with LMDB while running the application code in a deterministic simulation) However, unlike SQLite, LMDB seems to feature less impressive test suite. Its write amplification factor is also a thing to consider. ---
**Author:** ![wwilson](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/wwilson/32/88_2.png) [@wwilson](https://forums.foundationdb.org/u/wwilson)\ **Post date:** [April 21, 2018, 3:09pm UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/6 "2018-04-21T15:09:05Z")
I think some form of compressing storage engine could be very useful. Prefix compression might be especially handy given that many layers want to represent data hierarchically (and tuples/directories encourage this). The straw-man argument for why this should be low priority is that you could easily implement it in a layer, and it could be done “adaptively” by storing the versioned compression dictionary itself in the database, keeping metadata about what dictionary version(s) different portions of keyspace were written at, and having a background process that periodically re-compresses old data and garbage collects old dictionary versions. You could probably do better by writing something special-purpose for each layer that takes advantage of what it knows about the layer’s data model. But that seems like an optimization, and it’d be nice to have a default besides “no compression”. ---
**Author:** ![dave](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/dave/32/89_2.png) [@dave](https://forums.foundationdb.org/u/dave)\ **Post date:** [April 21, 2018, 3:44pm UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/7 "2018-04-21T15:44:49Z")
When integrating with off the shelf storage engines that do I/O synchronously (including via memory mapped files) I think some effort may be needed to get decent performance. FoundationDB uses a single-threaded asynchronous execution model and blocking that thread to read from disk will be disastrous for throughput and latency. Our adaptation of sqlite uses coroutines to let it use our asynchronous I/O abstraction. That approach won’t work for memory mapped I/O. You could use a real thread pool but I’m not sure how performance will be. A storage engine that doesn’t use our I/O abstractions also won’t be simulatable, so I will be somewhat nervous about correctness. ---
**Author:** ![amirouche](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/amirouche/32/2096_2.png) [@amirouche](https://forums.foundationdb.org/u/amirouche)\ **Post date:** [April 21, 2018, 7:54pm UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/8 "2018-04-21T19:54:37Z")
> [@daverosenthal](#): > > The only non-standard/tricky part is that erasing a range is expected to be a fast operation, even for large ranges. I don’t know about how fast is wiredtiger for that, but I think it’s a good candidate for those not bothered by gpl. ---
**Author:** ![cih.y2k](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/cih.y2k/32/423_2.png) [@cih.y2k](https://forums.foundationdb.org/u/cih.y2k)\ **Post date:** [May 18, 2018, 9:34am UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/9 "2018-05-18T09:34:53Z")
@dave, 1. although not fully support async I/O, but maybe thread pool solution works with this ([https://github.com/leo-yuriev/libmdbx](https://github.com/leo-yuriev/libmdbx)), a much better version of lmdb. And if it works, we can have long read operation. ![perf-slide-1](https://global.discourse-cdn.com/foundationdb/original/1X/e964ecc70c7462000ab2e459e44a79019fd8fe11.png) ![perf-slide-2](https://global.discourse-cdn.com/foundationdb/original/1X/f8719414b11dae78d7251a8f790f99ec586b3758.png) 1. About in-memory storage engine, maybe we can use B+ tree index and mentx engine from tarantool [https://github.com/tarantool/tarantool/blob/2.0/src/box/memtx\_tree.h](https://github.com/tarantool/tarantool/blob/2.0/src/box/memtx_tree.h) [https://github.com/tarantool/tarantool/blob/2.0/src/box/memtx\_engine.h](https://github.com/tarantool/tarantool/blob/2.0/src/box/memtx_engine.h) ---
**Author:** ![ryanworl](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ryanworl/32/440_2.png) [@ryanworl](https://forums.foundationdb.org/u/ryanworl)\ **Post date:** [June 1, 2018, 9:52am UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/10 "2018-06-01T09:52:58Z")
[Apparently Async IO is at least in the prototyping stage for RocksDB.](https://github.com/facebook/rocksdb/projects/1#card-6108940) RocksDB has seen a ton of mileage inside and outside Facebook. ---
**Author:** ![yangruifeng](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/yangruifeng/32/710_2.png) [@yangruifeng](https://forums.foundationdb.org/u/yangruifeng)\ **Post date:** [September 19, 2018, 2:06am UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/11 "2018-09-19T02:06:15Z")
what about the sqlite/lsm，any idea？ ---
**Author:** ![doublemax](https://avatars.discourse-cdn.com/v4/letter/d/8e7dd6/32.png) [@doublemax](https://forums.foundationdb.org/u/doublemax)\ **Post date:** [November 4, 2018, 3:16pm UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/12 "2018-11-04T15:16:14Z")
Range deletes are quick using a LSM, e.g.: > **[facebook/rocksdb](https://github.com/facebook/rocksdb/wiki/Delete-A-Range-Of-Keys)** > > A library that provides an embeddable, persistent key-value store for fast storage. - facebook/rocksdb ---
**Author:** ![davelester](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/davelester/32/1927_2.png) [@davelester](https://forums.foundationdb.org/u/davelester)\ **Post date:** [November 4, 2018, 4:03pm UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/13 "2018-11-04T16:03:46Z")
Related to this topic, one talk at FoundationDB Summit may of particular interest: “[Future of FoundationDB Storage Engines](https://fdbsummit18.sched.com/event/IAky?iframe=no)” ---
**Author:** ![deepankarsharma](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/deepankarsharma/32/422_2.png) [@deepankarsharma](https://forums.foundationdb.org/u/deepankarsharma)\ **Post date:** [November 19, 2018, 10:52pm UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/14 "2018-11-19T22:52:58Z")
Will the talks at the summit be recorded? (Especially this one) ---
**Author:** ![leo-yuriev](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/leo-yuriev/32/421_2.png) [@leo-yuriev](https://forums.foundationdb.org/u/leo-yuriev)\ **Post date:** [November 19, 2018, 11:22pm UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/15 "2018-11-19T23:22:16Z")
I will be glad to answer a questions about libmdbx. ---
**Author:** ![davelester](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/davelester/32/1927_2.png) [@davelester](https://forums.foundationdb.org/u/davelester)\ **Post date:** [November 19, 2018, 11:43pm UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/16 "2018-11-19T23:43:03Z")
Yes, talks will be recorded and published online after the event. ---
**Author:** ![Meai1](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/meai1/32/541_2.png) [@Meai1](https://forums.foundationdb.org/u/Meai1)\ **Post date:** [November 20, 2018, 3:09pm UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/17 "2018-11-20T15:09:56Z")
would be incredible to try this as a backing store: [https://github.com/Microsoft/FASTER](https://github.com/Microsoft/FASTER) ---
**Author:** ![cih.y2k](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/cih.y2k/32/423_2.png) [@cih.y2k](https://forums.foundationdb.org/u/cih.y2k)\ **Post date:** [November 20, 2018, 4:28pm UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/18 "2018-11-20T16:28:18Z")
It seems Faster only support point operations. Not ordered key value data store. ---
**Author:** ![doublemax](https://avatars.discourse-cdn.com/v4/letter/d/8e7dd6/32.png) [@doublemax](https://forums.foundationdb.org/u/doublemax)\ **Post date:** [December 1, 2018, 8:36pm UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/19 "2018-12-01T20:36:18Z")
SQLite has added a LSM backend. [https://www.sqlite.org/cgi/src/dir?ci=9b37bbf5f338dea9&name=ext/lsm1](https://www.sqlite.org/cgi/src/dir?ci=9b37bbf5f338dea9&name=ext/lsm1) FDB is currently running SQLite with BTREE. ---
**Author:** ![aseipp](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/aseipp/32/124_2.png) [@aseipp](https://forums.foundationdb.org/u/aseipp)\ **Post date:** [December 5, 2018, 9:29pm UTC](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101/20 "2018-12-05T21:29:23Z")
This may be a naive question, but one I haven’t seen addressed is: as a user, why would I want a new storage engine using these alternatives (RocksDB, LMDB, etc) instead of the existing ones, and what features would it offer me over using the existing engine? There are a lot of suggestions for adding different key/value engines onto FoundationDB in this thread, but there are no insights as to _why_ they are good additions, or what gaps they fill, or excel at, that the current storage engine does suboptimally at. It is worth noting that FoundationDB 6.1 will apparently feature a new [“Redwood Storage Engine”](https://github.com/apple/foundationdb/pull/819), which will be described in the upcoming Summit 2018 talk [“The Future of FoundationDB Storage Engines”](https://fdbsummit18.sched.com/event/IAky). This has already been merged into `master`. This new storage engine seems like it will have some major distinct advantages that require more changes than just coupling an existing storage engine onto the bottom of FDB. > In addition to higher write throughput, this new design will have a smaller space footprint via key prefix compression and a configurable ability to enable long running read transactions by trading off a reasonable amount of disk space to persist old data versions on disk. This seems like a great advancement and being able to plan my design around long-running read transactions being available is exciting. An interesting question would be what Redwood could further have improved about it, too (including a new low-level storage/tree layer, perhaps) * * * While we’re at it and I’m throwing out naive ideas here, it would be interesting to think about a persistent-memory backend for FoundationDB. Intel is soon going to ship Optane DIMMs with more widespread availability in their new Cascade Lake platforms. `pmem` support is also actively available in QEMU, so it can be programmed against, just not with realistic performance numbers. But having a storage engine designed for highly-dense persistent DIMMs (~7TB with lower latency than NVMe, per machine) would be very interesting to experiment with, and Optane seems like it could be very attractive for a lot of workloads. [Next page](https://forums.foundationdb.org/t/discussion-thread-for-new-storage-engine-ideas/101.md?page=2)