# Success Story: FoundationDB at SkuVault **URL:** \ **Category:** Community\ **Created:** [May 4, 2018, 3:01pm UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336 "2018-05-04T15:01:40Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![abdullin](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/abdullin/32/83_2.png) [@abdullin](https://forums.foundationdb.org/u/abdullin)\ **Post date:** [May 4, 2018, 3:01pm UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/1 "2018-05-04T15:01:40Z")
I’d like to share a success story of FoundationDB at SkuVault, where it is used as a multi-model database for event sourcing, queues, docs and metadata: [blog post with more details](https://abdullin.com/sku-vault/foundationdb-layers/). FoundationDB is running in production on Azure cloud for more than a year. Despite all the network glitches and reboots, it is the most reliable component in the distributed system. System administrators particularly love FDB - unlike the other databases, it never requires any babysitting or time-consuming maintenance. If you have any questions, please don’t hesitate to ask! 🙂 ---
**Author:** ![amirouche](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/amirouche/32/2096_2.png) [@amirouche](https://forums.foundationdb.org/u/amirouche)\ **Post date:** [May 4, 2018, 3:18pm UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/2 "2018-05-04T15:18:38Z")
- How do you deploy/update the fleet of fdb machines? - How do you handle schema changes / migrations? About lmdb you say: > Embrace the synergy between LMDB and FoundationDB, making sure that internal tools (debugging, dumps, REPL, code generation DSL) can target both from the start. Did you develop an abstraction layer on top of lmdb that mimics foundationdb API to be able to use the same “transaction” function on different backends. Basically, how do you swap between lmdb and fdb. Tx in advance! ---
**Author:** ![abdullin](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/abdullin/32/83_2.png) [@abdullin](https://forums.foundationdb.org/u/abdullin)\ **Post date:** [May 4, 2018, 4:58pm UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/3 "2018-05-04T16:58:27Z")
> [@amirouche](#): > > How do you deploy/update the fleet of fdb machines? We deployed a cluster manually to a set of pre-configured virtual machines. The production deployment started in the beginning of 2017 (before FDB was open-sourced), so there were no updates to expect. > [@amirouche](#): > > How do you handle schema changes / migrations? FoundationDB was essentially used as a distributed commit log and for intra-cluster coordination. These aspects don’t go through a lot of schema changes. Most of changes where within the node state (stored in LMDB). > [@amirouche](#): > > Did you develop an abstraction layer on top of lmdb that mimics foundationdb API to be able to use the same “transaction” function on different backends. Basically, how do you swap between lmdb and fdb. The answer is two-fold: 1. Node storage was in LMDB and it was highly optimised for in-process access. There was no point in supporting swapping between LMDB and FDB. So FDB toolset was used to develop against the LMDB directly (“toolset”: tuples, key-ranges and modelling techniques). 2. Cluster interaction layer (everything that ran on FoundationDB) was indeed developed on top of the abstraction. FDB implementation was plugged for the cloud deployments (PROD, QA, STAGE etc), while LMDB implementation was used for local development, demo mode and some tests. Does this help? ---
**Author:** ![amirouche](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/amirouche/32/2096_2.png) [@amirouche](https://forums.foundationdb.org/u/amirouche)\ **Post date:** [May 4, 2018, 5:34pm UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/4 "2018-05-04T17:34:02Z")
> [@abdullin](#): > > The answer is two-fold: > > Node storage was in LMDB and it was highly optimised for in-process access. There was no point in supporting swapping between LMDB and FDB. So FDB toolset was used to develop against the LMDB directly (“toolset”: tuples, key-ranges and modelling techniques). You use lmdb instead of sqlite as backend storage? AND you use lmdb in development environment without using the official client library? As it seems like, I don’t understad where lmdb is placed in your codebase. Is it only a dev dependency? ---
**Author:** ![abdullin](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/abdullin/32/83_2.png) [@abdullin](https://forums.foundationdb.org/u/abdullin)\ **Post date:** [May 4, 2018, 6:00pm UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/5 "2018-05-04T18:00:47Z")
I apologise for the confusion. By “_node storage_” I’ve meant “_application node storage_”. In other words, application nodes (containing business logic) persist all the relevant state inside an optimised LMDB database (which is sourced from events). These nodes communicate to each other using the various layers on top of the FoundationDB. The most important of these layers is the _Commit Log_. You can learn more about the overall design by reading through the older posts in the [SkuVault story](https://abdullin.com/sku-vault/). In particular, I’d recommend checking out [System Architecture Overview](https://abdullin.com/sku-vault/2017-07-15-high-availability-and-performance/) and [LMDB and DSL for it](https://abdullin.com/sku-vault/2017-01-22-dsl-impression/). ---
**Author:** ![amirouche](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/amirouche/32/2096_2.png) [@amirouche](https://forums.foundationdb.org/u/amirouche)\ **Post date:** [May 4, 2018, 7:13pm UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/6 "2018-05-04T19:13:12Z")
> [@abdullin](#): > > I apologise for the confusion. By “node storage” I’ve meant “application node storage”. No worries. It seems you use foundationdb as en event log like a single source of thruth and create images of consolidated data for particular service in lmdb. Is that correct? ---
**Author:** ![abdullin](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/abdullin/32/83_2.png) [@abdullin](https://forums.foundationdb.org/u/abdullin)\ **Post date:** [May 4, 2018, 7:36pm UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/7 "2018-05-04T19:36:49Z")
Correct. In other words, LMDB can be used to create various data representations (tables, indexes, graphs etc) that are populated by replaying events from the log and then used to serve write/read requests by the application logic. ---
**Author:** ![amirouche](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/amirouche/32/2096_2.png) [@amirouche](https://forums.foundationdb.org/u/amirouche)\ **Post date:** [May 4, 2018, 8:03pm UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/8 "2018-05-04T20:03:03Z")
> [@abdullin](#): > > In other words, LMDB can be used to create various data representations (tables, indexes, graphs etc) that are populated by replaying events from the log and then used to serve write/read requests by the application logic. Awesome, I did not think about that. My database does write 4 tuples (quad tuple or RDF) in a tree like history like in git. It’s branche-able database, I will put your feedback for good. ---
**Author:** ![panghy](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/panghy/32/19_2.png) [@panghy](https://forums.foundationdb.org/u/panghy)\ **Post date:** [May 5, 2018, 5:33am UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/9 "2018-05-05T05:33:36Z")
Wow, you guys managed to negotiate a license a year ago with Apple? We always thought it was closed source and not available for licensing since the acquisition and before the open-sourcing. ---
**Author:** ![abdullin](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/abdullin/32/83_2.png) [@abdullin](https://forums.foundationdb.org/u/abdullin)\ **Post date:** [May 5, 2018, 6:26am UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/10 "2018-05-05T06:26:13Z")
No, Apple would never do that. We used FoundationDB under the community license which still worked after Apple bought the company and the database. ---
**Author:** ![panghy](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/panghy/32/19_2.png) [@panghy](https://forums.foundationdb.org/u/panghy)\ **Post date:** [May 5, 2018, 11:03pm UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/11 "2018-05-05T23:03:45Z")
I see, I guess as long as you run really small production clusters, it’s still covered. That’s still a pretty gutsy call (to me), we had the source code for the last 4 years and we thought we ultimately would replace it with something else (we were in the process of that exploration when Ben reached out =p). You guys actually _started_ using it a year ago. ---
**Author:** ![abdullin](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/abdullin/32/83_2.png) [@abdullin](https://forums.foundationdb.org/u/abdullin)\ **Post date:** [May 6, 2018, 7:02am UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/12 "2018-05-06T07:02:02Z")
On that project it was a small calculated bet on the outstanding level of engineering employed by the FonudationDB team. Besides, no other database even came close in terms of the required features, performance and survivability. We had no other choice :] ---
**Author:** ![notxcain](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/notxcain/32/213_2.png) [@notxcain](https://forums.foundationdb.org/u/notxcain)\ **Post date:** [May 8, 2018, 9:00am UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/13 "2018-05-08T09:00:03Z")
Rinat, are these layers deployed as a separate services or it’s just a library? How do you handle authorization and authentication? ---
**Author:** ![abdullin](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/abdullin/32/83_2.png) [@abdullin](https://forums.foundationdb.org/u/abdullin)\ **Post date:** [May 8, 2018, 9:16am UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/14 "2018-05-08T09:16:23Z")
Denis, these layers were just an infrastructure library, available to the application logic. Application logic was responsible for the authorisation an authentication. Potential approach for strengthening the security at the app level is be to encrypt all messages with tenant-specific keys. This way FDB would be handed data that is already encrypted. ---
**Author:** ![notxcain](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/notxcain/32/213_2.png) [@notxcain](https://forums.foundationdb.org/u/notxcain)\ **Post date:** [May 8, 2018, 9:28am UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/15 "2018-05-08T09:28:29Z")
Regarding auth, I was talking about service -\> db auth, not application logic auth. Just to be sure that no other (micro-)service accidentally writes to (or deletes) a keyrange owned by another service. ---
**Author:** ![abdullin](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/abdullin/32/83_2.png) [@abdullin](https://forums.foundationdb.org/u/abdullin)\ **Post date:** [May 8, 2018, 9:38am UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/16 "2018-05-08T09:38:10Z")
No, there was nothing like that. Keyranges were protected by the layer logic (nodes could access FDB only through them, no raw access) and some extra simulation testing on top. ---
**Author:** ![notxcain](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/notxcain/32/213_2.png) [@notxcain](https://forums.foundationdb.org/u/notxcain)\ **Post date:** [May 8, 2018, 9:40am UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/17 "2018-05-08T09:40:33Z")
Okay, I got it. Raw FDB access is incapsulated by the layer. Thanks! ---
**Author:** ![abdullin](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/abdullin/32/83_2.png) [@abdullin](https://forums.foundationdb.org/u/abdullin)\ **Post date:** [May 20, 2018, 7:05pm UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/18 "2018-05-20T19:05:24Z")
Exactly. Also one of the great things of using layers with FoundationDB - you can **conveniently compose multiple layer calls in a single atomic transaction**. For example: mark job as processed, publish events and update some metadata. If something goes wrong, the entire transaction would roll back. ---
**Author:** ![bowlofstew](https://avatars.discourse-cdn.com/v4/letter/b/aca169/32.png) [@bowlofstew](https://forums.foundationdb.org/u/bowlofstew)\ **Post date:** [October 28, 2018, 10:27pm UTC](https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336/19 "2018-10-28T22:27:52Z")
Thanks for the write-ups on this and all of the information on your implementation!