I’m interested in the use and distribution of FoundationDB as a datastore within another piece of software. Anyone out there with ideas or tips for this use case? I’d like to shield users from having to install/setup their own FoundationDB instance as a separate step. Instead, when you install “Software X” a simple single-node FBD instance is created, and used by SoftwareX as its embedded data store (like e.g. SQLite). Obviously, the idea is that if and when the user needed to scale the datastore for SoftwareX they could create and migrate to an actual cluster, etc.
You should look into Deno and it’s KV store. The approach that they took was creating a local implementation of their distributed KV store using SQLite. If you want to use it at scale, then you can use their distributed implementation (same API) which runs on top of FDB.
Including a single-node FDB within your application seems like a bad idea from an efficiency perspective. A lot of the complexity of FDB should be due to it’s distributed nature, and this complexity would essentially be a waste when you’re running a single node.
What is the target: What OS? Do you ship a VM? Container? Or install script? Is it installed on a desktop computer? Or a remote server?
@janderland Thanks for the pointer. Nice. As they say, “Go from a side project to an enterprise platform…” FWIW, I’m not too concerned about the efficiency/complexity of FDB since as single node it kind of just works as bypass to the SQLite-based b-tree. The biggest issue is that you need different binaries for different platforms and you have to have a server service which is kind of awkward to manage compared to e.g. a native library. I’ll need to look a bit closer at DenoKV to see how much simpler/lighter than FDB it is to deploy and manage.
@amirouche You’re asking the right questions. The context here is looking to build a datastore into another software tool targeted at developers. I’m looking to make that new software have very low friction to get started with. So, installation via something like pip, or maybe just the “curl *** | sh” strategy. But, yes, a user could be on their MacBook just playing around, or on a linux server. The idea is that it would be easy to get going and then you could move that over to cloud infra at some point and scale the datastore as necessary.
It would be cool if there was a “local library” version of FDB that just looked like the normal client (so instant compatibility with all language bindings) but then just worked against a local file instead of talking to the server.
There’s also LevelDB which is embedded by design and has similar ordered key-value semantics. I have successfully used FDB Tuple Layer on top of LevelDB.
The lowest friction is to do what denokv does, implementing the set FoundationDB functions you rely on top of SQLite. SQLite is easier to ship. Otherwise, you can look into desktop containers.
I did the exact same thing with pebbledb which I personally recommend over leveldb. GitHub - cockroachdb/pebble: RocksDB/LevelDB inspired key-value database in Go
The main issue with both leveldb and pebbledb in emulating fdb is that last I looked there are no transactions. However, I solved that by using a read/write mutex to serialize transactions and batch writing any changes.