Flow is more of a stateful distributed system framework than an asynchronous library. It takes a number of highly opinionated stances on how the overall distributed system should be written, and isn’t trying to be a widely reusable building block, like ASIO. It is built out of a number of components, which we can pick apart instead:
The actor compiler, in today’s modern world, is essentially a backport of c++17 resumable functions to c++98. It breaks functions apart at suspend points as a source-to-source transformation. You get to write stright line, normal-looking code, and brings you the wonders of async / await that a number of other language camps have found preferable over callback style code. It’s worth noting that what Flow refers to as actors aren’t actually actors. There is no Erlang or Akka-style actor system in Flow. An ACTOR could be an infinite loop that listens for messages, which resembles something similar, but there’s no concept of behaviors. The majority of ACTOR functions are just ones that need to suspend at some point during their computation to wait for network, io, a time delay, or another actor function to finish.
Using the futures-based async/await framework, Flow provides an evented run loop, which uses boost’s ASIO. This choice of a run loop is not an important detail to flow’s implementation. I suspect that it would could be replaced by libuv, to trim some dependencies, or DPDK’s runloop, if one wished to explore high performance networking. The important part, is that Flow controls exactly which completed future is run, and in which order. This provides the control over scheduling that is required for deterministic simulation.
Flow also provides a number of smaller libraries that all provide asynchronous and future-ified APIs for interacting with the host system. All file IO, network IO, logging, system metrics gathering, etc., must be done through these libraries. Also transparently provided are simulated/faked implementations of each interface for when a process is being run in simulation mode.
However, programs will not directly use an asychronous network interface. Instead, Flow offers an RPC framework. An Interface defines a collection of PromiseStream<> s, each one representing an RPC method. An actor is on the other side, waiting on the matching set of FutureStream<> s. Sending an RPC is using a PromiseStream to send a completed Future to the receiver, where the value is the request. Part of the request is a ReplyPromise that the sender waits on its corresponding future for the reply. This models RPCs as just an extension of the future framework, and thus the caller is unaware and uncaring if the destination is local to the same process, or far away across the network. This provides the location agnosticism that is required to collapse all actors into one process for simulated testing.
This now gives us the pieces required to build deterministic simulation, but we need to build a bit more on top of it to gain the full benefit…
In addition to the more commonly discussed master, proxy, resolver, etc., roles that are in FDB, there also exist testers. Testers run workloads, which is roughly just a class that has a start() , run() , and verify() . This is used to build tests that perform specific, composable functions. Killing simulated machines is a workload. Verifying that the final state of a series of operations done to a simulated FDB matches that of an in-memory map is a workload. Fuzzing the client API, and asserting that any errors (or lack of errors) that occur are expected in the client API contract/model is a workload. Running concurrently with all of these is a comprehensive assortment of fault injection built into all of the simulated variants of file, network, etc. libraries that programs are required to use. This can also be used to test upgrades of FDB. An older version of FDB is run, and then killed at a predefined moment. A newer version is then started, and the previously run workloads are used to ensure that the resulting cluster is both available and correct.
Built into FDB itself is an extensive amount of code that proactively tries to make dangerous and hostile decisions, using the BUGGIFY macro. For a given test seed, a random assortment of BUGGIFY lines are enabled, and then each line will trigger 5% of the time. This means that FDB is proactively helping fault injection and simulated nemesis workloads to find correctness issues in FDB, by trying to make it continuously standing on the brink of disaster already.
So viewing Flow and Boost.ASIO as equivelents undersells what Flow was built to provide. However, it’s definitely not a silver bullet either. There hasn’t really been any special effort paid to trying to allow Flow to be easily broken out and reusable elsewhere. (Although the Document Layer might be a decent example of doing so.) I also think Flow is coming due for some overhauls. I keep on eyeing entirely replacing the actor compiler with C++'s resumable functions. A lot of the filesystem code could use some cleanups on all platforms. I’m not sure anyone is overly cheerful about the state of TLS. All that said, I don’t think anyone would be resistant either if one wished to submit changes to make it easier to separate Flow into its own reusable project. 