On the Data Engineering Podcast episode I appeared on recently I received a question I hadn’t really considered very hard.
Is it wise to set up servers where all the processes are configured to be storage processes?
The rationale is if a storage process fails there is no need to go through a recovery. The potential downside is you might require more servers than before to achieve the same level of fault tolerance.
Is this a common configuration? I had never heard it before and I assume someone here has at least tried it at some point if it is not obviously flawed.
In modest to large scale deployments, I think its more efficient to keep special servers separate from storages. An FDB cluster typically scales by adding storage servers, and a small number of TLogs, resolvers, and proxies can service a comparatively large number of storages, so you wouldn’t be saving much hardware.
Besides avoiding recoveries when a storage goes down, you can also optimize your resources by customizing your servers for their role. (more memory for tlog queues, more disk for storages, more CPU for master and cluster controller, etc). Coming from the cloud world, we choose our instance and volume types for each role carefully.
Another concern with having special roles share a process with the storage role is that it could make this storage consistently fall behind the others in its team, since its resources are split.
Good point. Related to the same consideration - I am quite certain that data-distribution will ensure that only a single replica of a given shard is hosted across all SS running on a common host (IP), but I cannot seem to quickly locate the doc or forum-post link to it. Could someone please help me with it?
You are correct - FoundationDB attempts to choose storage teams such that storage servers on the same physical hardware will not hold any of the same shards. One mechanism employed to achieve this is that all processes on the same host (and OS instance) will have the same “machine ID”. However, virtual machines or containers running on the same physical machine might interfere with this mechanism so the machine ID should be set explicitly for each process. See here for more details.
I actually cannot find anything in our documentation that explains exactly this feature, but the link above includes the general statement “FoundationDB takes pains to assure that data replication is protected from hardware-correlated failures” which does cover the case you are asking about.