Production deployment

[alloc (Alec Grieser)]

@ThomasJ

As each FDB instance is tied up to a CPU, it also has a different port. So to expose all of them to each other, do you put a proxy in front of it? So lets say I have 8 FDBs per server starting at the port 4500, then do I put a proxy on lets say 3333 in front of them and round-robin distribute the load on the server? Or is this somehow done on the client like it’s with aerospike and couple of others. Additionally, as I have 5 servers, do I again load balance the traffic evenly between all the servers through a proxy or is this done again on a client?

As @brk0v mentions, all of this kind of load balancing is done through the client in a way that is transparent to the user. In particular, the way this works is that each process in the cluster uses the cluster file as a discovery mechanism. Essentially, all of the servers get registered by a central authority (discovered through the cluster file) and given roles. Then the clients talk to that same central authority (directly or indirectly) and then can do things like load balance their subsequent requests amongst the cluster.

My guess is that because of how the networking works in fdb, adding a proxy between the client and servers would probably confuse the client. For example, each client reads keys be reading directly from a storage server that has the key it needs which it gets the address of through this registry information. If the IP and port registered don’t match the required one (because it’s through a load balancing proxy), you won’t be able to reach that server.

@ThomasJ

What are the best practices when deploying and managing this kind of cluster? I haven’t found any extensive provision scripts (ansible or others), so trying to figure out what’s the best way to do so, especially as the cluster is populated through the cluster file. Do you produce it manually before the deployment and then just distribute it everywhere, or?

Creating a cluster file manually and then distributing it out sounds reasonable to me as a cluster file distribution scheme. There’s more on the format of this file in the Administration Guide. You could spin one up and then copy it over, but I suspect that that will end up being more hassle than it’s worth in the long run (though maybe it’s fine for an initial cluster).

@ThomasJ

And finally, what’s the best structure of the process class in this kind of cluster?

I’m not sure I totally understand this question, but are you asking how to configure the process classes of each fdbserver process? I think the most helpful thing that may not be obvious from the docs is that the transaction log processes use way more disk I/O than the storage server processes, so for best performance, you will probably want to make sure that those processes are using a different disk (if possible, a better, faster disk) than the storage servers. If you’re looking for the number of each roll to include (e.g., logs=4, proxies=5, etc.), then that’s harder for me to give you more general advice, but that will depend on the number of machines in your cluster. Are you at 5 servers with 1 disk each? You might need to run an additional process per server of the “stateless” class (they won’t all get used). Also, it might be good if you can run at least three transaction logs and three storage servers (so that you can loose a disk from either one and still have two of each, the required number for double redundancy). So 6 disks (three large for the storage and three small for the transaction logs, if possible) would probably be my advice.