Cluster tuning cookbook

[alexmiller]

@KrzysFR

Q1 : What happens if host 6 (from the example above) goes down? Will the cluster_controller role be automatically assigned to another host? But if all the other others nodes either want to be proxy , log or storage , will the cluster_controller role be assigned to one of them anyway? And if host 6 goes back online, will it then take over the cluster_controller role immediately ? Or will it stick to the other host until it goes down, before returning to the preferred host 6?

It’s a preference, but not a requirement. A process with the process class of cluster_controller is viewed as a the best location for a cluster_controller role, but if one isn’t available, then any stateless process will do. If a process with cluster_controller process class joins the cluster, then we’ll notice that there’s a better location, and begin a recovery to re-recruit everything in a better location. (I believe this might not actually be true specifically for cluster_controller, but is true for everything else. You’d be looking for betterMasterExists() in the code.)

@KrzysFR

Q2 : If a process is only assigned CPU oriented roles like proxy , cluster_controller or master , can we safely reduce their memory usage down from 4GB to something like 2GB ?

Depending on the role… maybe, and it’ll depend heavily by role, and partly on your workload. I think that after #432, it’d be safer to reduce the memory on proxies, as long as you set SERVER_MEM_LIMIT accordingly. I don’t think I have specific advice here, you’d more have to look at usage stats and see.

There’s also knobs to reduce the amount of cache that a storage server holds, if you’d like to try reducing memory on storage servers.

@KrzysFR

Q3 : In your example above, you list one or more roles per host, but I assume that they will run more than one fdbserver process. Should we also assign specific roles to each process manually? For exeaple for host 2, should I assign the proxy class to process 4500, and storage class to process 4501? What If I have 4 process per host, but only have 2 roles to assign ?

More than one fdbserver process was my intention, yes. If you don’t specify a role, then it gets the “Unset” role, which is viewed as an okay place to put anything.

Upon further thought, I’m actually not sure that everything that has an “Unset” role wouldn’t be picked up by data distribution to be recruited as a storage server. I’ll try to remember to check this later today.

@KrzysFR

Q4 : When in doubt, is it better to assign more proxy roles to extra process (ex: 4 process on a machine with only two disks, the extra 2 could serve as proxy?), or is there a thing as too many proxies for a certain size of cluster?

GetReadVersion, which is done by a client at the beginning of every transaction, involves a proxy talking to every other proxy to find the maximum committed version. So there’s definitely a point where adding more proxies slows things down, rather than speeding things up.

@KrzysFR

Q5 : is it possible to configure a process to reject a specific role, instead of assigning one? Meaning blacklisting certain roles for some process, but letting it be free to choose any other role? I’m looking at for example banning log or storage class all 3rd or 4th process on each host, if I only have two disks (so that 1st and 2nd process each can have one, but the other 2 process cannot use them).

In general no, but in that specific case, you can assign the stateless role to a process, which then means we prefer not assigning anything that involves durable storage (logs and storage servers) to that process.

… I had actually thought there was a way to start up fdbserver and not point it at a disk as a data location, which would automatically configure it into a stateless process, but I appear to have been wrong.