Constrained RAM in an application development environment

Maybe this can be useful to someone.

Here is the actual memory consumption of each process running on a test cluster running 5 process per host (16 GB RAM, ubuntu linux, ssd engine), and without any changes to the settings or knobs! The data are comming straight out of the status json metrics.

The statistics are right after big import job, and after the cluster has settled down to idle, so all the storage nodes have had the opportunity to fill their cache with hot data.

You mileage may vary, values could be different under a different workload, don’t sue me!

Processes grouped by roles:

Storage processes tend to average 3.7 - 4.0 GB, Log processes are around 1 GB, Resolver is ~0.7 GB and the other stateless processes (master, proxy, controller) use 200 - 300 MB RAM

Here is another view, showing the usage per host: looks like only 11.5 - 12 GB of RAM are enough to fit 1 log + 3 storage + 1 stateless.

4 Likes