Smaller memory allocations for stateless pods?

Using FoundationDB Running FoundationDB
1

We’re building out a new FoundationDB cluster, and I’m thinking about resource allocations after an initial load test. We’re using the FDB Kubernetes Operator and previously ran a load test that handled 100% of our write traffic for 24 hours (hooray!). We used the default resource allocations of 1 CPU and 8 GiB of memory for all fdbserver containers.

In that load test, we observed:

  1. Storage processes are memory-hungry, and we’d like to throw some more memory at them to see if that yields any measurable improvements
  2. Log processes never seemed to use more than 4 GiB of memory, even under heavy sustained load
  3. Stateless processes never seemed to use more than 0.5 GiB, even under heavy sustained load

I’ve seen a variety of bits of advice here and in the docs. I regret that I don’t have a comprehensive set of citations, but my summary at this point is that 8 GiB is, indeed, the recommended minimum these days. More recently, though, I noticed this bit of advice:

As a general rule, I think setting memory to (1.5 * cache_memory + 4GB) would be a stable configuration.

My question: since stateless processes don’t use cache_memory at all (right?), is it safe to reduce their memory setting to 4 GiB, or does it make more sense to keep them at 8 GiB for reasons that didn’t become evident in our (admittedly short!) load test?

For context, the motivation for reducing the memory allocation for stateless processes would be to make room to increase the allocation for storage (and log) processes. We’re currently using machines with 4 vCPU, 32 GiB of memory, and one SSD. After subtracting a little overhead for background processes/sidecars, I think we can fit three fdbserver processes (we have three uncontested cores and one core that’s divided between those auxiliary processes) and divide ~30 GiB of memory between them. Since there’s just one disk, I think our most common configuration will be one storage process with two stateless processes. The more we can shrink the stateless processes’ memory requierment, the more we can add memory to the storage process.

Thanks!

1 Like