Scaling down for small data sets

Using FoundationDB
1

Hello all! I have recently surveyed FoundationDB’s documentation. It seems really impressive! I am now considering it as the future backend for a certain application. This post is a request for comments on how to do a “small setup”.

Our application is an asynchronous key management service. It is fundamentally read-only: in the absolute majority of the time, it just operates with keys. Only a few times per year the user “writes” new keys. The implementation currently offers read-only load balancing. When the time comes to renew keys, the user must manually insert the key in every instance.

I am studying options to evolve this application to allow distributed “writes”. Implementing raft + a cluster protocol is one of our options. Doing something like a FoundationBD layer is another. The data set in this application is infinitesimal for modern standards. Persistence frameworks are tuned to terabytes of data; our application is specialized to secure megabytes of key material.

I have searched this forum for “small setup” and “small memory” but found no discussion on this. Is it possible to scale FoundationDB down? Requiring an additional 4 GB of memory per node will multiply our current memory requirements. I appreciate any hint you may have!

2

In order to run with lower memory requirements, my advice (not based on experience running in production) would be to set --cache-memory to something lower than the 2GiB default, and use the ssd storage engine.

Some parts of foundationdb use a custom allocator that trades memory usage for speed, so that could be another opportunity to lower memory usage. There’s no supported way to disable “FastAlloc”, so this would currently require building a patched version from source. My guess is no one would be opposed to some kind of knob that disables FastAlloc, but it’s a bit complicated since FastAlloc is used before main is called so our normal knob mechanism wouldn’t work here.

FWIW it’s convenient that your application has a low write load since high write load can cause significant memory usage, and I can’t think of any way to mitigate that. I think the best available mitigation would be to tweak some knobs so that foundationdb will throttle aggressively to keep memory usage low.

1 Like
3

Hello @pedrolamarao!

Our application is an asynchronous key management service.

I am not sure what “asynchronous” means in your context.

The data set in this application is infinitesimal for modern standards.

Reading the rest of the paragraph, I can tell that is around megabytes of data.

Quoting the documentation:

memory: Maximum resident memory used by the process. The default value is 8GiB. When specified without a unit, MiB is assumed. Setting to 0 means unlimited. This parameter does not change the memory allocation of the program. Rather, it sets a hard limit beyond which the process will kill itself and be restarted. The default value of 8GiB is double the intended memory usage in the default configuration (providing an emergency buffer to deal with memory leaks or similar problems). It is not recommended to decrease the value of this parameter below its default value. It may be increased if you wish to allocate a very large amount of storage engine memory or cache. In particular, when the storage-memory or cache-memory parameters are increased, the memory parameter should be increased by an equal amount.

Configuration — FoundationDB 7.1

If a FoundationDB process reach 8GB of memory usage, it will be killed by FoundationDB, and the associated transactions aborted. The 8GB value does not mean that FoundationDB will allocate at startup time 8GB.

Screenshot of htop filtered on foundationdb that is at rest:

image
image1690×371 96.5 KB

Less than 1GB of RAM are used by ALL process on that machine.

Here is the fdbcli status:

$ fdbcli 
Using cluster file `/etc/foundationdb/fdb.cluster'.

The database is available.

Welcome to the fdbcli. For help, type `help'.
fdb> status

Using cluster file `/etc/foundationdb/fdb.cluster'.

Configuration:
  Redundancy mode        - single
  Storage engine         - memory-2
  Coordinators           - 1
  Usable Regions         - 1

Cluster:
  FoundationDB processes - 1
  Zones                  - 1
  Machines               - 1
  Memory availability    - 8.0 GB per process on machine with least available
  Fault Tolerance        - 0 machines
  Server time            - 07/08/22 12:46:11

Data:
  Replication health     - Healthy
  Moving data            - 0.000 GB
  Sum of key-value sizes - 21 MB
  Disk space used        - 245 MB

Operating space:
  Storage server         - 0.9 GB free on most full server
  Log server             - 1683.7 GB free on most full server

Workload:
  Read rate              - 21 Hz
  Write rate             - 0 Hz
  Transactions started   - 9 Hz
  Transactions committed - 0 Hz
  Conflict rate          - 0 Hz

Backup and DR:
  Running backups        - 0
  Running DRs            - 0

Client time: 07/08/22 12:46:11

fdb>

Canonical is working on dqlite that can help your use case. Tho, after looking around other distributed, safe, secure, and maintained resilient databases, it appears that FoundationDB, whether it is for a single box, large deployment, web-scale deployment and whether you use existing layers, or need only a mere key-value mapping, etc… FoundationDB is a better choice for you, and your employer. The only problem is that there is still no buzz around FoundationDB… It may look like a big “investment”, it is not.

Whether it is FoundationDB or something else, you should benchmark the software (ping @osamarin who was xp with that).

I am very interested in small scale deployment, many people around me think FoundationDB is overkill, and prefer REDIS or PostgreSQL for dubious reasons…

It appears, you need to test the following:

  • Backup and restore: given the size of the data you are working with, my guess is that you can set up your own backup strategy (record the restore process wall-clock time);
  • Cluster resilience: make sure the data is still readable when one process or machine is misbehaving;
  • [edit] setup monitoring !

Please share with us any findings :slight_smile:

1 Like
4

When you get a chance, could you please share the knobs that we might be able change in order to trade latency/throughput for CPU/memory?

I will be curious to see to what extent we might be able to scale down FoundationDB.

If we are able to scale FDB down for metadata use-case (instead of managing etcd, zookeeper or using a RAFT library), that would be really useful for many things. :slight_smile:

1 Like
5

I was thinking of TARGET_BYTES_PER_STORAGE_SERVER, and TARGET_BYTES_PER_TLOG