[Side note] Could we have some additional tags for cloud and/or aws? I feel like the fact that you’re running on K8s in the cloud, and what cloud provider you are using and thus what disk types are available, will have a big effect on the answers to a lot of my potential questions…
We’re looking at setting up a FoundationDB cluster in EKS on AWS. I think our initial back-of-envelope calcs were that after 5 years we’d expect somewhere around 500m keys and 2TB data or so.
I’m doing some load tests to check what sort of sustained throughput we can get, and I also want to be able to load a much larger amount of data into the cluster (our 5 yr estimated level) in a ‘reasonably short’ space of time (like, if it took a day or two for the initial load that’d be fine), so that we can test backup/restore and see how long that would likely take (we’re aiming for a couple of hours).
I’m getting the sort of numbers I want for normal throughput (only a couple hundred msgs or tx per sec), but if I try and sustain those for a significant period of time (say, by trying to write 1m k/v pairs to the cluster) the cluster goes unavailable, and fdbcli reports that none of the coordinators are reachable. On one occasion I left the cluster alone for ‘a while’ (a few hours) while doing something else and when I checked back it had come back with the coordinators reachable (albeit different IPs, it seemed to have moved all 3 coordinators) and data still present. On two other occasions I tried leaving it alone overnight and it was still broken.
At the moment I’m running a cluster with 3 storage nodes, which are also being selected as the coordinators. What I see when I started a load test is that the log node IOPS goes to 100% (according to status details) first, then the storage node IOPS climb and the log ones drop off a bit. status json | fdbtop shows ‘100%’ IOPS on the log nodes to be ~300, and on the storage nodes to be ~3k. I’ve yet to investigate increasing the number of storage or log nodes.
First question - How does FoundationDB calculate the ‘% of max’ for IOPS? I know I can look up the details of the gp2 EBS volumes that are being mounted into those containers, where IOPS is based on a multipler of disk size (up to a max limit), but as far as I can tell FDB has no knowledge that it’s running in AWS, its data volumes are EBS etc, so where does it get the max to calculate a % from?
Second question - I’m currently working on the assumption that the IOPS starvation is the main thing that’s causing issues, because I’m not seeing CPU or memory limils being hit. Should I be expecting things to work OK on gp2? If so, do I just need to configure bigger disks (for more IOPS) or more nodes (to spread the load), or has the default K8s operator setup done something really non-optimal with regards performance? Or should I really be looking at setting up gp3 (allows configurable IOPS and much higher max values for IOPS and throughput than gp2) on EKS to get ‘reasonable’ performance out of FDB?
Sorry if these are n00b questions and I’ve just failed to find the right bit of the docs for them. If anyone wants to just point me at existing documentation articles that help answer my questions, that’s fine too!