Continuing the discussion from here CPU limited storage processes, unfortunately I still have several problems with my FoundationDB cluster.
The fdb cluster worked perfectly until at some point 3-5 days after the cluster was deployed and scaled up, one of the storage processes got progressively increasing data lag (now at ~1h) from which it never recovered.
After several hours, another storage process seems to have sustained high CPU load without the data lag
Disk IO wise the cluster seems healthy, with storage processes having about 30-40% load and the logs having about 60-80% load
The CPU load on the storage processes is 50-85% with 2 processes stuck at 85%
Because of the increased moving data in flight and moving data in queue metrics 1GB and 20GB respectively, I added two process knobs in order to speed up the repartitioning, which so far hasn’t made any noticeable difference:
Is it possible that the write load is saturating one of the SS just above the edge and causing it to fall behind?
While high cpu% could indicate that the SS is not getting free cycles to perform the writes, but at 85% cpu, there should be enough remaining for SS to not be starved.
How does the disk io activity look like (iostat -x 2)? It might be good to rule out the disk saturation (use fio to benchmark disk iops) before going to other aspects like read/write hotspots and cpu saturation.
Disk io looks fine. But is this storage server constantly writing more than 100MB/s and still behind? That seems like a lot of write throughput. Are the other healthy SS doing similar amount of writes?
Can you check the iostat output over longer duration?
I left iostat running for 10 min on the lagging storage server and the results were similar with around 95-120 MB/s writes. I then checked one of the storage servers with the least amount of CPU load at the time and the results were lower on average, somewhere around 80MB/s, with a range from 60 to 140MB/s writes. If I remember correctly in another thread, it was mentioned that fdb had an artificial IOPS limit in order not to saturate SATA SSDs, maybe this limit does interfere?
AFAIK, FDB does not apply any artificial limit to throttle down the iops.
If there is any throttling, it could be due to the disk itself - from the specs you’ve attached, there seems to be a limit of sustained WIOPS of 4680. It could be that on the lagging SS, it may just be going over the limits.
In any case, this seems like a lot of write per server. Are you really doing so much logical writes (aggregated across all SS), from application point of view?
You seem to be running a very write-heavy workload. As the disk doesn’t seem to be saturated, my guess is that the write-queue within FDB is the problem.
The disk interface uses AIO with O_DIRECT, and we only allow to queue 64 operations at a time and reads will get a higher priority. This means that the max throughput is somewhere around 4KB * 64 * disk-latency - or in other words: your performance might be limited by disk latency instead of disk throughput…
There are a few things you can do:
If you have enough CPUs, I would suggest to start more storage servers per disk. You need one CPU core per disk - you could also try to oversubscribe and have two storages run on one CPU, but that might give you weird read-latency behavior, which is also weird.
There’s a knob called MAX_OUTSTANDING which controls how many operations fdb sends to the storage. You can try to set this to a higher value.
I think the first suggestion has a higher probability for success, so if you can I would try this first.
@v-pap can you post the iostat output over a longer duration? The field avgqu-sz, which measures the outstanding queue-depth on the disk, seems quite low (and does not look realistic to support such high iops being observed).
Unfortunately, it seems like the 1st option wont work in my case, since I am deploying this cluster with the Kubernetes operator which doesn’t seem to support this functionality yet. But I will try the 2nd option.
There are 8 SS. With each SS doing 100 MB/s, this amounts to 3TB/hr. But the posted key-Val size is nowhere near that. Can you think what could explain this discrepancy? How large is each key-val?
You are comparing disk writes to FDB key/value writes. They are very different, and this difference is often referred to as “write amplification”. FDB deals with KV pairs, but those KV pairs are stored in blocks on disk, and to update or insert a KV pair means to write the entire block it resides in or will reside in based on sorted order.
In the worst case, random writes of small values, each FDB kv pair written requires one 4KB disk block to be updated on disk. In the storage engine, on order to update that block atomically with other blocks in the commit, there are actually two total 4KB writes that must happen to change that single record, and then there are other writes that can happen was well for internal record keeping and slack space management.
To put some numbers on a worst-case scenario, writing 16 byte keys with 8 byte values where the keys are uniformly random across the key space and total 1 megabyte worth of 24-byte KV pairs per second would result in about 350MB of actual disk writes per second on the storage servers with the ssd storage engine (or basically any other btree with 4k pages). In triple replication, that number triples as well.
Agreed. I was trying to allude to this possibility of a very high write amplification factor due to many small-sized kv writes spread randomly around. I did miss to factor in replication factor.
If this is the limiting problem, then a change in data model will be needed at application side.
I think I have mentioned my data model already in another thread but I will mention it again just in case you have any idea or recommendation on how to improve it. Each monitored resource has its own directory path: monitored_resource_id/resolution where resource_id is its own unique id and resolution is a string: second, minute, hour or day. Under these directories exist key-value pairs of the form key: (metric, timeinfo) with value: a 64bit int float etc. Of course for other resolutions other than second, I use atomic operations as a way to calculate aggregations on the fly. So each metric is essentially written one time in the seconds resolution and for each other resolution 4 values are updated: 2 counters (count, sum) and 2 other values for max and min (all with atomic operations), in total 1 + 3*4 = 13 writes/counter updates. Each write transaction to the fdb is essentially scraped metric data coming every 5 seconds for each monitored resource. So each write transaction contains around 200 metrics which are to be written under the directory path of the appropriate resource. I guess that these inserts as the data is growing are getting further apart since more key-values are inserted between metrics. I had thought of changing the key to something like (timeinfo, metric) so inserts would possibly require less disk writes, but in this case querying specific metrics would get very expensive.
This is an interesting modeling problem. As a quick modification, is it possible to buffer 5-sec values in memory for a minute, before writing them to fdb. You can probably create one transaction per resource/resolution/metric to write all the raw/derived values.
While this may put more complexity and ram requirement on your layer, you will know if it helps in reducing your write load by ~10x due to the locality of kvs per transaction.
Just curious what really is the amount of data in bytes/sec you are writing, as measured by the application layer? It will be interesting to estimate current write application factor.
That would be an interesting approach but I am afraid at least for my case this would require something like a distributed memory system, since there are many writer processes which may have different bundles of metrics etc.
As far as bytes/sec I don’t have any numbers at the moment but I will try to measure it.
I was suggesting that you try out the buffering approach as a test only in some kind of test workload to see if it helps. That may give some pointers for future.
I have a three-node fdb cluster. I got a problem with growing the data lag of one fdb process.
I see that at first the durability lag start growing (probably because the i/o subsystem had been saturated). Then, a dozen of minutes later, the data lag started growing too.
How to troubleshoot the reason of growing data lag? Does the high durability lag influence the data lag? How does it? Are they any parameters that control this relation?
May be the TLog process itself instead of the SS process responsible for the high data lag?
knob_max_outstanding controls the maximum number of outstanding operations. knob_min_submit controls the number of IO operations below the max outstanding that you should reach before you submit more IO operations (due to io_submit latency).
