Segmentation fault error and broken cluster

[brk0v (Viacheslav Biriukov)]

@ajbeamon

What about the disk IO errors? I’d like to get an idea what the IO errors were to form a more complete picture of what happened.

Now I can see only “No space left on device” errors. But I think, it’s not a cause. My data set is about 40GB, as I mentioned before.

<Event Severity="10" Time="1528154524.618570" Type="AsyncFileKAIOOpenFailed" Machine="129.168.1.100:4500" ID="0000000000000000" Filename="/var/lib/foundationdb/data/4500/logqueue-8b2b67814ccc537bc3635f6bae188ff4-0.fdq" Flags="f0006" OSFlags="4242" mode="02600" Error="io_error" ErrorDescription="Disk i/o operation failed" ErrorCode="1510" UnixErrorCode="1c" UnixError="No space left on device" logGroup="default"/>

@ajbeamon

Are there any Severity="40" events in this process’s logs?

I have logs with Severity="40" but they all after above space issues.

I’m going to update setting for holding more logs and restart my load process to reproduce the issue. I’ve done it before with 5.1 version.

@ajbeamon

… is it having disk issues

I don’t see any errors in logs and dmesg.

@ajbeamon

No, reads are done from the storage servers. The storage servers get their data from the logs, and typically this data is served straight out of memory. If we have to spill some data from memory onto disk (into this B-tree for the ssd storage engine), then the log will serve the data out of the B-tree instea

Mmm… Sorry for architecture question, I’ve read https://github.com/apple/foundationdb/blob/master/documentation/sphinx/source/kv-architecture.rst, but still want to clarify commit path and your comment.

So with a slow disk and/or high write rate, I can get in a situation, where I successfully set a key/value pair into a saturated cluster. But can’t get key from the storage servers in the next transaction? Or the first transaction returns no errors only if the storage engines already got all changes?

@ajbeamon

There is a proposed feature that will make it easier to set up what we call batch workloads to run at higher rates. These run at a lower priority and the goal is to make them automatically shut off at a point lower than saturation. See Batch priority enhancements · Issue #377 · apple/foundationdb · GitHub.

Interesting, thank you for the link.

@ajbeamon

At any rate, as I mentioned, this is not the intended behavior.

As I mentioned before, I’ll achieve more logs during next try of load.

@ajbeamon

It is not required to have storage servers on every node. By default, a cluster has a number of logs equal to its replication, so in double it would have 2 logs. If you are going to go the route of configuring some processes to act specifically as logs, then I would have at least 3 processes configured as transaction class so that you have a backup if one fails. In triple replication, I would have 5 (3 + 2 backup). It’s also possible to configure your cluster to run more logs to support a higher write throughput (e.g. using configure logs=8 in fdbcli), in which case you would want to have that many transaction class processes, plus some backups.

So if I configure some process with a log class, these processes could not do any other work and could not be promoted as a proxy for instance, right?