We’re using FDB with the FDB record layer, though this question isn’t specific to the record layer.
We recently ran a large data job that made a lot of FDB calls. Our monitoring saw that the storage_server_durability_lag was going up. In the past, when this has happened for a sustained period of time, it’s caused the cluster to become unhealthy. This time, luckily we were able to catch it and stop the job.
In the future, we’d like to have whatever is doing transactions against FDB be able to auto-detect this kind of situation, and slow down the data load, so it doesn’t put too much load on FoundationDB. It seems right now, the client is just going as fast as it can, without regard for server health. We’d like to be able to limit the client’s interactions with the server to a healthy level. Ideally, not fail, but just slow things down to a manageable level.
Is there a recommended way to do this?