High rate of transaction retries with error code 1009 (Request for future version)

After digging some more, I’ve found that 1009 errors don’t always correlate to high durability_lag.

In my latest test, I do some 500 writes per second. Starting on an empty database, everything is fine for some time, but after a while, I start getting retries:

image

I don’t see any considerable change in data_lag nor durability_lag.

When I restart the writer, I get retries from the start.

Surprisingly, when I add another writer effectively doubling the load, all retries go away: