From time to time under load spikes we are getting client app overflowed with work and we started to get errors that transaction is too old (since it exceeds 5 secs) while this is indeed a bad thing, but there are no way to exit this state easily since it retries, retries and retries. Today i got that transaction was retrying for over 30 minutes.
Since it is quite hard to predict workload and we are starting to look for lower lever throttling to avoid this errors.
Is there are any best practices for this? Throttling starting of a transaction? Limiting number of parallel transactions?