Binding tester heisenbug with API version 710 and tenant

[PierreZ (Pierre Zemb)]

@ajbeamon

I’m not super familiar with what the details of what these scripts do, but I can give a summary from my understanding. The basic idea is that for a given run, we start with bindingTest.sh, which does a little setup and then runs bindingTestScript.sh. This creates a local cluster (using localClusterStart.sh) and then starts run_binding_tester.sh to actually run the binding tester. This last script has some logic to choose a particular set of arguments for the test and runs it.

We basically package all of this up and run it across a test framework that executes this whole process a bunch of times (with random seeds, etc., each time). I believe each run only does one run of the binding tester, and if the binding tester reports that the run failed (I think by exit code), then we will consider the test failed. We don’t really retry the same set of arguments in any automated way, but when we get reports of failures we would re-run the test manually.

Thanks a lot for the information, it is very helpful!
I will keep digging on the tenants errors and improve our CI infra