Binding tester invocation

Development
1

I am currently working on developing binding tester for my rust bindings.

As I was trying to get a better understanding of how binding tester works, I wrote a very simple program that does nothing more than just print out the operation key-values that binding tester adds for the rust program to execute. It does not modify the database in any way. The working code is here.

Upon running the program using bindingtester.py, for some reason, it reports that the tests ran successfully, even though it was running a dummy program.

Here is the output of the invocation.

./bindingtester-630/bindingtester/bindingtester.py --test-name api rust --num-ops 2 --concurrency 3

Creating test at API version 630
Generating api test at seed 2432741821 with 2 op(s) and 3 concurrent tester(s)...

Inserting test into database...

Running tester '/home/rajiv/zfs/learning/foundation-db/dev-env/fdb/fdb-stacktester/bindingtester-630/bindingtester/../../../target/debug/fdb-stacktester test_spec 630'...
KeyValue { key: Key(b"\x01test_spec\0\x14"), value: Value(b"\x02PUSH\0\x01thread_spec0\0") }
KeyValue { key: Key(b"\x01test_spec\0\x15\x01"), value: Value(b"\x02START_THREAD\0") }
KeyValue { key: Key(b"\x01test_spec\0\x15\x02"), value: Value(b"\x02PUSH\0\x01thread_spec1\0") }
KeyValue { key: Key(b"\x01test_spec\0\x15\x03"), value: Value(b"\x02START_THREAD\0") }
KeyValue { key: Key(b"\x01test_spec\0\x15\x04"), value: Value(b"\x02PUSH\0\x01thread_spec2\0") }
KeyValue { key: Key(b"\x01test_spec\0\x15\x05"), value: Value(b"\x02START_THREAD\0") }

Reading results from '('tester_output', 'workspace')'...
Reading results from '('tester_output', 'stack')'...


Test with seed 2432741821 and concurrency 3 had 0 incorrect result(s) and 0 error(s) at API version 630
Completed api test with random seed 2432741821 and 2 operations

I was a bit surprised to see that it reported 0 incorrect result(s) and 0 error(s)

I was wondering if I was making any mistake in my invocation or use of bindingtester.py?

2

The API test by default doesn’t do any result validation. It has two primary intended modes – 1) to run via comparison with another tester, and 2) to run with concurrency and verify the program does not crash. There are also other tests which do some validation of the results. For example, if you run the scripted test, it will make sure that your stack machine generates a DB state consistent with what it expects.

For the api test, I would recommend mainly focusing on the first case I described above, and only running the second case once everything else works. To do that, you need to use the --compare flag and remove the --concurrency flag, and it will then run the same test with your binding tester and the Python one. If they produce different results, the test will fail. You can also change it to use a different comparison tester than Python if you prefer.

3

Thanks @ajbeamon for the reply. :slight_smile:

4

I was also biased by this output :tongue: Maybe we should improve it?