Internal error when running KVStoreMemTest

Using FoundationDB
1

This is on branch release-6.0. (tested both on Ubuntu 18.04 and mac)
I tried to run the KVStoreMemTest (except I changed the nodeCount to be 2000) using command
./bin/fdbserver -r simulation -f tests/KVStoreMemTest.txt -s 6

Insertion part works fine, however, when it came to scan part, i got the
Internal Error @ fdbserver/workloads/KVStoreTest.actor.cpp 168
My first guess would be the assertion failure, wondering if anyone else has the same problem.

2

Is there a specific reason that you’re running KVStoreMemTest? Our correctness testing framework only run tests that are in subdirectories of tests/, and the top level folder is used to hold tests that were disabled, or only meant for one-off testing. So them doing something meaningful and passing isn’t actually guaranteed.

On master, you can look at fdbserver/CMakeLists.txt for the list of tests that were enabled to run under ctest as a useful guide of what should work in simulation.

… we really need to add a README to tests/ to this effect or something.

EDIT: Actually reading the testspec, this looks like a test that was meant to be run via multitest to generate load against a real cluster, and not something meant for simulation.

3

Thanks @alexmiller
We are trying to develop a new memory storage engine, basically to replace the IndexedSet with a Radix Tree like data structure. And i was wondering if there is any simulation tests I can take advantage of for testing the correctness of our new component.

1 Like
4

I’d suggest just editing fdbserver/SimulatedCluster.actor.cpp's SimulationConfig via:

- if (g_random->random01() < 0.5) {
-   set_config("ssd");
- } else {
-   set_config("memory");
- }
+ set_config("memory-radix");

or whatever your new storage engine’s name is, and then all defined simulation tests would use your new memory storage engine when running.

Which if you haven’t already, it would mean you’d need to create a new name for your storage engine and add it to DatabaseConfiguration and ManagementAPI. You’re more than welcome to post links to either your branch or a throwaway PR to be able to get early feedback on what you’re doing and suggestions/advice/help on what to do.

5

I’m curious which parameter in the test file [1] shows it is against a real cluster instead of for simulation?
(Is the setup=true parameter?)

[1] [The test file: KVStoreMemTest.txt](https://github.com/apple/foundationdb/blob/master/tests/KVStoreMemTest.txt)

6

I’m kind of guessing off of the fact that there’s five workloads going on, all targeting 2 million keys, none of them trigger faults, and they expect things like 28k commit/s. That seems … not attainable in simulation. But this is the sort of thing we’d do on a performance testing cluster, hence my guess of it being meant for multitest.

This is why Mengran would have had to drop the nodeCount to make it runnable in simulation at all, but even then, it’s still not really providing a useful test of correctness due to the lack of any sort of attrition or fault injection workload.

1 Like