Use TOML for test specifications

Development FoundationDB Core
1

As I’ve been walking through pieces of simulation to try and write bits up about how they work, I’ve ended up looking at the testspec again, and realizing that maybe it’s not quite something that I want to use as the described template of how to make a good declarative test spec. TOML is a slightly-more-featureful INI-like language for config files, and testspec is sort of like INI already, so this seems like possibly the most natural transition to something that seems sane to describe?

I’d like to propose adding support reading and running .toml structured test specs in tests/, and then later converting all .txt files to .toml and removing testspec from trunk.

As an example,

testTitle=CloggedCausalConsistencyTest
    testName=Sideband
    testDuration=30.0
    operationsPerSecond=500

    testName=RandomClogging
    testDuration=30.0
    scale=0.1
    clogginess=2.0

    testName=Attrition
    machinesToKill=10
    machinesToLeave=3
    reboot=true
    testDuration=30.0

would become

[test]
title = SimplifiedCloggedCausalConsistencyTest
# Other test-wide options like waitForQuiesenceEnd go here

[[workload]]
testName=Sideband
testDuration=30.0
operationsPerSecond=500

[[workload]]
testName=RandomClogging
testDuration=30.0
scale=0.1
clogginess=2.0

[[workload]]
testName=Attrition
machinesToKill=10
machinesToLeave=3
reboot=true
testDuration=30.0

This raises the question of how would restarting tests work? Restarting tests would try to run fdbserver-5.2.0 with a TOML testspec, which obviously wouldn’t work. If we would do a complete transition, then there would need to be a small script which consumes a TOML and translates it back into an equivalent testspec, which the test harness would use when running .toml tests on <= 7.0.0. As the change is mostly structural, TOML->testspec conversion should be pretty simple (and almost achievable with sed).

Of the C++ TOML libraries that exist, toml11 (MIT Licensed) is looking like the most supported and easy to use, so I’d like to look into incorporating it into FDB.

1 Like
2

I apparently missed that one can set testTitle multiple times to indicate different phases of a test.

testTitle=Clogged
    testName=Cycle
    transactionsPerSecond=2500.0
    testDuration=10.0
    expectedRate=0

    testName=RandomClogging
    testDuration=10.0

testTitle=Unclogged
    testName=Cycle
    transactionsPerSecond=250.0
    testDuration=10.0
    expectedRate=0.80

It appears that a minor change can be done to the TOML-ification to mirror this, which would now look like:

[[test]]
title="Clogged"

    [[test.workload]]
    testName="Cycle"
    transactionsPerSecond=2500.0
    testDuration=10.0
    expectedRate=0

    [[test.workload]]
    testName="RandomClogging"
    testDuration=10.0

[[test]]
title="Unclogged"

    [[test.workload]]
    testName="Cycle"
    transactionsPerSecond=250.0
    testDuration=10.0
    expectedRate=0.80

which is equivalent to

{
  "test": [
    {
      "title": "Clogged",
      "workload": [
        {
          "testName": "Cycle",
          "expectedRate": 0,
          "testDuration": 10.0,
          "transactionsPerSecond": 2500.0
        },
        {
          "testDuration": 10.0,
          "testName": "RandomClogging"
        }
      ]
    },
    {
      "title": "Unclogged",
      "workload": [
        {
          "testName": "Cycle",
          "expectedRate": 0.8,
          "testDuration": 10.0,
          "transactionsPerSecond": 250.0
        }
      ]
    }
  ]
}

I think in the end, the TOML variant is equivalent to the testspec with three minor modifications:

  • strings are quoted
  • [[test]] is inserted before every testTitle
  • [[test.workload]] is inserted before every testName

And then we can probably look into dropping the test prefix on a number of things afterwards.

3

I don’t have a strong enough opinion to actually chip in, but I, personally, am a big fan of using text literal protobufs for config. It ends up looking like JSON, which is harder to read than TOML but still reasonable, and gives you types which protects against typos / type errors in config and gives more reasonable C++ interface.

4

I like the idea of using a more capable config-file format than what we currently have. I also would like to add some more configuration options to the simulator, but this is currently hard to do as the file format is so limited in functionality.

I don’t like the idea of using protobuf - if we want a schema language I would rather stick with XML (which is imho as readable as the protobuf stuff but with a proper standard). Or we could use YAML (which I also don’t like).

I was not aware of TOML, but it looks super interesting. I think I favor @alexmiller’s suggestion to use TOML.

5

With PR #3500 now merged, TOML is now a fully supported way to write tests.

Anyone that has infrastructure that generates testspec files should change their code to generate TOML instead.

1 Like