Setting value size limit knob during fdbrestore

I’ve been trying to restore from a cluster where my Java client that used to write to it, had the value_size_limit knob set to 3MB. How can I set this knob for the new cluster I’m trying to restore to? It’s a client knob so I believe putting it in foundationdb.conf wouldn’t be useful.
I keep getting the error:

‘Value length exceeds limit’ on ‘restore_range_data’

I’m not exactly sure how the restore works now, but I believe you are going to need to set the value_size_limit knob on the agents responsible for actually doing the restore, which I think are the backup_agent processes. I believe this can be done the same way as for server processes, by using --knob_value_size_limit=3000000 or using knob_value_size_limit=3000000 in foundationdb.conf.

Also be aware that there are a couple places that we make use of this value on the server side, so I would recommend also having this value set on all of your servers to avoid any weird behaviors.

1 Like

That makes sense, I will try out setting the value size knob in all processes of the server and backup_agent as well.

I think I might have seen at least one weird behaviour as a result of setting them only on the client - we do some parsing of backup files for some internal analysis, and noticed that log ranges had incomplete mutation blocks for certain versions. The size encoding at the beginning of each block mentioned (10000x + y) bytes for a particular version, but we found only 10000z bytes in that block, where x > z. Here, I am accounting for the fact that the mutation logs for each version can sometimes be spread across multiple parts and we have to concatenate those parts to get the final set of logs per version. This happened for a significant number of mutation blocks.

@ajbeamon if i want to allow bigger size values to be allowed into the database, does setting it at the cluster level from the CLI sound a better option ? I want to do that for the transaction sizes as well.
Just to call it out , i’m aware that there might be potential issues by tweaking these knobs beyond the recommended sizes as discussed in a few other posts, but the number of transactions is fairly infrequent

We don’t currently have any mechanism to configure value size or transaction size via fdbcli. If you want to tinker with these values, I think it will be necessary for you to set the relevant knobs on every server process as well as any client process that might need to make use of the increased sizes.

Having other ways to generally configure knob values, potentially even on a live cluster, is an idea that has been considered from time to time, and I think it’s something that we’d ultimately like to have, but there is some trickiness involved that needs to be sorted out. In particular, here are some thoughts that have come up:

  1. Some knobs can be changed live, some can be changed on restart, and some can only be set upon database creation. Our scheme will need to take that into account.
  2. If we decide to allow different classes of knobs as described above, we’d need adequate testing to make sure a knob is categorized correctly and doesn’t break anything when changed.
  3. There are various other complexities, such as knobs that typically need to be changed in tandem or knobs where alternate values have never been tried and may not work. We can maintain the philosophy that knobs are use at your own risk, but there’s probably more likelihood for them to be misused when they are easily and prominently configurable.

If there are particular knobs that people are configuring frequently, then it may make sense to special case them and make it easier to adjust the values (we’ve done this, for example, with the size of the page cache). It’s possible these knobs fit that bill, though I don’t know how often they are being set.