Backup_agent configuration for running fdbrestore

Using FoundationDB
1

Hi All,

I am trying to understand few things about fdbrestore in general and backup_agent in special.

  1. First case: I tried to restore fdb in some docker in which I have an available empty fdb instance with the cluster file which is not in the default location, I saw that fdbrestore can’t start. From the threads in this resource, I got that backup_agent has to be configured with the cluster file. I configured backup_agent cluster file with -C, and it worked. So the question is in which case before running fdbrestore start -r file://... --dest_cluster_file=<fdb.cluster path> -w I must run backup_agent -C <fdb.cluster path>?

  2. Second case: I just performed a regular fdb installation according to this guide (Getting Started on Linux — FoundationDB 6.3) on a Linux PC and tryied to restore fdb from some snapshot. It worked, I saw that restoring was completed successfully. But as far as I understand, this worked since I used the default cluster file(/etc/foundationdb/fdb.cluster). When I tried to configure backup_agent in any case with the cluster file, it turned out that backup_agent did not run. There is no such daemon. So the question, if I need to raise backup_agent somehow for successful restoring to a cluster file that is not in the default location? Or what is the correct procedure for the case when backup_agent does not run and I need to run fdbrestore?

thank you very much

2

So the question is in which case before running fdbrestore start -r file://... --dest_cluster_file=<fdb.cluster path> -w I must run backup_agent -C <fdb.cluster path> ?

The commands you run using fdbbackup signal to the cluster what actions you want to have taken, and the backup_agents are responsible for carrying out the operation. The state for the operation is stored in the cluster, so the backup_agents don’t need to be around to receive the command right when you issue it. That means you do need to have at least one backup_agent running to perform a backup or restore, but it could be started before or after you issue the command to do so.

Second case: I just performed a regular fdb installation according to this guide (Getting Started on Linux — FoundationDB 6.3) on a Linux PC and tryied to restore fdb from some snapshot. It worked, I saw that restoring was completed successfully. But as far as I understand, this worked since I used the default cluster file( /etc/foundationdb/fdb.cluster ). When I tried to configure backup_agent in any case with the cluster file, it turned out that backup_agent did not run. There is no such daemon. So the question, if I need to raise backup_agent somehow for successful restoring to a cluster file that is not in the default location? Or what is the correct procedure for the case when backup_agent does not run and I need to run fdbrestore ?

When you install FoundationDB from packages, it will write a /etc/foundationdb/foundationdb.conf file for you that configures a backup_agent to start. This backup_agent inherits the cluster file path from the [general] section, so it will automatically connect to the cluster specified there. You can specify a different cluster file by adding a line like:

[backup_agent.1]
cluster_file = /path/to/other/fdb.cluster

Or you could even start another backup agent for your second cluster:

[backup_agent.2]
cluster_file = ...
3

That means you do need to have at least one backup_agent running to perform a backup or restore, but it could be started before or after you issue the command to do so.

Thank you so much for answering! I would like to explain about my case a little bit.

I am writing a tool for fdb restoring from snapshot. Tool runs in container. My tool accepts destination empty cluster file (to restore in) and snapshot. From python script I call at some point such a code:

    # configure fdb backup agent with the correct fdb cluster name 
    subprocess.check_call(
        "backup_agent -C " + args.dest_cluster_file + " &",
        shell=True
        )

    # call fdbrestore utility on the backup dir
    subprocess.check_call(
        "fdbrestore start -r file://" + backup_dir_path + \
        " --dest_cluster_file " + args.dest_cluster_file + " -w",
        shell=True
        )

This tool will run each time on a different destination cluster file. So my question is about do I need and in what cases I need to configure manually backup_agent with -C <fdb.cluster path>? What is the proper way to configure it with a cluster file?

Does the backup agent start automatically or do I need to call it manually? If it starts automatically, which cluster file it listens to and how to configure it with the input destination cluster file?

Wanted to emphasise that I don’t run fdbbackup, only fdbrestore on a downloaded snapshot.

Thanks!!!

4

I just tested this, without backup_agent -C " + args.dest_cluster_file & the restore can’t start. I am getting LastError: ''File not found' on 'restore_range_data'' error and Blocks remained 0/0.

So as I understand, the configuration of backup_agent is mandatory, right?

5

It is necessary to run a backup_agent for the restore to make progress. Whether it’s necessary for the backup_agent to be running before you start the restore I’m not certain, though I had previously been under the impression that you could start it after.