# Regarding issue faced while running fdbbackup **URL:** \ **Category:** Using FoundationDB\ **Created:** [June 17, 2019, 12:33pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458 "2019-06-17T12:33:04Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![pragawaran](https://avatars.discourse-cdn.com/v4/letter/p/f07891/32.png) [@pragawaran](https://forums.foundationdb.org/u/pragawaran)\ **Post date:** [June 17, 2019, 12:33pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/1 "2019-06-17T12:33:04Z")
When I run fdbbackup (generated binary using GitHub source code), I am facing this issue “The backup on tag `default’ was successfully submitted but no backup agents are responding”. How to resolve this issue…? ---
**Author:** ![ajbeamon](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ajbeamon/32/13_2.png) [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)\ **Post date:** [June 17, 2019, 4:24pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/2 "2019-06-17T16:24:18Z")
Running fdbbackup starts the backup job, but it’s the agents that are responsible for actually doing the work. You’ll need to have at least one running for backup to actually make any progress. Depending on the size of your cluster and how quickly you want backup to go, you may want to run more than that. If you are using fdbmonitor to run your processes, these can be started by configuring them in foundationdb.conf, like so: ```auto command = /usr/lib/foundationdb/backup_agent/backup_agent logdir = /var/log/foundationdb [backup_agent.1] ``` If you are running your processes through some other mechanism, then you just need to start the agent processes with the cluster file for your cluster. See [https://apple.github.io/foundationdb/backups.html?highlight=backup#backup-agent-command-line-tool](https://apple.github.io/foundationdb/backups.html?highlight=backup#backup-agent-command-line-tool) for more details. ---
**Author:** ![viveksinghggits](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/viveksinghggits/32/737_2.png) [@viveksinghggits](https://forums.foundationdb.org/u/viveksinghggits)\ **Post date:** [December 16, 2019, 11:11am UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/3 "2019-12-16T11:11:11Z")
> [@ajbeamon](#): > > If you are using fdbmonitor to run your processes, these can be started by configuring them in foundationdb.conf, like so: hey @ajbeamon how do I verify that I am running the processes using fdbmonitor. I installed the cluster using the foundationDB operator documented [here](https://github.com/foundationdb/fdb-kubernetes-operator). A part from that the foundationDB documentation says, in the link that you shared, that we dont need to start backup\_agent manually it usually runs automatically on the machine. ---
**Author:** ![gaurav](https://avatars.discourse-cdn.com/v4/letter/g/b487fb/32.png) [@gaurav](https://forums.foundationdb.org/u/gaurav)\ **Post date:** [December 16, 2019, 11:56am UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/4 "2019-12-16T11:56:59Z")
You can do `ps -ef | grep fdb` and the verify if fdbmonitor is running and if its pid is same as fdbserver process parent pid. somehting like this: ```auto ubuntu 1087 1 0 Dec12 ? 00:00:00 /usr/bin/fdbmonitor --conffile /etc/foundationdb/foundationdb.conf --lockfile /var/run/fdb/fdbmonitor.pid --daemonize ubuntu 1099 1087 0 Dec12 ? 00:18:54 /usr/lib/foundationdb/backup_agent/backup_agent --cluster_file /etc/foundationdb/fdb.cluster --logdir /var/log/foundationdb ubuntu 1100 1087 10 Dec12 ? 09:29:54 /usr/bin/fdbserver --cluster_file /etc/foundationdb/fdb.cluster --datadir /var/lib/foundationdb/data/4500 --listen_address public --logdir /var/log/foundationdb --public_address auto:4500 ``` ---
**Author:** ![viveksinghggits](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/viveksinghggits/32/737_2.png) [@viveksinghggits](https://forums.foundationdb.org/u/viveksinghggits)\ **Post date:** [December 16, 2019, 12:00pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/5 "2019-12-16T12:00:12Z")
Hi @gaurav Below is the output `ps -ef` in my case ```auto ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 06:58 ? 00:00:00 sh -c fdbmonitor --conffile /var/dynamic-conf/fdbmonitor.conf --lockfile /var/fdb/fdbmonitor.lockfile root 6 1 0 06:58 ? 00:00:00 fdbmonitor --conffile /var/dynamic-conf/fdbmonitor.conf --lockfile /var/fdb/fdbmonitor.lockfile root 7 6 2 06:58 ? 00:06:04 /var/dynamic-conf/bin/6.2.11/fdbserver --class storage --cluster_file /var/fdb/data/fdb.cluster --datadir /var/fdb/data --knob_disable_posix_kernel_aio 1 --locality_instance_id 1 --locali root 122 0 0 08:27 pts/0 00:00:00 bash ``` so it seems I have `fdbmonitor` running and its `PID` is same as the `fdbserver`'s `PPID`. ---
**Author:** ![viveksinghggits](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/viveksinghggits/32/737_2.png) [@viveksinghggits](https://forums.foundationdb.org/u/viveksinghggits)\ **Post date:** [December 16, 2019, 1:52pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/6 "2019-12-16T13:52:48Z")
To resolve it I changed the `fdbmonitor.conf` file to have details of backup\_agent, because it seems the backup\_agent is not being started by default. So insert below snipped in your `fdbmonitor.conf` file. ```auto [backup_agent] command = /usr/bin/backup_agent -C [backup_agent.1] ``` in my case `fdb-cluster-file` is at `/var/dynamic-conf/fdb.cluster`. After that you dont have to restart the service because this dir is always listened and if anything is changed the processes will be loaded accordingly. After doing the mentioned changes I was able to take the backup successfully. ---
**Author:** ![john\_brownlee](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/john_brownlee/32/22_2.png) [@john\_brownlee](https://forums.foundationdb.org/u/john_brownlee)\ **Post date:** [December 16, 2019, 3:23pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/7 "2019-12-16T15:23:43Z")
When you’re running FDB through the Kubernetes Operator, it doesn’t start backup agents. You can start the backup agents through a deployment and have them get the cluster file from the config map that the operator creates. We should an example of this pattern to the documentation. We should also consider integrating backup management into the operator. ---
**Author:** ![viveksinghggits](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/viveksinghggits/32/737_2.png) [@viveksinghggits](https://forums.foundationdb.org/u/viveksinghggits)\ **Post date:** [December 16, 2019, 3:42pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/8 "2019-12-16T15:42:36Z")
Hey @john_brownlee that makes sense but I was able to take backup somehow, as mentioned in my above post, but the problem now is I am getting some issues while restoring that backup. I have created a post about that, can you please look into that when you have time. > [@Restoring a completed backup version results in an error](https://forums.foundationdb.org/t/restoring-a-completed-backup-version-results-in-an-error/1845): > > I have created backup of the foundationDB successfully, if I check the status of the backup below is what I get The previous backup on tag `default' at file:///data/fdbbackup/backup-2019-12-16-12-43-45.243778 completed at version 21655918762. BackupUID: 4b6d2cf2e97bd4a673b32a008ddaa17d BackupURL: file:///data/fdbbackup/backup-2019-12-16-12-43-45.243778 But When I try to restore the same backup using below command fdbrestore start --dest\_cluster\_file /var/dynamic-conf/fdb.cluster -r file:///da… ---
**Author:** ![viveksinghggits](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/viveksinghggits/32/737_2.png) [@viveksinghggits](https://forums.foundationdb.org/u/viveksinghggits)\ **Post date:** [December 17, 2019, 10:53am UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/9 "2019-12-17T10:53:11Z")
Hi @john_brownlee I will bother you once again about this, I started experiencing the same issue once again today, and when I do the exactly same thing that I have mentioned above about changing the fdbmonitor.config file to add backup\_agent section, things dont work. But this worked as expected yesterday. When I change the file with below content and same it ```auto [backup_agent] command = /usr/bin/backup_agent -C [backup_agent.1] ``` below is the output that i get in the pod’s logs ```auto Time="1576579785.791398" Severity="10" LogGroup="default" Process="fdbmonitor": Watching conf file /var/dynamic-conf/fdbmonitor.conf Time="1576579785.791595" Severity="10" LogGroup="default" Process="fdbmonitor": Watching conf dir /var/dynamic-conf/ (18) Time="1576579785.791615" Severity="10" LogGroup="default" Process="fdbmonitor": Loading configuration /var/dynamic-conf/fdbmonitor.conf Time="1576579785.791903" Severity="10" LogGroup="default" Process="fdbmonitor": Updated configuration for fdbserver.1 ``` Can you please suggest how to about taking the backup of the database. ---
**Author:** ![john\_brownlee](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/john_brownlee/32/22_2.png) [@john\_brownlee](https://forums.foundationdb.org/u/john_brownlee)\ **Post date:** [December 17, 2019, 3:07pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/10 "2019-12-17T15:07:19Z")
In general, I would recommend running the backup agents through a Deployment rather than through fdbmonitor. ---
**Author:** ![viveksinghggits](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/viveksinghggits/32/737_2.png) [@viveksinghggits](https://forums.foundationdb.org/u/viveksinghggits)\ **Post date:** [December 17, 2019, 4:18pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/11 "2019-12-17T16:18:49Z")
yeah, but there also even if we mount the configmap to the new backup\_agent deployment and then provide the fdb cluster file using -r flag, do you think fdbbackup will be able to backup the data from the foundationdb pod, have you tested that. One more thing, if my database is running through the operator pods and I want to run `fdbbackup` for the remote database, I am not on the same pod where my cluster file and other configuration are, how can we achieve that. Is there a way to provide a flag to `fdbbackup` so that it will take backup of the remote database. ---
**Author:** ![ajbeamon](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ajbeamon/32/13_2.png) [@ajbeamon](https://forums.foundationdb.org/u/ajbeamon)\ **Post date:** [December 17, 2019, 5:30pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/12 "2019-12-17T17:30:29Z")
Also as a side note, it shouldn’t be necessary to specify the cluster file in the `command` part of your foundationdb.conf file. If the correct cluster file is listed in the `general` section, I think it will automatically get picked up by the process. If not, you should be able to specify it in the `backup` section in the same way: ```auto [backup_agent] command = /usr/bin/backup_agent cluster_file = ``` ---
**Author:** ![john\_brownlee](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/john_brownlee/32/22_2.png) [@john\_brownlee](https://forums.foundationdb.org/u/john_brownlee)\ **Post date:** [December 17, 2019, 6:55pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/13 "2019-12-17T18:55:33Z")
The backup agents get their data by connecting to the cluster, rather than reading off of a disk, so they do not need to be in the same pod or on the same machine as the fdbserver processes. All you need to provide to backup\_agent and fdbbackup is the path to the cluster file. ---
**Author:** ![viveksinghggits](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/viveksinghggits/32/737_2.png) [@viveksinghggits](https://forums.foundationdb.org/u/viveksinghggits)\ **Post date:** [December 17, 2019, 7:14pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/14 "2019-12-17T19:14:27Z")
Yeah, I understood that but the problem is lets say if I run below command to run the backup\_agent on the other pod ```auto /usr/bin/backup_agent -C /var/dynamic-conf/fdb.cluster ``` here `-C` provides the mechanism to pass the cluster problem, now the confusion that I have is, where is `backup_agent` going to look for the file `/var/dynamic-conf/fdb.cluster`. Because backup\_agent will not be able to find the file where it (`backup_agent`) is running. ---
**Author:** ![john\_brownlee](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/john_brownlee/32/22_2.png) [@john\_brownlee](https://forums.foundationdb.org/u/john_brownlee)\ **Post date:** [December 17, 2019, 7:23pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/15 "2019-12-17T19:23:20Z")
You will need to mount the cluster file into the container where the backup agent is running. You should be able to mount the same config map that the operator creates for the fdbserver processes, and get the cluster file from there. ---
**Author:** ![viveksinghggits](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/viveksinghggits/32/737_2.png) [@viveksinghggits](https://forums.foundationdb.org/u/viveksinghggits)\ **Post date:** [December 17, 2019, 7:27pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/16 "2019-12-17T19:27:38Z")
Hi @john_brownlee I got that when we first discussed about that for the first time, but after looking into that configmap I had some doubts about how are we going to figure out where the cluster actually is. But when I looked into that file again, I think below is how backup\_agent is going to figure out the cluster. Thanks ```auto cluster-file: | # DO NOT EDIT! # This file is auto-generated, it is not to be edited by hand foundationdbcluster_sample:VPnLf5797TLX6qSQeACWSSdt6k7OHYGO@10.244.0.84:4500:tls,10.244.0.90:4500:tls,10.244.0.91:4500:tls ``` Can you suggest the binaries that should be there in the other pod that we are going to spin up for the backup\_agent or any documentation link would help. ---
**Author:** ![john\_brownlee](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/john_brownlee/32/22_2.png) [@john\_brownlee](https://forums.foundationdb.org/u/john_brownlee)\ **Post date:** [December 17, 2019, 7:51pm UTC](https://forums.foundationdb.org/t/regarding-issue-faced-while-running-fdbbackup/1458/17 "2019-12-17T19:51:11Z")
We have more documentation on backup in general here: [https://apple.github.io/foundationdb/backups.html](https://apple.github.io/foundationdb/backups.html)