Fdbmonitor error handling symlink in config path [bug]

jehiah · June 26, 2018, 5:50pm

I seem to have stumbled upon an edge case where fdbmonitor fails to watch the right directory when the config path starts with a symlink (thus failing to start). It seems to handle symlinks at other points in the config path just fine.

I’m not quite sure where the bug is because log entries appear to list the absolute path (no symlinks) just fine.

steps to reproduce

# /a/ file path
$ mkdir -p /a/fdbcfg
$ cp foundationdb.conf /a/fdbcfg/
# /b/ symlink not at path root
$ mkdir -p /b
$ ln -s /a /b/symlink
# /c/ symlink at path root
$ ln -s /a /c

Works:

--conffile /a/fdbcfg/foundationdb.conf
--conffile /b/symlink/fdbcfg/foundationdb.conf

Fails:

--conffile /c/fdbcfg/foundationdb.conf

logs

$ /usr/lib/foundationdb/fdbmonitor --conffile /a/fdbcfg/foundationdb.conf 
Time="1530034963.433985" Severity="10" LogGroup="default" Process="fdbmonitor": Started FoundationDB Process Monitor 5.1 (v5.1.5)
Time="1530034963.434218" Severity="10" LogGroup="default" Process="fdbmonitor": Watching conf file /a/fdbcfg/foundationdb.conf
Time="1530034963.434290" Severity="10" LogGroup="default" Process="fdbmonitor": Watching conf dir /a/fdbcfg (2)
Time="1530034963.434305" Severity="10" LogGroup="default" Process="fdbmonitor": Loading configuration /a/fdbcfg/foundationdb.conf
Time="1530034963.434863" Severity="10" LogGroup="default" Process="fdbmonitor": Starting backup_agent.1
Time="1530034963.435098" Severity="10" LogGroup="default" Process="fdbmonitor": Starting fdbserver.4700
Time="1530034963.435365" Severity="10" LogGroup="default" Process="backup_agent.1": Launching /usr/lib/foundationdb/backup_agent/backup_agent (8624) for backup_agent.1
Time="1530034963.439435" Severity="10" LogGroup="default" Process="fdbserver.4700": Launching /usr/sbin/fdbserver (8625) for fdbserver.4700
Time="1530034963.449245" Severity="10" LogGroup="default" Process="fdbserver.4700": FDBD joined cluster.
^CTime="1530034963.904522" Severity="20" LogGroup="default" Process="fdbmonitor": Received signal 2 (Interrupt), shutting down

$ /usr/lib/foundationdb/fdbmonitor --conffile /b/symlink/fdbcfg/foundationdb.conf 
Time="1530034975.996090" Severity="10" LogGroup="default" Process="fdbmonitor": Started FoundationDB Process Monitor 5.1 (v5.1.5)
Time="1530034975.996343" Severity="10" LogGroup="default" Process="fdbmonitor": Watching conf file /a/fdbcfg/foundationdb.conf
Time="1530034975.996357" Severity="10" LogGroup="default" Process="fdbmonitor": Watching conf dir /a/fdbcfg (2)
Time="1530034975.996368" Severity="10" LogGroup="default" Process="fdbmonitor": Watching parent directory of symlink /b/symlink (3)
Time="1530034975.996379" Severity="10" LogGroup="default" Process="fdbmonitor": Loading configuration /a/fdbcfg/foundationdb.conf
Time="1530034975.996888" Severity="10" LogGroup="default" Process="fdbmonitor": Starting backup_agent.1
Time="1530034975.997039" Severity="10" LogGroup="default" Process="fdbmonitor": Starting fdbserver.4700
Time="1530034975.998014" Severity="10" LogGroup="default" Process="backup_agent.1": Launching /usr/lib/foundationdb/backup_agent/backup_agent (8641) for backup_agent.1
Time="1530034976.001580" Severity="10" LogGroup="default" Process="fdbserver.4700": Launching /usr/sbin/fdbserver (8642) for fdbserver.4700
Time="1530034976.011770" Severity="10" LogGroup="default" Process="fdbserver.4700": FDBD joined cluster.
^CTime="1530034976.833575" Severity="20" LogGroup="default" Process="fdbmonitor": Received signal 2 (Interrupt), shutting down

$ /usr/lib/foundationdb/fdbmonitor --conffile /c/fdbcfg/foundationdb.conf 
Time="1530034981.019368" Severity="10" LogGroup="default" Process="fdbmonitor": Started FoundationDB Process Monitor 5.1 (v5.1.5)
Time="1530034981.019569" Severity="10" LogGroup="default" Process="fdbmonitor": Watching conf file /a/fdbcfg/foundationdb.conf
Time="1530034981.019582" Severity="10" LogGroup="default" Process="fdbmonitor": Watching conf dir /a/fdbcfg (2)
Time="1530034981.019593" Severity="40" LogGroup="default" Process="fdbmonitor": Unable to add watch to parent directory  (inotify_add_watch error 2: No such file or directory)

jehiah · June 26, 2018, 5:55pm

Oh i should note that i’ve been able to work around this issue once figuring it out.

ajbeamon · June 26, 2018, 10:48pm

Good find. It looks like fdbmonitor isn’t correctly computing the parent directory for paths if the parent is the root:

github.com

apple/foundationdb/blob/4f6054a3e6ff9321de3ff26b39b6151ada83e725/fdbmonitor/fdbmonitor.cpp#L243


				return joinPath( abspath( "." ), filename );
			}
		}


		log_err("realpath", errno, "Unable to get real path for %s", filename.c_str());
		return "";
	}
	return std::string(r);
}


std::string parentDirectory(std::string const& filename) {
	size_t sep = filename.find_last_of( CANONICAL_PATH_SEPARATOR );
	if (sep == std::string::npos) {
		return "";
	}


	return filename.substr(0, sep);
}


int mkdir(std::string const& directory) {
	size_t sep = 0;

I’ll try to submit a fix for that shortly.

ajbeamon · June 26, 2018, 11:35pm

See https://github.com/apple/foundationdb/pull/533

Topic		Replies	Views
Did something change in fdbmonitor in 5.2.5? Using FoundationDB	2	953	July 4, 2018
Starting/Stoping the Server Using FoundationDB	7	2176	May 31, 2019
Why fdb monitor is needed in foundationdb and what it's purpose? Using FoundationDB	7	1177	May 24, 2019
Segfault in version 6.2.11 Using FoundationDB	4	751	March 6, 2020
Fdbmonitor configuration reload not stopping old fdbserver processes Using FoundationDB	3	562	October 16, 2019

Fdbmonitor error handling symlink in config path [bug]

steps to reproduce

logs

Related topics