FoundationDB

Fdbmonitor error handling symlink in config path [bug]


(Jehiah) #1

I seem to have stumbled upon an edge case where fdbmonitor fails to watch the right directory when the config path starts with a symlink (thus failing to start). It seems to handle symlinks at other points in the config path just fine.

I’m not quite sure where the bug is because log entries appear to list the absolute path (no symlinks) just fine.


steps to reproduce

# /a/ file path
$ mkdir -p /a/fdbcfg
$ cp foundationdb.conf /a/fdbcfg/
# /b/ symlink not at path root
$ mkdir -p /b
$ ln -s /a /b/symlink
# /c/ symlink at path root
$ ln -s /a /c

Works:

  • --conffile /a/fdbcfg/foundationdb.conf
  • --conffile /b/symlink/fdbcfg/foundationdb.conf

Fails:

  • --conffile /c/fdbcfg/foundationdb.conf

logs

$ /usr/lib/foundationdb/fdbmonitor --conffile /a/fdbcfg/foundationdb.conf 
Time="1530034963.433985" Severity="10" LogGroup="default" Process="fdbmonitor": Started FoundationDB Process Monitor 5.1 (v5.1.5)
Time="1530034963.434218" Severity="10" LogGroup="default" Process="fdbmonitor": Watching conf file /a/fdbcfg/foundationdb.conf
Time="1530034963.434290" Severity="10" LogGroup="default" Process="fdbmonitor": Watching conf dir /a/fdbcfg (2)
Time="1530034963.434305" Severity="10" LogGroup="default" Process="fdbmonitor": Loading configuration /a/fdbcfg/foundationdb.conf
Time="1530034963.434863" Severity="10" LogGroup="default" Process="fdbmonitor": Starting backup_agent.1
Time="1530034963.435098" Severity="10" LogGroup="default" Process="fdbmonitor": Starting fdbserver.4700
Time="1530034963.435365" Severity="10" LogGroup="default" Process="backup_agent.1": Launching /usr/lib/foundationdb/backup_agent/backup_agent (8624) for backup_agent.1
Time="1530034963.439435" Severity="10" LogGroup="default" Process="fdbserver.4700": Launching /usr/sbin/fdbserver (8625) for fdbserver.4700
Time="1530034963.449245" Severity="10" LogGroup="default" Process="fdbserver.4700": FDBD joined cluster.
^CTime="1530034963.904522" Severity="20" LogGroup="default" Process="fdbmonitor": Received signal 2 (Interrupt), shutting down
$ /usr/lib/foundationdb/fdbmonitor --conffile /b/symlink/fdbcfg/foundationdb.conf 
Time="1530034975.996090" Severity="10" LogGroup="default" Process="fdbmonitor": Started FoundationDB Process Monitor 5.1 (v5.1.5)
Time="1530034975.996343" Severity="10" LogGroup="default" Process="fdbmonitor": Watching conf file /a/fdbcfg/foundationdb.conf
Time="1530034975.996357" Severity="10" LogGroup="default" Process="fdbmonitor": Watching conf dir /a/fdbcfg (2)
Time="1530034975.996368" Severity="10" LogGroup="default" Process="fdbmonitor": Watching parent directory of symlink /b/symlink (3)
Time="1530034975.996379" Severity="10" LogGroup="default" Process="fdbmonitor": Loading configuration /a/fdbcfg/foundationdb.conf
Time="1530034975.996888" Severity="10" LogGroup="default" Process="fdbmonitor": Starting backup_agent.1
Time="1530034975.997039" Severity="10" LogGroup="default" Process="fdbmonitor": Starting fdbserver.4700
Time="1530034975.998014" Severity="10" LogGroup="default" Process="backup_agent.1": Launching /usr/lib/foundationdb/backup_agent/backup_agent (8641) for backup_agent.1
Time="1530034976.001580" Severity="10" LogGroup="default" Process="fdbserver.4700": Launching /usr/sbin/fdbserver (8642) for fdbserver.4700
Time="1530034976.011770" Severity="10" LogGroup="default" Process="fdbserver.4700": FDBD joined cluster.
^CTime="1530034976.833575" Severity="20" LogGroup="default" Process="fdbmonitor": Received signal 2 (Interrupt), shutting down
$ /usr/lib/foundationdb/fdbmonitor --conffile /c/fdbcfg/foundationdb.conf 
Time="1530034981.019368" Severity="10" LogGroup="default" Process="fdbmonitor": Started FoundationDB Process Monitor 5.1 (v5.1.5)
Time="1530034981.019569" Severity="10" LogGroup="default" Process="fdbmonitor": Watching conf file /a/fdbcfg/foundationdb.conf
Time="1530034981.019582" Severity="10" LogGroup="default" Process="fdbmonitor": Watching conf dir /a/fdbcfg (2)
Time="1530034981.019593" Severity="40" LogGroup="default" Process="fdbmonitor": Unable to add watch to parent directory  (inotify_add_watch error 2: No such file or directory)

(Jehiah) #2

Oh i should note that i’ve been able to work around this issue once figuring it out.


(A.J. Beamon) #3

Good find. It looks like fdbmonitor isn’t correctly computing the parent directory for paths if the parent is the root:

I’ll try to submit a fix for that shortly.


(A.J. Beamon) #4

See https://github.com/apple/foundationdb/pull/533