In Februrary I was looking at implementing encrypted FDB backups, I hit an issue which I couldn’t process past - posted in the forum here.
I’ve since had some time to look at it, the actual error is being caused by
It appears an error occurrs while trying to load the encryption key from disk, my understanding of the code is this is executed within fdbbackup, rather than via RPC in fdbserver.
Specificially the fault is caused by access to: SERVER_KNOBS->DISK_METRIC_LOGGING_INTERVAL.
The trace shows ClientKnobCollection::getServerKnobs() which suggests that AsyncFileKAIO is being used in the Client (fdbbackup)? But the knob seems to only be present on the ServerKnobCollection?
Try starting your backup_agent processes with --knob_disable_posix_kernel_aio 1 to see if that avoids the issue. This disables use of KAIO which should avoid initializing its metrics.
Conceptually, this knob should be a FLOW_KNOB not a SERVER_KNOB but I’m actually surprised that the server knobs object is not initialized as its definition is also in the fdbclient code. Assuming the server knobs init is the issue, moving the knob to flow/include/flow/Knobs.h and flow/Knobs.cpp should fix it.
I wouldn’t expect that this issue is specific to encrypted backup files but rather just writing to a file:// destination with backup, which is something I don’t think is done often outside of simulation tests which run from fdbserver so you’re the first person to find/report it.
[root@ip-10-1-245-241 ~]# /opt/foundationdb/current/bin/fdbbackup start -d blobstore://REDACTED@s3.eu-west-1.amazonaws.com:443/REDACTED?bucket=REDACTED -s 604800 -z --encryption-key-file /mnt/fdb/backup-encryption-key.dat --log --logdir . --knob_disable_posix_kernel_aio 1
The backup on tag `default' was successfully submitted.
Seems like the server knob may be the issue? I assume the command line argument only effects fdbbackup, I assume backup_agent and fdbserver will keep using AIO for actually performing the backup?
Yes, the knob is just changing the file access mode for the fdbbackup command and any file operations it does on backup data. In the case of start it is just creating the output folder.