# FDB Encrypted Backup bug caused by addition of AsyncFileKAIO Latency Histograms? **URL:** \ **Category:** FoundationDB Core\ **Created:** [April 26, 2023, 9:25am UTC](https://forums.foundationdb.org/t/fdb-encrypted-backup-bug-caused-by-addition-of-asyncfilekaio-latency-histograms/3930 "2023-04-26T09:25:01Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![pwood](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/pwood/32/1653_2.png) [@pwood](https://forums.foundationdb.org/u/pwood)\ **Post date:** [April 26, 2023, 9:25am UTC](https://forums.foundationdb.org/t/fdb-encrypted-backup-bug-caused-by-addition-of-asyncfilekaio-latency-histograms/3930/1 "2023-04-26T09:25:01Z")
In Februrary I was looking at implementing encrypted FDB backups, I hit an issue which I couldn’t process past - posted in the forum here. > [@Encrypted backups fail to initialise](https://forums.foundationdb.org/t/encrypted-backups-fail-to-initialise/3782): > > Hello! I’ve been trying to get encrypted backups to work for a couple of days with no luck. I’ve placed a 16 byte encryption key on all nodes of the cluster, in the same location (/mnt/fdb/backup-encryption-key.dat). I attempt to start the backup with: [root@XXX ec2-user]# fdbbackup start -d blobstore://XXX@s3.eu-west-1.amazonaws.com:443/main-dev-2023020601?bucket=XXX -s 604800 -z --encryption-key-file /mnt/fdb/backup-encryption-key.dat --log --logdir . With no delay I receive back: Interna… I’ve since had some time to look at it, the actual error is being caused by > It appears an error occurrs while trying to load the encryption key from disk, my understanding of the code is this is executed within `fdbbackup`, rather than via RPC in `fdbserver`. Specificially the fault is caused by access to: `SERVER_KNOBS->DISK_METRIC_LOGGING_INTERVAL`. The trace shows `ClientKnobCollection::getServerKnobs()` which suggests that AsyncFileKAIO is being used in the Client (`fdbbackup`)? But the knob seems to only be present on the ServerKnobCollection? ---
**Author:** ![SteavedHams](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/steavedhams/32/18_2.png) [@SteavedHams](https://forums.foundationdb.org/u/SteavedHams)\ **Post date:** [April 27, 2023, 11:34pm UTC](https://forums.foundationdb.org/t/fdb-encrypted-backup-bug-caused-by-addition-of-asyncfilekaio-latency-histograms/3930/2 "2023-04-27T23:34:09Z")
Try starting your `backup_agent` processes with `--knob_disable_posix_kernel_aio 1` to see if that avoids the issue. This disables use of KAIO which should avoid initializing its metrics. Conceptually, this knob should be a `FLOW_KNOB` not a `SERVER_KNOB` but I’m actually surprised that the server knobs object is not initialized as its definition is also in the fdbclient code. Assuming the server knobs init is the issue, moving the knob to `flow/include/flow/Knobs.h` and `flow/Knobs.cpp` should fix it. I wouldn’t expect that this issue is specific to encrypted backup files but rather just writing to a `file://` destination with backup, which is something I don’t think is done often outside of simulation tests which run from `fdbserver` so you’re the first person to find/report it. ---
**Author:** ![pwood](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/pwood/32/1653_2.png) [@pwood](https://forums.foundationdb.org/u/pwood)\ **Post date:** [May 2, 2023, 9:32am UTC](https://forums.foundationdb.org/t/fdb-encrypted-backup-bug-caused-by-addition-of-asyncfilekaio-latency-histograms/3930/3 "2023-05-02T09:32:07Z")
Hello! Thanks for the feedback, sorry for the delay - we’ve had a public holiday in the UK. TL;DR: Adding `--knob_disable_posix_kernel_aio 1` allows the `fdbbackup` command to executed. We took the opportunity to upgrade to 7.1.31, first we replicated the issue: ```auto [root@ip-10-1-245-241 ~]# fdbbackup start -d blobstore://REDACTED@s3.eu-west-1.amazonaws.com:443/REDACTED?bucket=REDACTED -s 604800 -z --encryption-key-file /mnt/fdb/backup-encryption-key.dat --log --logdir . bash: fdbbackup: command not found [root@ip-10-1-245-241 ~]# /opt/foundationdb/current/bin/fdbbackup start -d blobstore://REDACTED@s3.eu-west-1.amazonaws.com:443/REDACTED?bucket=REDACTED -s 604800 -z --encryption-key-file /mnt/fdb/backup-encryption-key.dat --log --logdir . Internal Error @ /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/ClientKnobCollection.h 45: addr2line -e fdbbackup.debug -p -C -f -i 0xe5f47d 0xd8c585 0xe1b6b4 0xe1befb 0xe0539a 0xe48af7 0xe3fdee 0xe056f4 0x62a78e 0x610684 0x64f7d1 0x600680 0x5666ad 0x550cf1 0x7fa05be6c13a ERROR: Could not create backup container: An internal error occurred ERROR: An error was encountered during submission Fatal Error: Backup error ``` Then we tested with the knob to disable AIO: ```auto [root@ip-10-1-245-241 ~]# /opt/foundationdb/current/bin/fdbbackup start -d blobstore://REDACTED@s3.eu-west-1.amazonaws.com:443/REDACTED?bucket=REDACTED -s 604800 -z --encryption-key-file /mnt/fdb/backup-encryption-key.dat --log --logdir . --knob_disable_posix_kernel_aio 1 The backup on tag `default' was successfully submitted. ``` Seems like the server knob may be the issue? I assume the command line argument only effects `fdbbackup`, I assume `backup_agent` and `fdbserver` will keep using AIO for actually performing the backup? ---
**Author:** ![SteavedHams](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/steavedhams/32/18_2.png) [@SteavedHams](https://forums.foundationdb.org/u/SteavedHams)\ **Post date:** [May 4, 2023, 12:34am UTC](https://forums.foundationdb.org/t/fdb-encrypted-backup-bug-caused-by-addition-of-asyncfilekaio-latency-histograms/3930/4 "2023-05-04T00:34:09Z")
> [@pwood](#): > > `backup_agent` and `fdbserver` will keep using AIO for actually performing the backup? Yes, the knob is just changing the file access mode for the `fdbbackup` command and any file operations it does on backup data. In the case of `start` it is just creating the output folder.