Hello!
I’ve been trying to get encrypted backups to work for a couple of days with no luck. I’ve placed a 16 byte encryption key on all nodes of the cluster, in the same location (/mnt/fdb/backup-encryption-key.dat).
I attempt to start the backup with:
[root@XXX ec2-user]# fdbbackup start -d blobstore://XXX@s3.eu-west-1.amazonaws.com:443/main-dev-2023020601?bucket=XXX -s 604800 -z --encryption-key-file /mnt/fdb/backup-encryption-key.dat --log --logdir .
With no delay I receive back:
Internal Error @ /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/ClientKnobCollection.h 45:
addr2line -e fdbbackup.debug -p -C -f -i 0xe4b63d 0xd7a935 0xe08f14 0xe0975b 0xdf338a 0xe34df7 0xe2bf3e 0xdf36e4 0x62582e 0x60b394 0x6487b1 0x5fb070 0x5620ed 0x54c2ca 0x7f63d716113a
ERROR: Could not create backup container: An internal error occurred
ERROR: An error was encountered during submission
Fatal Error: Backup error
Decoding the trace gives us:
[root@ip-10-1-244-15 ec2-user]# addr2line -e fdbbackup.debug.x86_64 -p -C -f -i 0xee35bc 0xee21d0 0xee25ce 0xe4b736 0xd7a935 0xe08f14 0xe0975b 0xdf338a 0xe34df7 0xe2bf3e 0xdf36e4 0x62582e 0x60b394 0x6487b1 0x5fb070 0x5620ed 0x54c2ca 0x7f63d716113a
std::string::_M_rep() const at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.h:3322
(inlined by) std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc:613
(inlined by) std::basic_string<char, std::char_traits<char>, std::allocator<char> > std::operator+<char, std::char_traits<char>, std::allocator<char> >(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.h:5967
(inlined by) BaseTraceEvent::backtrace(std::string const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Trace.cpp:1216
std::string::_M_rep() const at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.h:3322
(inlined by) std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.h:3640
(inlined by) BaseTraceEvent::log() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Trace.cpp:1235
BaseTraceEvent::~BaseTraceEvent() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Trace.cpp:1275
internal_error_impl(char const*, int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Error.cpp:61
ClientKnobCollection::getServerKnobs() const at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/ClientKnobCollection.h:45
AsyncFileKAIO::AsyncFileKAIO(int, int, std::string const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/AsyncFileKAIO.actor.h:652 (discriminator 1)
Reference<AsyncFileKAIO>::Reference(AsyncFileKAIO*) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/FastRef.h:101
(inlined by) AsyncFileKAIO::open(std::string, int, int, void*) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/AsyncFileKAIO.actor.h:153
Net2FileSystem::open(std::string const&, long, long) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/Net2FileSystem.cpp:70 (discriminator 3)
Future<Reference<IAsyncFile> >::Future(Future<Reference<IAsyncFile> > const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:807
(inlined by) StrictFuture<Reference<IAsyncFile> >::StrictFuture(Future<Reference<IAsyncFile> > const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:893
(inlined by) AsyncFileCached::Open_implActorState<AsyncFileCached::Open_implActor>::a_body1(int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/AsyncFileCached.actor.h:341
AsyncFileCached::Open_implActor::Open_implActor(std::string const&, int const&, int const&, Reference<EvictablePageCache> const&) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/AsyncFileCached.actor.g.h:1248
(inlined by) AsyncFileCached::open_impl(std::string const&, int const&, int const&, Reference<EvictablePageCache> const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/AsyncFileCached.actor.h:331
(inlined by) AsyncFileCached::open_impl(std::string, int, int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/AsyncFileCached.actor.cpp:81
std::string::_M_rep() const at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.h:3322
(inlined by) std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.h:3640
(inlined by) AsyncFileCached::open(std::string, int, int) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.h:3639
(inlined by) Net2FileSystem::open(std::string const&, long, long) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/Net2FileSystem.cpp:59
Future<Reference<IAsyncFile> >::Future(Future<Reference<IAsyncFile> > const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:807
(inlined by) StrictFuture<Reference<IAsyncFile> >::StrictFuture(Future<Reference<IAsyncFile> > const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/flow.h:893
(inlined by) BackupContainerFileSystemImpl::ReadEncryptionKeyActorState<BackupContainerFileSystemImpl::ReadEncryptionKeyActor>::a_body1(int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/BackupContainerFileSystem.actor.cpp:1148
BackupContainerFileSystemImpl::ReadEncryptionKeyActor::ReadEncryptionKeyActor(std::string const&) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbclient/BackupContainerFileSystem.actor.g.cpp:6054
(inlined by) BackupContainerFileSystemImpl::readEncryptionKey(std::string const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/BackupContainerFileSystem.actor.cpp:1144
(inlined by) BackupContainerFileSystem::setEncryptionKey(Optional<std::string> const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/BackupContainerFileSystem.actor.cpp:1485
std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >::begin() const at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_map.h:364
(inlined by) BackupContainerS3BlobStore::BackupContainerS3BlobStore(Reference<S3BlobStoreEndpoint>, std::string const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, Optional<std::string> const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/BackupContainerS3BlobStore.actor.cpp:152
Reference<S3BlobStoreEndpoint>::~Reference() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/FastRef.h:124
(inlined by) Reference<BackupContainerS3BlobStore> makeReference<BackupContainerS3BlobStore, Reference<S3BlobStoreEndpoint>&, std::string&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >&, Optional<std::string> const&>(Reference<S3BlobStoreEndpoint>&, std::string&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >&, Optional<std::string> const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/FastRef.h:185
(inlined by) IBackupContainer::openContainer(std::string const&, Optional<std::string> const&, Optional<std::string> const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/BackupContainer.actor.cpp:284
Reference<IBackupContainer>::operator=(Reference<IBackupContainer>&&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/FastRef.h:140
(inlined by) openBackupContainer(char const*, std::string const&, Optional<std::string> const&, Optional<std::string> const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbbackup/backup.actor.cpp:2311
Reference<IBackupContainer>::~Reference() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/FastRef.h:124
(inlined by) main at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbbackup/backup.actor.cpp:4014
It seems to hint at it being unable to open the encryption key, but I can’t find a reason as to why that might occur. I’m assuming this might not actually be an internal error, but bad input data, perhaps the encryption key format?
A link to the trace log from the fdbbackup run is here: fdbbackup encrypted failure · GitHub
Anyone got any advice?