Recovering data from corrupt folder

[potterandy96 (Andy Potter)]

I’m using FoundationDB 7.3.43 on Linux 6.10.10. Due to reasons most of the auxiliary files in my directory got deleted before I could add another server to the cluster. I was left with the large data file that doubtless holds most of it, at /var/lib/foundationdb/data/4500/storage-$HASH.sqlite. I need to recover data from this, and/or import it into a new database, but can’t figure out how to read it. Please advise.

P.S. I did try setting up an fdb instance, running configure new single ssd, and copying in the old storage-HASH.sqlite with the name of the new one while deleting the .sqlite-wal file. No dice; fdb seems to just hang. I skimmed the trace log and don’t see anything obvious, but could be missing something since there’s quite a lot of output.

[jzhou (Jingyu Zhou)]

there is a way to dump storage file content to stdout: fdbserver -r kvfile dump --kvfile your-storage-file.sqlite. See foundationdb/fdbserver/KeyValueStoreSQLite.actor.cpp at release-7.3 · apple/foundationdb · GitHub

There are 3 environmental variables to control the behavior: FDB_DUMP_STARTKEY, FDB_DUMP_ENDKEY, and FDB_DUMP_DEBUG.

[potterandy96 (Andy Potter)]

Thanks. It barfs a ton of output to terminal but how do I import the keys/values to a new cluster?

Below here’s how I got it working, in case anyone else runs across this thread:

I tried FDB_DUMP_DEBUG=true fdbserver --role kvfiledump --kvfile storage-HASH.sqlite but get only this:

Dump start: , end: :\xff\xff, debug: true
Fatal Error: Broken promise

I found this thread showing I needed to shutdown the cluster first: How would I recover from this failed cluster move? - #8 by swr

However, I tried doing that and no difference. I tried copying it to a new cluster in place of that cluster’s storage .sqlite file and got the same, only with database corruption line 2411 before the broken promise. I tried removing the .sqlite-wal file and got only Fatal Error: Broken promise. Finally, I tried running touch storage-HASH.sqlite-wal` and I could finally get a working dump

[swr (Sam Weaver)]

I’m pretty sure that the kvfiledump role is for “emergency” recoveries like this… I don’t think it’s part of a supported backup/restore path. I suspect you would need to build your own tooling to write these K/V pairs back to the database again (perhaps as part of your application that’s already connecting to FDB.)

The fdbbackup and fdbrestore tools (and the disk snapshot backup alternative) both have supported restore paths, if you want a more reliable way to do this in the future.

[potterandy96 (Andy Potter)]

Thanks. I ended up hacking the code a little to dump each kv pair in base64 format and figured I’d reimport it with python. But what if the pairs were in a directory? Will the dumped keys carry that info?

[swr (Sam Weaver)]

What do you mean by “in a directory” here?

[potterandy96 (Andy Potter)]

The python documentation mentions the directory layer: Python API — FoundationDB OFF documentation

My concern is if the application is expecting to find something in a directory, and this dump doesn’t capture directory structure, it might not be able to see the data.

[swr (Sam Weaver)]

Ah, I see the misunderstanding. FoundationDB is overloading the use of the term “directory” here. From the FDB docs (emphasis mine):

@FoundationDB Docs > Developer Guide > Directories

FoundationDB provides directories (available in each language binding) as a tool for managing related subspaces. Directories are a recommended approach for administering applications. Each application should create or open at least one directory to manage its subspaces.

Directories are identified by hierarchical paths analogous to the paths in a Unix-like file system. A path is represented as a tuple of strings. Each directory has an associated subspace used to store its content. The directory layer maps each path to a short prefix used for the corresponding subspace. In effect, directories provide a level of indirection for access to subspaces.

(source)

The directory “layer” represents directories as tuples within the tuple “layer”, which are themselves represented as a single key within the database. Another snippet from the docs (emphasis mine):

@FoundationDB Docs > Data Modeling

FoundationDB’s keys are ordered, making tuples a particularly useful tool for data modeling. FoundationDB provides a tuple layer (available in each language binding) that encodes tuples into keys. This layer lets you store data using a tuple like (state, county) as a key. Later, you can perform reads using a prefix like (state,). The layer works by preserving the natural ordering of the tuples.

(source)

So the TL;DR is: if you’ve retrieved all of your keys and values from the dump of the storage server, any “tuples” or “directories” should be encoded within the key/value pairs themselves. They’re not treated any differently than any other data, and wouldn’t result in actual directories on the filesystem.