# How to save partial data in case of lost all tlog nodes and some of storage nodes

**URL:** <https://forums.foundationdb.org/t/how-to-save-partial-data-in-case-of-lost-all-tlog-nodes-and-some-of-storage-nodes/3114>\
**Category:** Using FoundationDB\
**Created:** [January 15, 2022, 11:38am UTC](https://forums.foundationdb.org/t/how-to-save-partial-data-in-case-of-lost-all-tlog-nodes-and-some-of-storage-nodes/3114 "2022-01-15T11:38:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ppggff](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ppggff/32/855_2.png) [@ppggff](https://forums.foundationdb.org/u/ppggff)\
**Post date:** [January 15, 2022, 11:38am UTC](https://forums.foundationdb.org/t/how-to-save-partial-data-in-case-of-lost-all-tlog-nodes-and-some-of-storage-nodes/3114/1 "2022-01-15T11:38:40Z")

</div>

We had a 40 node cluster that expanded to 60 nodes. These additional 20 nodes  
include 8 tlog (it’s all tlogs) nodes and 12 storage nodes.

During the expansion rebalance process(20% data transferred), the new 20 nodes were accidentally  
lost during the rebalance process.  
(processes were killed and the data directory was deleted).

Then restarting the cluster reports the “Locking coordination state.” and hangs.

In this case:

1. is there any way to start the cluster? Data lost is acceptable.
2. is there any way to dump the data from the originial remaining 40 nodes?  
For example, read data from sqlite directly, and we can fix the \<key, value\> data and load into an new FDB cluster.

---

<div class="post-metadata">

**Author:** ![mengxu](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/mengxu/32/893_2.png) [@mengxu](https://forums.foundationdb.org/u/mengxu)\
**Post date:** [January 16, 2022, 6:47am UTC](https://forums.foundationdb.org/t/how-to-save-partial-data-in-case-of-lost-all-tlog-nodes-and-some-of-storage-nodes/3114/2 "2022-01-16T06:47:08Z")

</div>

Which FDB version are you using?  
What configuration did you use? (triple ssd?)  
Did you configure HA (aka fearless) for this cluster?  
How much data loss can you accept? (if you configure a new db, all data will get lost, but cluster will be back. I guess that’s not what you want.)

If this is a test cluster, you can try `fdbcli force_recovery_with_data_loss`. It will drop tLog data and try to get cluster back. Its implementation is at:

> <https://github.com/apple/foundationdb/blob/de39293b8d6033c23702e0cd920eb6b09a16eb79/fdbcli/fdbcli.actor.cpp#L2053-L2058>

---

<div class="post-metadata">

**Author:** ![ppggff](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ppggff/32/855_2.png) [@ppggff](https://forums.foundationdb.org/u/ppggff)\
**Post date:** [January 16, 2022, 7:15am UTC](https://forums.foundationdb.org/t/how-to-save-partial-data-in-case-of-lost-all-tlog-nodes-and-some-of-storage-nodes/3114/3 "2022-01-16T07:15:52Z")

</div>

@mengxu Thanks a lot, this is the same problem in [Recovery from lost all transaction node](https://forums.foundationdb.org/t/recovery-from-lost-all-transaction-node/3116)  
we will try it.

---

<div class="post-metadata">

**Author:** ![ppggff](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ppggff/32/855_2.png) [@ppggff](https://forums.foundationdb.org/u/ppggff)\
**Post date:** [January 27, 2022, 2:48pm UTC](https://forums.foundationdb.org/t/how-to-save-partial-data-in-case-of-lost-all-tlog-nodes-and-some-of-storage-nodes/3114/4 "2022-01-27T14:48:58Z")

</div>

I create a [PR](https://github.com/apple/foundationdb/pull/6302) to dump data from storage file (sqlite) directly. I hope it will help to recover as much data as possible.