I’m trying to remove a process on one machine. I tried deleting it from the foundationdb.conf in the server section and restart service but it didn’t work, the error says
Locking coordination state. Verify that a majority of coordination server
processes are active.
100.90.21.229:4500 (reachable)
Unable to locate the data distributor worker.
Unable to locate the ratekeeper worker.
Thank you!
There’s a section of the documentation on removing machines from a cluster that shows how to use exclude to do this safely.
Even I’m facing similar issue and I didn’t touch the existing setup. It was working fine an hour before and able to get and set key-values. Tried after sometime to get the data’s in DB using fdbcli. But, fdbcli shows The database is unavailable; typestatus’ for more information.`
.
When I check the status
Locking coordination state. Verify that a majority of coordination server
processes are active.
127.0.0.1:4500 (reachable)
Unable to locate the data distributor worker.
Unable to locate the ratekeeper worker.
Looks lost my data. Is it possible to bring it back ?Why I’m getting this issue ?
I’m going to need more information about your cluster’s setup to be able to provide any theories as to why you’re seeing this. What environment are you running in? How many processes? How are they configured? What is your clusters configuration?
It’d also be good to scan your trace.*.xml log files for any Severity=“40” events.
Running one fdbserver process with memory configured “single ssd”.
I have verified logs and there is no Severity=“40”.
After following Alex’s suggestion, does the problem still exist?
When we get into these kind of situations, what are the possibilities to recover the data from storage. Is there any specific steps to recover database ?
Based on the information you’ve provided so far, I’m not entirely sure what’s gone wrong in order to recommend a resolution. Are you able to provide the output of running status json in fdbcli? Also, it would be useful to see the full trace log for your single process if possible. In order to limit its size, it may be easiest initially to restart the process and let it run for a minute before copying its file (restarting it will cause it to start a new one).
I have brought up completely new setup and there is no log present now for above mentioned issue. If I get similar issue again, will update details whatever you need. thanks @ajbeamon for your support.