Using storage engine = memory and a single fdb instance on Mac OS X, I notice that our development instances of fdb seem to get into an interesting state on a relatively frequent basis and I’m trying to learn enough to understand what’s going on.
I have pasted status from fdbcli below and was hoping someone could provide some tips. The state does not resolve itself; the database is quite wedged at this point. Advice on how to unwedge it would be nice.
Sometimes, for reasons we haven’t figured out, the fdb instance becomes unhealthy. On occasion what we see is that waits on a FutureNil as returned by tr.Watch(key) will start seeing “FoundationDB error code 1009 (Request for future version)” errors. This is relatively easy to reproduce as it will occur after approximately 219 iterations of a test program at a very low rate of change (measured in ~1 key write per second); after that the database itself becomes unresponsive. i/o-wise, the laptop is idle.
WARNING: Long delay (Ctrl-C to interrupt)
Using cluster file `/usr/local/etc/foundationdb/fdb.cluster’.
Unable to start default priority transaction after 5 seconds.
Unable to start batch priority transaction after 5 seconds.
Unable to retrieve all status information.
Redundancy mode - single
Storage engine - memory
Coordinators - 1
FoundationDB processes - 1 (less 0 excluded; 1 with errors)
Machines - 1
Memory availability - 7.9 GB per process on machine with least available
Fault Tolerance - 0 machines
Server time - 08/08/18 17:53:46
Replication health - Healthy (Repartitioning.)
Moving data - 0.237 GB
Sum of key-value sizes - 544 MB
Disk space used - 1.292 GB
Storage server - 399.5 GB free on most full server
Log server - 399.5 GB free on most full server
Read rate - 1 Hz
Write rate - 0 Hz
Transactions started - 0 Hz
Transactions committed - 0 Hz
Conflict rate - 0 Hz
Performance limited by process: Storage server performance (storage queue).
Most limiting process: 127.0.0.1:4689
Backup and DR:
Running backups - 0
Running DRs - 0
Client time: 08/08/18 17:53:38