FDB initialization timeouts in application

Using FoundationDB
1

Hi

We have multiple applications that write/fetch data from FDB and we have recently started seeing RecordCoreException timeout exceptions in those applications. We use a MetaDataVersionStampStoreStateCacheFactory and would like to understand the cause for this increase in timeouts.

FDB Versions used:

api(“org.foundationdb:fdb-java:6.1.8”)
api(“org.foundationdb:fdb-record-layer-core:2.8.110.0”)
api(“org.foundationdb:fdb-extensions:2.8.110.0”)

FoundationDB 6.2 (v6.2.15)
source version 20566f2ff06a7e822b30e8cfd91090fbd863a393
protocol fdb00b062010001

Error stack trace:

ERROR  [00:21:38.423] [Timer-0] i.b.s.i.p.IPStorePurgeDao -  Error scanning index records: {} java.util.concurrent.CompletionException: com.apple.foundationdb.record.RecordCoreException
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:769)
	at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778)
	at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140)
	at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.runAsync(FDBDatabase.java:864)
	at io.branch.service.ip_store.purge.IPStorePurgeDao.scanRecordsBeforeTimestamp(IPStorePurgeDao.java:109)
	at io.branch.service.ip_store.purge.IPStorePurgeJob.scanAndPurgeRecords(IPStorePurgeJob.java:77)
	at io.branch.service.ip_store.purge.IPStorePurgeJob.access$000(IPStorePurgeJob.java:19)
	at io.branch.service.ip_store.purge.IPStorePurgeJob$1.run(IPStorePurgeJob.java:44)
Caused by: com.apple.foundationdb.record.RecordCoreException: null
	at com.apple.foundationdb.record.provider.foundationdb.FDBExceptions.wrapException(FDBExceptions.java:199)
	at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.lambda$new$0(FDBDatabase.java:162)
	at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.asyncToSync(FDBDatabase.java:1002)
	at com.apple.foundationdb.record.provider.foundationdb.FDBRecordContext.asyncToSync(FDBRecordContext.java:986)
	at com.apple.foundationdb.record.provider.foundationdb.FDBRecordStoreBase$BaseBuilder.createOrOpen(FDBRecordStoreBase.java:1948)
	at io.branch.service.persona.fdb.IPRecordStore.lambda$getIPRecordStore$0(IPRecordStore.java:53)
	at io.branch.service.persona.fdb.IPRecordStore.getFDBRecordStore(IPRecordStore.java:46)
	at io.branch.service.ip_store.purge.IPStorePurgeDao.lambda$scanRecordsBeforeTimestamp$2(IPStorePurgeDao.java:114)
	at com.apple.foundationdb.record.provider.foundationdb.FDBDatabaseRunnerImpl$RunRetriable.lambda$runAsync$4(FDBDatabaseRunnerImpl.java:331)
	at com.apple.foundationdb.async.AsyncUtil$LoopPartial.apply(AsyncUtil.java:355)
Caused by: java.util.concurrent.TimeoutException: null
	at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
	at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.asyncToSync(FDBDatabase.java:993)
	at com.apple.foundationdb.record.provider.foundationdb.FDBRecordContext.asyncToSync(FDBRecordContext.java:986)
	at com.apple.foundationdb.record.provider.foundationdb.FDBRecordStoreBase$BaseBuilder.createOrOpen(FDBRecordStoreBase.java:1948)
	at io.branch.service.persona.fdb.IPRecordStore.lambda$getIPRecordStore$0(IPRecordStore.java:53)
	at io.branch.service.persona.fdb.IPRecordStore.getFDBRecordStore(IPRecordStore.java:46)
	at io.branch.service.ip_store.purge.IPStorePurgeDao.lambda$scanRecordsBeforeTimestamp$2(IPStorePurgeDao.java:114)
	at com.apple.foundationdb.record.provider.foundationdb.FDBDatabaseRunnerImpl$RunRetriable.lambda$runAsync$4(FDBDatabaseRunnerImpl.java:331)
	at com.apple.foundationdb.async.AsyncUtil$LoopPartial.apply(AsyncUtil.java:355)
3

Just for those looking at this post in the future, I believe this was found to be a networking issue between the FDB client and the FDB server which resulted in these timeouts (if this is the same as FDB TimeoutExceptions degrading application performance - #2 by ajames).