Wrong rank index when concurrently adding records

Using Layers Record Layer
1

There are some unexpected behaviors of rank indexing when adding records concurrently. I’m having a record that has rank index. When i try to adding multiple records concurrently, there is something wrong with ranking. I tried to scan all records on the store, it has all records but using scoreForRank and rankForScore gave me wrong result.

I’m trying to re-procedure my use case on RankIndexTest.java (sorry, i’m not really familiar with java).
I change the loadRecords to concurrently load records

public static class RunnableSaveRecord implements Runnable {
        private final String name;
        private final Integer score;
        private final String sex;
        private final FDBRecordStore recordStore;
        private final CountDownLatch cd;

        RunnableSaveRecord(String _name, Integer _score, String _sex, FDBRecordStore _store, CountDownLatch _cd) {
            this.name = _name;
            this.score = _score;
            this.sex = _sex;
            this.recordStore = _store;
            this.cd = _cd;
        }

        @Override
        public void run() {
            System.out.printf("------------ write %s %d %s", name, score, sex);
            recordStore.saveRecord(TestRecordsRankProto.BasicRankedRecord.newBuilder()
                    .setName(name)
                    .setScore(score)
                    .setGender(sex)
                    .build());
            cd.countDown();
        }

    }

    @BeforeEach
    public void loadRecords() throws Exception {
        CountDownLatch cd = new CountDownLatch(RECORDS.length);
        ExecutorService executor = Executors.newFixedThreadPool(RECORDS.length);
        try (FDBRecordContext context = openContext()) {
            openRecordStore(context);
            for (Object[] rec : RECORDS) {
                RunnableSaveRecord saver = new RunnableSaveRecord((String)rec[0], (Integer)rec[1], (String)rec[2], recordStore, cd);
                executor.execute(saver);
            }
            cd.await();
            commit(context);
        }
    }

With the code above, test will be failed. (Randomly, i think it’s because of concurrency).

1 Like
2

Unfortunately, this is a known issue with not a great solution, at at the moment.

It looks like we don’t have an issue filed about this at the moment, but it is the root cause of: https://github.com/FoundationDB/fdb-record-layer/issues/482 , and there’s a somewhat oblique reference to this in this comment: https://github.com/FoundationDB/fdb-record-layer/pull/914#discussion_r413906868

The root cause is that the underlying data structure that the rank index is not thread safe, so it is not safe to insert two records which touch the rank index at the same time in the same transaction. Note that you can have as much concurrency as you’d like if the different updates use different transactions (that’s completely safe), but within the same transaction, the objects share some state in an unsafe way.

The difficulty here, though, is that we don’t want to block any threads waiting for async work, so the only way to “actually” solve this is to introduce some kind of async queueing or locking mechanism. I’ve thought about how we might want to do this at some point, but it’s not a trivial thing to implement.

So, in the meantime, the only solution is to insert your records in serial, unfortunately. In practice, our solution would be to serialize writes to the index, which seems like something the library should handle correctly for you, but it doesn’t at the moment.

Filed: https://github.com/FoundationDB/fdb-record-layer/issues/972

2 Likes