Metric Collection for Record Layer

Using Layers Record Layer
1

I haven’t found any documentation for APIs which exposes the different metrics collected by FoundationDB Record Layer. The only source for metrics I have discovered so far is the FDB Status json file.

So, for Record Layer specifically, are there APIs which exposes the metrics? Or if not this, are there any third party tools which can help with the same?

2

Hi Mukul, record layer exposes metrics through the StoreTimer interface. When creating a FDBRecordContext, if you provide an instance of this interface, all subsequent operations performed through the context will be instrumented as counters for events in the timer: There is a provided implementation of StoreTimer called FDBStoreTimer that provides the event definitions for the operations performed within the record layer itself:

FDBRecordContext context = fdbDatabase.openContext(null /* no mdcContext */, new FDBStoreTimer());

at any point (typically after, say, a commit) you can fetch the set of events that have been collected and do what you want with them, like so:

final StoreTimer timer = context.getTimer();
for (Event event : timer.getEvents()) {
    String name = event.toString();
    int count = timer.getCount(event);
    long durationNanos = timer.getTimeNanos();
}

Events are subclassed to indicate what they convey. A Count event is a pure counter with no time component, and has an attribute isSize() to indicate if it is a measure of something, like bytes read. A raw Event has a count and duration. A DetailEvent is very low level detail that you normally wouldn’t want to log, and a Wait is an event that is used during asyncToSync() operations to time how long you waited for the operation to complete.

It is expected that most applications should extend FDBStoreTimer to introduce their own metrics as well, for example if you dispatch multiple operations in parallel for a record store, then wait on them, you may want to introduce your own custom Wait to track that.

You can find documentation for the existing Events in FDBStoreTimer (granted, we could probably use more documentation in this area).

3

I was wondering if there is information available on record layer metrics that we should consider monitoring in production?

Something similar to Monitored Metrics, but for Record Layer?

1 Like