OpenTelemetry distributed tracing: is it possible to propagate trace IDs from client to server?

Using FoundationDB
1

I have been trying to figure out from documentation and practical tests how to pass an OpenTelemetry (parent) trace id from a client connected to FoundationDB, so that FoundationDB server would emit spans with the same trace ids and that client spans would become parent spans.

Is this even possible? There is no explicit mention in documentation or in the original announcement.

What I have tried so far:

  • using the SPAN_PARENT transaction option
  • writing the special keys \xff\xff/tracing/transaction_id and \xff\xff/tracing/token, in their expected format

I am also setting the --tracer parameter for fdbserver, and the corresponding network option for the client, plus knob_tracing_sample_rate=1.0, without any success.

I can see that client is receiving the trace id but it doesn’t seem to be sent to server as a SpanContextMessage mutation.

Any ideas? Thanks!

1 Like
2

I see you added documentation on the SPAN_PARENT transaction option. Thank you for that!

@gm42 Did you ever get FDB traces working end-to-end from Application → FDB client (libfdb.so) → FDB server?

Currently, I’m setting \xff\xff/tracing/token to b"true", and TransactionOption::SpanParent using the format described in your documentation. That results in a single Transaction span from FoundationDB (presumably libfdb.so), but nothing from the FDB server. I tried setting \xff\xff/tracing/transaction_id but it did not have an effect.

Code snippet for reference

fn mk_span_parent(span_ctx: &SpanContext) -> Vec<u8> {
    let mut parent_bytes = [0u8; 33];
    parent_bytes[0..8].copy_from_slice(&[0x00, 0x00, 0x00, 0x73, 0xB0, 0x00, 0xDB, 0x0F]);
    parent_bytes[8..24].copy_from_slice(&span_ctx.trace_id().to_bytes());
    parent_bytes[24..32].copy_from_slice(&span_ctx.span_id().to_bytes());
    parent_bytes[32] = u8::from(span_ctx.is_sampled());
    parent_bytes.to_vec()
}

fn bytes_to_hex(bytes: &[u8]) -> String {
    bytes.iter()
        .map(|b| format!("{:02x}", b))
        .collect::<String>()
}

// ... inside transaction closure ...

if trx.set_option(TransactionOption::SpecialKeySpaceEnableWrites).is_ok() {
    if trx
        .set_option(TransactionOption::SpanParent(mk_span_parent(trx_span.span_context())))
        .is_ok()
    {
        trx.set(b"\xff\xff/tracing/token", b"true");
        // trx.set(b"\xff\xff/tracing/transaction_id", bytes_to_hex(&trx_span.span_context().trace_id().to_bytes()).as_bytes());
    }
}

Example trace

Incomplete trace
Incomplete trace2708×234 16.1 KB

Interestingly enough, if I set the sample rate knob to 1.0 (begin; setknob tracing_sample_rate 1.0; commit "sample rate") , I see some full FDB server traces, but they’re not connected to any client spans, and are presumably for transactions initiated internally within FDB itself

Full trace
Full trace1786×800 63.6 KB

3

No; sorry for the late reply, I got pretty much at the same level of progress as you did.

Only detail I can add is that there is tracing information emitted by the server and tracing information emitted by the client-side (libfdbc.so), and I suppose that both must match for tracing spans to work.

One of the experiments I wanted to try was to use a different SDK (e.g. C++) to see if somehow it’s easier to make this work there.