OpenTelemetry distributed tracing: is it possible to propagate trace IDs from client to server?

[gm42 (Giuseppe)]

I have been trying to figure out from documentation and practical tests how to pass an OpenTelemetry (parent) trace id from a client connected to FoundationDB, so that FoundationDB server would emit spans with the same trace ids and that client spans would become parent spans.

Is this even possible? There is no explicit mention in documentation or in the original announcement.

What I have tried so far:

• using the SPAN_PARENT transaction option
• writing the special keys \xff\xff/tracing/transaction_id and \xff\xff/tracing/token, in their expected format

I am also setting the --tracer parameter for fdbserver, and the corresponding network option for the client, plus knob_tracing_sample_rate=1.0, without any success.

I can see that client is receiving the trace id but it doesn’t seem to be sent to server as a SpanContextMessage mutation.

Any ideas? Thanks!

[ArniDagur (Árni Dagur)]

I see you added documentation on the SPAN_PARENT transaction option. Thank you for that!

@gm42 Did you ever get FDB traces working end-to-end from Application → FDB client (libfdb.so) → FDB server?

Currently, I’m setting \xff\xff/tracing/token to b"true", and TransactionOption::SpanParent using the format described in your documentation. That results in a single Transaction span from FoundationDB (presumably libfdb.so), but nothing from the FDB server. I tried setting \xff\xff/tracing/transaction_id but it did not have an effect.

Code snippet for reference

fn mk_span_parent(span_ctx: &SpanContext) -> Vec<u8> {
    let mut parent_bytes = [0u8; 33];
    parent_bytes[0..8].copy_from_slice(&[0x00, 0x00, 0x00, 0x73, 0xB0, 0x00, 0xDB, 0x0F]);
    parent_bytes[8..24].copy_from_slice(&span_ctx.trace_id().to_bytes());
    parent_bytes[24..32].copy_from_slice(&span_ctx.span_id().to_bytes());
    parent_bytes[32] = u8::from(span_ctx.is_sampled());
    parent_bytes.to_vec()
}

fn bytes_to_hex(bytes: &[u8]) -> String {
    bytes.iter()
        .map(|b| format!("{:02x}", b))
        .collect::<String>()
}

// ... inside transaction closure ...

if trx.set_option(TransactionOption::SpecialKeySpaceEnableWrites).is_ok() {
    if trx
        .set_option(TransactionOption::SpanParent(mk_span_parent(trx_span.span_context())))
        .is_ok()
    {
        trx.set(b"\xff\xff/tracing/token", b"true");
        // trx.set(b"\xff\xff/tracing/transaction_id", bytes_to_hex(&trx_span.span_context().trace_id().to_bytes()).as_bytes());
    }
}

Example trace

Interestingly enough, if I set the sample rate knob to 1.0 (begin; setknob tracing_sample_rate 1.0; commit "sample rate") , I see some full FDB server traces, but they’re not connected to any client spans, and are presumably for transactions initiated internally within FDB itself