feat(otel): scaffold OTel consistent probability sampling (ot.th/ot.rv) conformance tests

[genesor]

Motivation

RFC for cross-tracer support of OTel consistent probability sampling (ot.th/ot.rv) tracestate members across all 9 dd-trace-* SDKs (APMAPI-2171 distributed tracing, APMAPI-2172 trace export).

Changes

Scaffolding only — every new test class is landed disabled for all 9 tracers via manifest entries (missing_feature). No tracer implements this yet.

• tests/test_otel_tracestate_sampling.py (new): A1-A6 from APMAPI-2171
  • Test_EmitOtOnProbabilityDecision_Rate0_{01,1,2,5,99} — golden-vector rv/th derivation at 5 fixed rates, crossed with the 23 trace IDs from tests/fixtures/sampling_rates.csv, verified against the Go Knuth-hash sampler formula
  • Test_ForwardInboundOtUnchanged — inbound th/rv forwarded unchanged, never re-derived
  • Test_PreserveDdAndOtherVendors — ot= handling doesn't disturb dd= or unrelated vendor tracestate members
  • Test_ForceKeepClearsTh — a non-probability (force-keep, via ASM /waf) decision erases th but forwards an inherited rv
  • Test_SampledWithoutOtNotFabricated — an inherited sampling decision with no ot= must not fabricate th/rv
  • Test_MalformedOtHandling — malformed th/rv treated as absent, dd=/vendors preserved, trace never rejected
• tests/otel/test_tracing_otlp.py: B1/B2 from APMAPI-2172 — Test_Otlp_Carries_Ot, Test_Otlp_Forwards_Inherited_Ot, asserting on the exported OTLP span's traceState
• utils/_context/_scenarios/__init__.py: 4 new fixed-rate scenarios (otel_sampling_rate_0_{01,1,2,99}); rate 0.5 reuses the existing sampling scenario
• manifests/*.yml (all 9 languages): missing_feature (APMAPI-2171) / (APMAPI-2172) entries for every new class

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team (note: utils/_context/_scenarios/__init__.py is modified to add the 4 new scenarios)
• A docker base image is modified? N/A
• A scenario is added, removed or renamed? Yes — 4 new scenarios added, needs R&P team review

[github-actions]

CODEOWNERS have been resolved as:

tests/test_otel_tracestate_sampling.py                                  @DataDog/system-tests-core
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
manifests/c.yml                                                         @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/cpp_httpd.yml                                                 @DataDog/dd-trace-cpp
manifests/cpp_kong.yml                                                  @DataDog/system-tests-core
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
tests/otel/test_tracing_otlp.py                                         @DataDog/system-tests-core
tests/parametric/test_headers_precedence.py                             @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core

[datadog-datadog-prod-us1-2]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 84fa876 | Docs | Datadog PR Page | Give us feedback!}

[vpellan]

I also believe that we should add tests for imprecision workaround while going from 64 to 56 bit (see the annex of the RFC about 64->56 bit imprecision)

[vpellan]

For these cases, as we generate new th and rv, should th be derived from the sampling rate, and rv from the trace id last 56 bits?

[genesor]

No, we do not do anything. We have no way of knowing how the decision was made upstream. We just to cleanup

[vpellan]

There are a bunch of tests intests/parametric/test_headers_precedence.py that needs to be fixed, because they were checking that we had exactly 1 vendor in the tracestate which is now not true anymore, and I believe we should do it on this PR as well.
Basically in all these tests we need to change

assert len(tracestate_1_arr) == 1
assert tracestate_1_arr[0].startswith("dd=")

into

assert any(item.startswith("dd=") for item in tracestate_1_arr)

The test are:

• test_headers_precedence_propagationstyle_tracecontext_datadog: Only part 1, with no headers
• test_headers_precedence_propagationstyle_tracecontext: Part 1, with no headers, but also part 5, with Only Datadog headers, as the datadog headers should be ignored because DD_TRACE_PROPAGATION_STYLE is set to tracecontext only. I believe for 6 too, with an invalid tracecontext, we should generate a new trace id, which means we're the root span, and we should add ot to the tracestate
• test_headers_precedence_propagationstyle_datadog_tracecontext: Only part 1, with no headers as well

[genesor]

Good catch, fixed in 52e0d3a. Updated the 5 assertions across the 3 tests to assert any(item.startswith("dd=") for item in tracestate_arr).

Part 6 already falls into the "new Datadog span context" path, so the same fix covers it.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: efe7f14b08

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".