Skip to content

test(rust): enable passing ot.th/ot.rv tracestate sampling scenarios [rust@milan.garnier/otel-tracestate-ot-th] - #7391

Draft
MilanGarnier wants to merge 24 commits into
mainfrom
milan.garnier/feat-rs-ot.th
Draft

test(rust): enable passing ot.th/ot.rv tracestate sampling scenarios [rust@milan.garnier/otel-tracestate-ot-th]#7391
MilanGarnier wants to merge 24 commits into
mainfrom
milan.garnier/feat-rs-ot.th

Conversation

@MilanGarnier

@MilanGarnier MilanGarnier commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Motivation

Stacked on #7372 (the ot.th/ot.rv conformance scaffolding). Enables the Rust entries against dd-trace-rs's wire implementation (branch milan.garnier/otel-tracestate-ot-th), verified locally against a rust@0.5.1-dev+c36937d build (axum weblog) across every scenario (DEFAULT + all five sample-rate scenarios).

What does this PR do?

Flips all tests/test_otel_tracestate_sampling.py Rust entries from missing_feature to >=0.5.0-dev in manifests/rust.yml, and backfills 3 previously-unlisted classes (Test_ForwardInboundOtUnchangedWhenDropped, Test_ThOnlyDoesNotFabricateRv, Test_ThOnlyDoesNotFabricateRvWhenDropped) as missing_feature across the other 8 language manifests to preserve the "disabled for every tracer until it ships" invariant. OTLP entries (test_tracing_otlp.py, APMAPI-2172) are untouched per scope.

Local verification results (ai generated draft not reviewed)

  • DEFAULT scenario: 6/9 selected tests pass. The 3 Test_ForceKeepClearsTh (A4) sub-tests fail, but not because of an ot.th gap: the local weblog used for verification (a not-yet-merged Rust Axum weblog) has no AppSec/WAF integration, so the Arachni-user-agent-triggered force-keep this test relies on never fires. The underlying "clear th, forward inherited rv" wire logic it exercises is already unit-tested and passing (tracecontext.rs::force_keep_clears_th_forwards_rv).
  • Rate scenarios: only SAMPLING (rate 0.5) passes outright. OTEL_SAMPLING_RATE_0_01/0_1/0_2/0_99 all fail on a th mismatch of a few ULPs against TH_BY_RATE — two independent rounding bugs, not one:
    • TH_BY_RATE[0.1] in the test fixture itself is stale (e6666666666668); the exact-fraction value is e6666666666666, which matches both what Rust emits and dd-trace-rs's own corrected example.
    • For rate 0.01, Rust emits one ULP above the exact-fraction floor (fd70a3d70a3d71 vs fd70a3d70a3d70) — a rounding discrepancy in the unreleased libdd-sampling::OtelConsistentSampling threshold computation dd-trace-rs depends on, not in dd-trace-rs itself.

Both issues need a fix (one in this repo's fixture, one upstream in libdatadog) before these entries are actually green. Opening as draft now since the manifest/scope work itself is done and the base branch is expected to move; will flip out of draft once the rounding fixes land and a clean local rerun confirms all scenarios pass.

Base

Targets ben.db/feat-otel-tracestate-sampling-scaffolding (stacked). Retarget to main once #7372 merges.

genesor and others added 10 commits July 23, 2026 11:12
Stacked on #7372 (the ot.th/ot.rv conformance scaffolding). Flips the
Rust entries in manifests/rust.yml from missing_feature to
'>=0.5.0-dev' and backfills the 3 previously-unlisted classes
(Test_ForwardInboundOtUnchangedWhenDropped, Test_ThOnlyDoesNotFabricateRv,
Test_ThOnlyDoesNotFabricateRvWhenDropped) as missing_feature across the
other 8 manifests to keep the "disabled for every tracer" invariant
intact until each ships. OTLP entries (APMAPI-2172) are left untouched.
@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

manifests/rust.yml                                                      @DataDog/apm-rust

@datadog-official

datadog-official Bot commented Jul 27, 2026

Copy link
Copy Markdown

Pipelines  Tests

⚠️ Warnings

🚦 5 Pipeline jobs failed

Testing the test | System Tests (rust, prod) / End-to-end #1 / axum 1   View in Datadog   GitHub Actions

🔧 Fix in code. This looks caused by changes in this PR. Test run failed due to missing 'ot' tracestate member on probability decisions and unexpected 'th' values.

🧪 7 Tests failed · 🎯 related to the job error above

🎯 tests.test_otel_tracestate_sampling.Test_EmitOtOnProbabilityDecision_Rate0_01.test_emit_ot_on_probability_decision[axum] from system_tests_suite   View in Datadog
AssertionError: trace_id=1: no ot= tracestate member emitted on a probability decision
assert 'ot' in Tracestate(dd=s:-1;p:8133ffb164d19032;t.dm:-3)

self = <tests.test_otel_tracestate_sampling.Test_EmitOtOnProbabilityDecision_Rate0_01 object at 0x7ff56566c560>

    def test_emit_ot_on_probability_decision(self):
        for req, trace_id, expected_rv, expected_th, expected_sampled in self.requests:
            assert req.status_code == 200, f"trace_id={trace_id}: /make_distant_call failed"
    
            tracestate = _outbound_tracestate(req)
...
🎯 tests.test_otel_tracestate_sampling.Test_EmitOtOnProbabilityDecision_Rate0_1.test_emit_ot_on_probability_decision[axum] from system_tests_suite   View in Datadog
AssertionError: trace_id=1: no ot= tracestate member emitted on a probability decision
assert 'ot' in Tracestate(dd=s:2;p:3f64fd37061f33d5;t.dm:-3)

self = <tests.test_otel_tracestate_sampling.Test_EmitOtOnProbabilityDecision_Rate0_1 object at 0x7fc774d71a30>

    def test_emit_ot_on_probability_decision(self):
        for req, trace_id, expected_rv, expected_th, expected_sampled in self.requests:
            assert req.status_code == 200, f"trace_id={trace_id}: /make_distant_call failed"
    
            tracestate = _outbound_tracestate(req)
...
🎯 tests.test_otel_tracestate_sampling.Test_EmitOtOnProbabilityDecision_Rate0_2.test_emit_ot_on_probability_decision[axum] from system_tests_suite   View in Datadog
AssertionError: trace_id=1: no ot= tracestate member emitted on a probability decision
assert 'ot' in Tracestate(dd=s:2;p:d9fd7ef6edde749f;t.dm:-3)

self = <tests.test_otel_tracestate_sampling.Test_EmitOtOnProbabilityDecision_Rate0_2 object at 0x7f514c86d9d0>

    def test_emit_ot_on_probability_decision(self):
        for req, trace_id, expected_rv, expected_th, expected_sampled in self.requests:
            assert req.status_code == 200, f"trace_id={trace_id}: /make_distant_call failed"
    
            tracestate = _outbound_tracestate(req)
...
View all failed tests

Testing the test | System Tests (rust, prod) / End-to-end #2 / axum 2   View in Datadog   GitHub Actions

🔧 Fix in code. This looks caused by changes in this PR. Test run failed due to assertion errors related to sampling priority mismatches and missing trace state members.

🧪 5 Tests failed

tests.test_otel_tracestate_sampling.Test_EmitOtOnProbabilityDecision_Rate0_5.test_emit_ot_on_probability_decision[axum] from system_tests_suite   View in Datadog
AssertionError: trace_id=1: no ot= tracestate member emitted on a probability decision
assert 'ot' in Tracestate(dd=s:2;p:7959353d0c28d299;t.dm:-3)

self = <tests.test_otel_tracestate_sampling.Test_EmitOtOnProbabilityDecision_Rate0_5 object at 0x7fdfc4081d00>

    def test_emit_ot_on_probability_decision(self):
        for req, trace_id, expected_rv, expected_th, expected_sampled in self.requests:
            assert req.status_code == 200, f"trace_id={trace_id}: /make_distant_call failed"
    
            tracestate = _outbound_tracestate(req)
...
tests.test_otel_tracestate_sampling.Test_EmitOtOnProbabilityDecision_Rate0_99.test_emit_ot_on_probability_decision[axum] from system_tests_suite   View in Datadog
AssertionError: trace_id=1: no ot= tracestate member emitted on a probability decision
assert 'ot' in Tracestate(dd=s:2;p:077ce93d1d290336;t.dm:-3)

self = <tests.test_otel_tracestate_sampling.Test_EmitOtOnProbabilityDecision_Rate0_99 object at 0x7f5268a6a120>

    def test_emit_ot_on_probability_decision(self):
        for req, trace_id, expected_rv, expected_th, expected_sampled in self.requests:
            assert req.status_code == 200, f"trace_id={trace_id}: /make_distant_call failed"
    
            tracestate = _outbound_tracestate(req)
...
tests.test_otel_tracestate_sampling.Test_ForceKeepClearsTh.test_force_keep_forwards_inherited_rv[axum] from system_tests_suite   View in Datadog
assert 0.0 == <SamplingPriority.USER_KEEP: 2>
 +  where 0.0 = <bound method DataDogLibrarySpanLegacy.get_sampling_priority of <utils.dd_types._datadog_library_trace.DataDogLibrarySpanLegacy object at 0x7f39486f7f80>>()
 +    where <bound method DataDogLibrarySpanLegacy.get_sampling_priority of <utils.dd_types._datadog_library_trace.DataDogLibrarySpanLegacy object at 0x7f39486f7f80>> = <utils.dd_types._datadog_library_trace.DataDogLibrarySpanLegacy object at 0x7f39486f7f80>.get_sampling_priority
 +  and   <SamplingPriority.USER_KEEP: 2> = SamplingPriority.USER_KEEP

self = <tests.test_otel_tracestate_sampling.Test_ForceKeepClearsTh object at 0x7f396661b740>

    def test_force_keep_forwards_inherited_rv(self):
        """An inherited rv (no th, so no upstream probability decision) is still forwarded on a local force-keep."""
        assert self.rv_only_request.status_code == 200
...
View all failed tests

Testing the test | Fail if target branch is specified   View in Datadog   GitHub Actions

See error Job failed because the PR title specifies a target branch which is not allowed.

View all 5 failed jobs.

📋 Copy prompt for your agent
CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: milan.garnier/feat-rs-ot.th

Testing the test | System Tests (rust, prod) / End-to-end #1 / axum 1
Commit: 6eae59bee2c250bfbc2abf8a52f196fea70bf772
Error (code / test):
4 failed tests due to missing 'ot' tracestate member on probability decisions and unexpected 'th' values.
CI job: https://github.com/DataDog/system-tests/actions/runs/30643052656/job/91198729430

Testing the test | System Tests (rust, prod) / End-to-end #2 / axum 2
Commit: 6eae59bee2c250bfbc2abf8a52f196fea70bf772
Error (code / test):
3 failed tests due to assertion errors related to sampling priority mismatches and missing trace state members.
CI job: https://github.com/DataDog/system-tests/actions/runs/30643052656/job/91198729415

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: de49294 | Docs | Datadog PR Page | Give us feedback!

@MilanGarnier MilanGarnier changed the title test(rust): enable passing ot.th/ot.rv tracestate sampling scenarios test(rust): enable passing ot.th/ot.rv tracestate sampling scenarios [rust@milan.garnier/otel-consistent-sampling] Jul 28, 2026
@MilanGarnier MilanGarnier changed the title test(rust): enable passing ot.th/ot.rv tracestate sampling scenarios [rust@milan.garnier/otel-consistent-sampling] test(rust): enable passing ot.th/ot.rv tracestate sampling scenarios [rust@milan.garnier/otel-tracestate-ot-th] Aug 3, 2026
Base automatically changed from ben.db/feat-otel-tracestate-sampling-scaffolding to main August 3, 2026 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants