Skip to content

fix(live-debugger): fix probe hook use-after-free and tags leak on config removal#4036

Open
Leiyks wants to merge 4 commits into
masterfrom
leiyks/fix-live-debugger-shared-probe-id
Open

fix(live-debugger): fix probe hook use-after-free and tags leak on config removal#4036
Leiyks wants to merge 4 commits into
masterfrom
leiyks/fix-live-debugger-shared-probe-id

Conversation

@Leiyks

@Leiyks Leiyks commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Two related live-debugger probe-lifetime bugs surfaced from a flaky LEAKED in test_extension_ci on debugger_enable_dynamic_config.phpt.

1. Use-after-free (the original failure)

spans_map (installed hook → id) was keyed inconsistently: apply_config/remove_config used probe.id, while the DI-enable reinstall path used config_id (the active key). config_id is unique per RC path; probe.id is not. When a probe's hook is tracked under one key but looked up for removal under another (or two configs share a probe.id), removal misses → the hook is orphaned → its parsed-config box is freed → the orphan's next fire reads the freed probe.id in ddog_debugger_diagnostics_create_unboxedUAF. Only valgrind observes it (plain runs read freed-but-intact bytes), hence the intermittent LEAKED. Reproduced identically on PHP 7.1 and 8.3 (not version-specific).

Fix: key spans_map by config_id everywhere, so every config's hook is tracked and removed independently.

2. Probe tags leak

probe.into() heap-allocates a CharSliceVec for the probe tags, but dd_probe_dtor only freed the nested span-decoration / log allocations — never the tags vec. So any probe carrying tags leaked it (LeakSanitizer: 32 B / 2 objects in the ASAN "multiple observers" job).

Fix: add ddog_drop_probe, which consumes the FFI probe by value so its drop glue frees the tags CharSliceVec together with the nested allocations; dd_probe_dtor calls it in place of the piecemeal drops (one consistent cleanup).

Test

tests/ext/live-debugger/debugger_remove_shared_probe_id.phpt installs two configs sharing a probe.id (with distinct tags), removes both without firing, then calls the target — exercising both the UAF (orphaned hook) and the tags allocation. It leaks/UAFs before the fixes and is clean after.

Verification (under valgrind, PHP 7.1 & 8.3)

  • Regression test: reproduces the UAF before the fix, passes after.
  • Whole live-debugger suite: clean (no UAF, no double-free).
  • valgrind --leak-check=full on a tagged probe: 0 bytes definitely lost, no CharSliceVec leak record.

…use-after-free

The live debugger tracks installed probe hooks in `spans_map`, but keyed it
inconsistently: apply_config and remove_config used the probe's `id` field,
while the DI-enable reinstall path used the `active` map key (the config_id).

`active` is keyed by config_id, which is unique per remote-config path, whereas
probe.id is not: two distinct configs can carry the same probe id. When they do
(or when a probe is installed via different paths), the second install overwrites
the probe.id-keyed spans_map entry, orphaning the first hook. Removing a config
then tears down the wrong hook (or none) and frees the parsed-config box while an
orphaned hook still borrows its strings. The orphan's first fire afterwards reads
the freed probe id in ddog_debugger_diagnostics_create_unboxed -> use-after-free.

Only valgrind observes it (plain runs read freed-but-intact bytes), which is why
it surfaced intermittently as a LEAKED test in test_extension_ci.

Key spans_map by config_id everywhere (apply_config, remove_config, and the
already-correct reinstall path) so every config's hook is tracked and removed
independently. Adds tests/ext/live-debugger/debugger_remove_shared_probe_id.phpt,
which reproduces the UAF (two configs sharing a probe id) and is clean with the fix.

Verified under valgrind on PHP 7.1 and 8.3: the reproducer leaks before the fix
and passes after; the whole live-debugger suite is clean.
@Leiyks Leiyks requested review from a team as code owners July 8, 2026 13:02
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jul 8, 2026

Copy link
Copy Markdown

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 14 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-php | PHP language tests: [8.3, amd64, zts]   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-php | check-big-regressions   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-php | merge-gate   View in Datadog   GitLab

View all 14 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 121 jobs - 95 passed on retry View in Datadog

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 54.08% (+0.00%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: cef3cd0 | Docs | Datadog PR Page | Give us feedback!

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2d4ab496dd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread components-rs/remote_config.rs
Comment thread tests/ext/live-debugger/debugger_remove_shared_probe_id.phpt Outdated
Leiyks added 2 commits July 8, 2026 15:59
The regression test used a distinct `tags` entry per config only to give the two
configs different remote-config paths. But a non-empty `tags` makes probe.into()
allocate an FFI CharSliceVec that the C side never frees (a separate latent leak
for tagged probes), which LeakSanitizer flags in the ASAN 'multiple observers'
job (32 bytes / 2 objects) and breaks the test's expected output.

Use an ignored unknown field instead: same two distinct config paths, same probe
id, no per-config allocation -> no unrelated LSan noise. Verified under valgrind
(clean) and the two configs still install at distinct paths with the shared id.
probe.into() heap-allocates a CharSliceVec for the probe `tags`, but the C side
(def->probe) only freed the nested span-decoration / log allocations in
dd_probe_dtor -- never the tags vec -- so every probe carrying tags leaked it
(LeakSanitizer: 32 bytes / 2 objects in the ASAN 'multiple observers' job).

Add ddog_drop_probe, which consumes the FFI probe by value so its drop glue
frees the tags CharSliceVec together with the nested span-decoration / log
allocations, and call it from dd_probe_dtor in place of the piecemeal drops.
Cleanup is now a single, consistent operation covering every FFI-owned field.

Restore the shared-probe-id regression test to distinguish its two configs by
`tags`, so it also exercises this path (a tags leak would resurface under LSan).

Verified on PHP 8.3: the live-debugger suite is clean under valgrind (no UAF,
no double-free), and valgrind --leak-check=full on a tagged probe shows 0 bytes
definitely lost with no CharSliceVec leak record.
@Leiyks Leiyks changed the title fix(live-debugger): key spans_map by config_id to prevent probe hook use-after-free fix(live-debugger): fix probe hook use-after-free and tags leak on config removal Jul 8, 2026
…; trim comments

Address Codex review:
- apply_config: remove any hook already installed for a config id before
  installing the replacement, so an in-place Add (Occupied entry) can't orphan
  the previous hook into the dropped parsed config.
- regression test: await both colliding probes (2) before removing, so it
  actually exercises the shared-probe-id collision.

Also condense the comments added in this PR.
@pr-commenter

pr-commenter Bot commented Jul 8, 2026

Copy link
Copy Markdown

Benchmarks [ tracer ]

Benchmark execution time: 2026-07-08 16:34:27

Comparing candidate commit cef3cd0 in PR branch leiyks/fix-live-debugger-shared-probe-id with baseline commit 7e61dc9 in branch master.

Found 0 performance improvements and 2 performance regressions! Performance is the same for 192 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:MessagePackSerializationBench/benchMessagePackSerialization

  • 🟥 execution_time [+8.336µs; +9.864µs] or [+7.873%; +9.315%]

scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache

  • 🟥 execution_time [+12.162µs; +13.158µs] or [+11.479%; +12.419%]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant