Add CI Visibility instrumentation for JMH benchmarks#11498
Draft
robertpi wants to merge 4 commits into
Draft
Conversation
Instruments JMH's Runner constructor to wrap its OutputFormat with a DDOutputFormat decorator. The decorator fires once per benchmark method (after all forks and iterations complete) to emit CI Visibility test spans — zero overhead on the benchmark hot path. Each benchmark method produces a suite span + test span with benchmark metrics (score, error, unit, percentiles, run config) attached as tags. Parameterised @Param benchmarks follow the same test.parameters convention as JUnit 5 parameterized tests. Changes: - New module: dd-java-agent/instrumentation/jmh/jmh-1.0 - Tags.java: add benchmark.* tag constants - TestFrameworkInstrumentation: add JMH enum value - TestDecorator: add TEST_TYPE_BENCHMARK constant - Design spec: docs/design/jmh-ci-visibility.md Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Groovy/Spock integration tests extending CiVisibilityInstrumentationTest that run JMH benchmarks in-process (forks=0) and verify the emitted CI Visibility spans against FTL fixture templates. Covers: - Simple (unparameterized) benchmark: suite + test spans with benchmark run config metrics (mode, unit, iterations, forks, threads, time_unit) - Parameterised benchmark (@Param): two test spans with test.parameters set following the JUnit 5 convention Also fixes: - BaseRunner instrumented instead of Runner (JDK 17+ rejects PUTFIELD on a final field of a superclass from advice injected into the subclass) - JMH annotation processor added to testAnnotationProcessor so that META-INF/BenchmarkList is generated at test compile time - DD_TRACE_JMH_ENABLED registered in supported-configurations.json Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Java JUnit 5 smoke test that forks a real JVM subprocess with the dd-java-agent attached, runs a JMH benchmark in-process (forks=0) against a MockBackend, and verifies that the expected CI Visibility spans arrive with correct tags: - test.framework = "jmh" - test.name, test.suite, test.status - benchmark.run.mode, benchmark.unit - benchmark.value > 0 (measured score actually present) The benchmark class (SmokeTestBenchmark) lives in src/main/java so the JMH annotation processor can generate META-INF/BenchmarkList at compile time, making it available on the classpath that is passed to the subprocess. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
1. splitBenchmarkName returned the full parameterised suffix as the test
name (e.g. "myMethod:size=1000") instead of just the method name
("myMethod"). Fix: use baseName (param-stripped) for the method slice.
2. endBenchmark had no null guard — if called without a prior
startBenchmark the handler would receive null keys. Fix: early-return
when suiteKey/testKey are null.
3. handler.close() in endRun was not in a finally block, so a crash in
close() would swallow delegate.endRun(); and an exception in
endBenchmark could bypass close() entirely. Fix: try/finally in both
endBenchmark and endRun.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
robertpi
added
type: feature request
comp: ci visibility
Contributor
|
robertpi
added
the
tag: override-groovy-enforcement
Member
Author
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds CI Visibility support for JMH (Java Microbenchmark Harness,
org.openjdk.jmh), the dominant Java microbenchmarking framework. Benchmark runs are now reported as test spans in the Datadog test explorer with performance metrics attached.Closes SDTEST-930.
How it works
JMH's
OutputFormatinterface receives lifecycle callbacks exactly once per benchmark method. We instrumentBaseRunner.<init>with bytecode advice to wrap the user'sOutputFormatwith ourDDOutputFormatdecorator — this is the only hook needed, with zero overhead on the benchmark hot path.Each benchmark method produces:
test_suite_end) for the benchmark classtest) for the benchmark methodWith benchmark-specific metric tags on the test span:
benchmark.valuebenchmark.errorbenchmark.unit"ns/op","ops/ms"benchmark.run.mode"avgt","thrpt"benchmark.run.iterationsbenchmark.run.warmup_iterationsbenchmark.run.forksbenchmark.run.threadsbenchmark.run.time_unit"NANOSECONDS"benchmark.p50/p90/p95/p99benchmark.min/benchmark.maxbenchmark.sample_count@Param-parameterised benchmarks follow the sametest.parametersconvention as JUnit 5 parameterized tests:{"metadata":{"test_name":"myMethod:size=1000"}}.Changes
dd-java-agent/instrumentation/jmh/jmh-1.0/— instrumentation + integration tests + fixture templatesdd-smoke-tests/jmh/— end-to-end smoke test with real agentTags.java— 17 newbenchmark.*tag constantsTestFrameworkInstrumentation— newJMHenum valueTestDecorator— newTEST_TYPE_BENCHMARKconstant (for future use)supported-configurations.json—DD_TRACE_JMH_ENABLEDregisteredNote on integration test style
JmhInstrumentationTestis a Groovy/Spock test extendingCiVisibilityInstrumentationTest. This is an intentional exception to the JUnit 5 convention:CiVisibilityInstrumentationTestis a SpockSpecificationsubclass whosesetup()/cleanup()lifecycle cannot be triggered by the JUnit 5 runner. All other CI Visibility instrumentation tests in the codebase follow this same pattern.Test plan
./gradlew :dd-java-agent:instrumentation:jmh:jmh-1.0:test— unit tests (JmhUtilsTest) + integration tests (simple benchmark, parameterised benchmark via fixture comparison)./gradlew :dd-smoke-tests:jmh:test— end-to-end smoke test: forks a JVM with the agent, runs a JMH benchmark, asserts span tags andbenchmark.value > 0🤖 Generated with Claude Code