Telemetry for OTLP traces/metrics/logs#12057
Conversation
mcculls
added
type: feature
|
@codex review |
There was a problem hiding this comment.
Pull request overview
This PR adds telemetry coverage for OTLP-based exporting across traces, metrics, and logs in dd-trace-java, by emitting agent telemetry metrics for OTLP exporter activity and by wiring OTLP trace sends into existing Datadog tracer health metrics reporting.
Changes:
- Add a new
OtlpTelemetrymetric collector (internal-api) and a telemetry periodic action (telemetry) to report OTLP metrics/logs telemetry metrics with protocol/encoding tags. - Refactor OTLP senders (
OtlpSender) to returnRemoteApi.Response, centralizing HTTP execution/handling inOtlpSenderSupport, and use responses to drive success/failure counters. - Feed OTLP trace send results into
HealthMetricsviaOtlpPayloadDispatcher(used byOtlpWriter), and update tests for the new sender contract.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| telemetry/src/main/java/datadog/telemetry/TelemetrySystem.java | Registers the new OTLP telemetry periodic action in the telemetry loop. |
| telemetry/src/main/java/datadog/telemetry/metric/OtlpTelemetryPeriodicAction.java | Adds a MetricPeriodicAction to drain OTLP telemetry metrics into telemetry. |
| internal-api/src/main/java/datadog/trace/api/telemetry/OtlpTelemetry.java | Implements OTLP exporter telemetry metrics (metrics export counters + log record count) tagged by protocol/encoding. |
| internal-api/src/test/java/datadog/trace/api/telemetry/OtlpTelemetryTest.java | Adds unit tests validating metric names, tags, draining behavior, and count handling. |
| dd-trace-core/src/main/java/datadog/trace/core/otlp/common/OtlpSender.java | Changes sender contract to return RemoteApi.Response so callers can track success/failure. |
| dd-trace-core/src/main/java/datadog/trace/core/otlp/common/OtlpSenderSupport.java | New shared helper for executing OTLP requests with retries and mapping outcomes to RemoteApi.Response. |
| dd-trace-core/src/main/java/datadog/trace/core/otlp/common/OtlpHttpSender.java | Uses OtlpSenderSupport and returns RemoteApi.Response. |
| dd-trace-core/src/main/java/datadog/trace/core/otlp/common/OtlpGrpcSender.java | Uses OtlpSenderSupport and returns RemoteApi.Response. |
| dd-trace-core/src/main/java/datadog/trace/core/otlp/metrics/OtlpMetricsService.java | Records OTLP metrics export attempts/successes/failures to OtlpTelemetry based on response. |
| dd-trace-core/src/main/java/datadog/trace/core/otlp/logs/OtlpLogsCollector.java | Extends collector API to expose the number of log records collected. |
| dd-trace-core/src/main/java/datadog/trace/core/otlp/logs/OtlpLogsProtoCollector.java | Tracks log record count during protobuf collection. |
| dd-trace-core/src/main/java/datadog/trace/core/otlp/logs/OtlpLogsJsonCollector.java | Tracks log record count during JSON collection. |
| dd-trace-core/src/main/java/datadog/trace/core/otlp/logs/OtlpLogsService.java | Emits otel.log_records telemetry on successful OTLP log export. |
| dd-trace-core/src/main/java/datadog/trace/common/writer/OtlpWriter.java | Passes HealthMetrics into the OTLP payload dispatcher. |
| dd-trace-core/src/main/java/datadog/trace/common/writer/OtlpPayloadDispatcher.java | Reports OTLP trace send success/failure to HealthMetrics using RemoteApi.Response. |
| dd-trace-core/src/test/java/datadog/trace/common/writer/OtlpPayloadDispatcherTest.java | Updates mocks/stubbing for new OtlpSender.send return type. |
| dd-trace-core/src/test/java/datadog/trace/core/otlp/metrics/OtlpStatsMetricWriterTest.java | Updates test sender to satisfy new OtlpSender.send signature. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 62f5b3cdb7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
This comment has been minimized.
This comment has been minimized.
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
ab35f2c to
839240f
Compare
dd-octo-sts
Bot
added
the
tag: ai generated
![datadog-datadog-prod-us1-2[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
The diff preserves the existing OTLP retry and payload paths while adding counters that reset at collection boundaries and only count successfully exported traces or log records. Focused Gradle execution was blocked before compilation because the environment lacks the repository-required Java 25 toolchain; no concrete behavioral regression was identifiable from the diff and surrounding code.
🤖 Datadog Autotest · Commit 839240f · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 839240f710
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
839240f to
12930e9
Compare
* Datadog tracer health metrics
* otel.metrics_export_{attempts,successes,failures}
* otel.log_records
OTLP telemetry metrics are tagged by protocol and encoding (per signal)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
12930e9 to
74e91ad
Compare
mhlidd
left a comment
mhlidd
left a comment
There was a problem hiding this comment.
Small nit about Log metrics, otherwise LGTM
…tatsD client) and use telemetry instead
|
@codex review |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (3)
dd-trace-core/src/main/java/datadog/trace/common/writer/WriterFactory.java:82
- In the OTLP writer branch,
WriterFactory.createWriter(...)no longer passes itshealthMetricsargument into the OTLP writer builder. This means enabling tracer health metrics has no effect for OTLP, unlike other writer types handled in this factory.
If OTLP writers are still expected to emit the standard tracer health metrics, the OTLP builder should accept HealthMetrics and this factory should forward the healthMetrics instance here.
} else if (OTLP_WRITER_TYPE.equals(configuredType)) {
return OtlpWriter.builder()
.endpoint(config.getOtlpTracesEndpoint())
.headers(config.getOtlpTracesHeaders())
.protocol(config.getOtlpTracesProtocol())
.compression(config.getOtlpTracesCompression())
.timeoutMillis(config.getOtlpTracesTimeout())
.spanSamplingRules(singleSpanSampler)
.flushIntervalMilliseconds(flushIntervalMilliseconds)
.build();
}
internal-api/src/test/java/datadog/trace/api/telemetry/OtlpTelemetryTest.java:66
- This test hard-codes expected OTLP protocol/encoding tags (
protocol:http,encoding:protobuf). Since the tags inOtlpTelemetryare derived fromConfig.get().getOtlpMetricsProtocol(), assert against the protocol-derived values to avoid test-order dependence on system properties.
assertTrue(metric.tags.contains("protocol:http"));
assertTrue(metric.tags.contains("encoding:protobuf"));
internal-api/src/test/java/datadog/trace/api/telemetry/OtlpTelemetryTest.java:95
- This test assumes logs exporter tags are always
protocol:httpandencoding:protobuf, butOtlpTelemetryderives them fromConfig.get().getOtlpLogsProtocol(). Computing the expected tag values from the configured protocol will keep the test stable if other tests set OTLP-related system properties.
assertTrue(metric.tags.contains("protocol:http"));
assertTrue(metric.tags.contains("encoding:protobuf"));
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f446803c0f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
dd-trace-core/src/test/java/datadog/trace/common/writer/OtlpPayloadDispatcherTest.java:144
OtlpTelemetry.drain()does not include newly incremented counters unlessprepareMetrics()has been called to stage them into the queue. As written, this helper will always return an empty map, making the telemetry assertions fail.
Stage metrics via prepareMetrics() before draining.
private static Map<String, OtlpTelemetry.OtlpMetric> drainTracesTelemetry() {
Map<String, OtlpTelemetry.OtlpMetric> byName = new HashMap<>();
for (OtlpTelemetry.OtlpMetric metric : OtlpTelemetry.getInstance().drain()) {
byName.put(metric.metricName, metric);
}
1a0ba1d to
66cce16
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1a0ba1d4dd
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
What Does This Do
Telemetry for OTLP traces/metrics/logs:
OTLP telemetry metrics are tagged by protocol and encoding (per signal)
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]