Add optional native OpenTelemetry export for runner, jobs, and steps#4366
Draft
stefanpenner wants to merge 60 commits into
Draft
Add optional native OpenTelemetry export for runner, jobs, and steps#4366stefanpenner wants to merge 60 commits into
stefanpenner wants to merge 60 commits into
Conversation
60ab864 to
4c1b74f
Compare
stefanpenner
added a commit
to stefanpenner/otel-explorer
that referenced
this pull request
Jun 22, 2026
The self-hosted runner's native OpenTelemetry (actions/runner#4366) emits a trace whose root is the JOB span (SpanKind=SERVER), carrying run-level attributes (cicd.pipeline.run.id, vcs.*) and task-level attributes (cicd.pipeline.task.*), with step spans as its children. There is no separate workflow-run span above the job, and the job-root's parent ref points at a run span the runner never exports (so the ref dangles). The span classifier only treated a truly-parentless span as a run and never marked cicd task spans as leaves, so it counted the job-root as a plain job and its step children as jobs. The summary reported Runs: 0. Recognize the runner contract in a shared classifier (classifySpans): - a trace-root span (no parent, or dangling parent ref) with cicd.pipeline.run.id is the run; if it also has cicd.pipeline.task.name it is the collapsed job-as-root, counted as both run and job. - its cicd.pipeline.task.* children are steps. - cicd.pipeline.task.run.result drives the success-rate math. Used by CombinedMetricsFromSpans (markdown summary), RunsFromSpans (timeline reconstruction), and CalculateSummary (terminal report) so all three agree. The GitHub-API path and multi-level pipelines are unchanged: a dangling parent alone no longer promotes a span to a run; only the cicd.pipeline.run.id signal does. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
7d1d20a to
0daba49
Compare
When ACTIONS_RUNNER_OTLP_ENDPOINT is set, the runner emits OTel spans for each step as they complete. Spans use deterministic IDs (MD5-based) compatible with otel-explorer's GitHub Actions trace view, allowing runner step spans to merge into workflow traces with zero configuration. Attributes follow OTel CI/CD semantic conventions (cicd.pipeline.*) and include GitHub-specific fields (github.action, github.step_type, etc.). Spans are batched per job and flushed via OTLP/HTTP JSON before job completion is reported to the server. Export is best-effort — failures are silently swallowed to never impact job execution. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Broadens the step-only exporter into OTelTracer, which emits a full
job subtree natively from the runner:
- Runner info as the OTLP Resource (service.*, host.arch, os.type,
github.runner.{name,id,group,ephemeral}), set in JobRunner.
- Job span (ExecutionContext.Complete, Job branch), parented to the
workflow span (NewSpanID(run_id)).
- Step spans parented to the job span, with CI/CD semconv task attrs
(cicd.pipeline.task.name/run.id/run.result/run.url.full), error.type
and ERROR status on failure, and typed attribute values.
Spans use a name-based deterministic ID contract derived only from
identifiers the runner always has (run id, run attempt, job display
name, step name), captured once at job init so step->job linkage stays
consistent. IDs match otel-explorer's reconstruction so runner spans
merge into the GitHub Actions trace. Cross-language golden values are
pinned in OTelTracerL0.
Still opt-in via ACTIONS_RUNNER_OTLP_ENDPOINT; export is best-effort.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds Constants.Runner.Features.RunnerOtelExport (actions_runner_otel_export) as a kill switch on top of the ACTIONS_RUNNER_OTLP_ENDPOINT operator opt-in: enabled = endpoint configured && (feature flag ?? true) The flag defaults to true so the endpoint opt-in keeps working on self-hosted / GHES where the flag isn't provisioned, while letting GitHub disable export fleet-wide without a runner redeploy. Captured once at job init via Global.Variables.GetBoolean, matching the runner's feature-flag convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Usage, configuration, ID contract, lifecycle, and the env-var + feature-flag gating for the native OpenTelemetry exporter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ACTIONS_RUNNER_OTLP_HEADERS: comma-separated key=value headers sent with the export, for authenticated collectors (Honeycomb, etc.). - Run every exported string (span names + attribute/resource values) through HostContext.SecretMasker, matching how the runner scrubs all other telemetry sent off-box. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Define the OTLP env var names as constants in Constants.Variables.Agent (OtlpEndpoint/OtlpHeaders/OtlpInsecure) instead of inlining the literals, matching the runner's convention for runner env vars. - Add an L0 test asserting export to an unreachable endpoint never throws (best-effort guarantee). - dotnet format. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Converts the static OTelTracer into IOTelTraceExporter : RunnerService, resolved via HostContext.GetService<>() at the call sites — matching the runner's DI/service convention. This unlocks: - Proxy-aware export via HostContext.CreateHttpClientHandler() instead of a raw HttpClientHandler (satisfies RunnerWebProxyL0.IsNotUseRawHttpClientHandler). - Failure logging via Trace.Info (best-effort, never throws) instead of a silent catch. - Secret masking via HostContext.SecretMasker directly (no injected delegate). TestHostContext registers a default (disabled) exporter so the many tests that exercise a real ExecutionContext don't each need to wire it up. Drops the no-op demo harness. Full L0 suite (961) green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rdening - Skip composite/embedded sub-steps (no !IsEmbedded guard before): they flatten under the job and collide by display name with no API counterpart. Only top-level steps are emitted now. - Serialize OTLP with System.Text.Json instead of hand-built string concat: a control character in a step name no longer invalidates the whole batch (still zero new NuGet deps). - Flush: 2s budget (was 5s) so a slow collector can't delay job completion; check the HTTP status and dispose the response/content; log non-2xx. - Register OTLP header values with the SecretMasker (commonly auth tokens). - Accept an endpoint that already includes /v1/traces (no double path). - ToUnixNano treats Unspecified DateTimeKind as UTC (no local-offset skew). - Write _featureEnabled under the lock. Adds L0 tests for the embedded-step skip and control-char JSON safety. Full L0 suite (963) green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- TestHostContext.GetService now instantiates the ServiceLocator default when no mock is registered, mirroring HostContext. Removes the per-test wiring (and the earlier Worker reference in the harness) for the OTel exporter and any future leaf service exercised through real production code. - Step span IDs include the 1-based step number, so two top-level steps sharing a display name get distinct IDs (matches the API's step.number, still merges). - ADR: document the feature-flag default-on as a deliberate decision, and that re-runs need no special handling (current-attempt parenting matches the API). Full L0 suite (963) green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… logs - VCS + actor context on every span (vcs.revision, vcs.ref.head.name, github.actor, cicd.pipeline.run.url.full); job span carries throttling delay. - RecordSpan: generic child span (parented to job) for finer-grained timing; wired to action download (action_download spans). - Trace-context propagation (actions#4): inject W3C TRACEPARENT + OTEL_EXPORTER_OTLP_* + OTEL_RESOURCE_ATTRIBUTES into each step's env (opt-in via ACTIONS_RUNNER_OTLP_PROPAGATE) so in-job tools/actions emit spans parented to the step. Namespaced runner vars (ACTIONS_RUNNER_OTLP_*) avoid clobbering these. - OTel Logs signal: step errors/warnings emitted as log records correlated to the step span (trace_id/span_id), flushed to {endpoint}/v1/logs, masked. New L0 tests for each. Full suite 969 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…schema_url Audited against the OTel attribute registries: - cicd.pipeline.task.run.result: map skipped -> "skip" (spec enum is success|failure|cancellation|skip|timeout|error, not "skipped"). - VCS: emit vcs.ref.head.revision (spec name) instead of vcs.revision; add vcs.provider.name=github, vcs.owner.name, vcs.repository.name. - Resource: add cicd.worker.name/id and service.instance.id alongside github.runner.*. - Declare conformance: schemaUrl on resource/scope spans + scope version. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Exception span events (semconv) on failed steps/jobs: exception.type + exception.message from the step's first error annotation. - PR context on pull_request: vcs.change.id (PR number parsed from github.ref) and vcs.ref.base.name (base branch). - cicd.pipeline.task.type inferred (build|test|deploy) from step/action name. L0 tests for each. Full suite 974 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On job completion, export OTLP metrics to {endpoint}/v1/metrics:
- cicd.pipeline.run.duration (histogram, seconds) tagged cicd.pipeline.name +
cicd.pipeline.result.
- cicd.pipeline.run.errors (monotonic counter) emitted only on failure, tagged
error.type. Best-effort, same 2s budget as the other signals.
L0 tests for the histogram + error-counter and the success (no-errors) case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the runner's OTel resource Kubernetes/ARC-aware the spec-native way: - Parse the standard OTEL_RESOURCE_ATTRIBUTES env var (comma-separated, percent-encoded key=value) and merge it into the resource. This is how a deployment attaches k8s.pod.name / k8s.namespace.name / k8s.node.name (e.g. ARC via the Kubernetes Downward API) with no runner-specific env names. Explicitly-set keys take precedence. - Add cicd.system.component=agent (semconv): the runner is the CI/CD agent. Preferring the standard env var over bespoke K8S_* names keeps the runner emitting portable, any-backend telemetry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MD5.HashData throws on a FIPS-enabled host. Switch the trace/span ID helpers to SHA-256 truncated to the ID length (16 bytes trace, 8 bytes span); the scheme is otherwise unchanged and stays byte-identical to otel-explorer's contract. Golden L0 values + the cross-language check updated to SHA-256. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Enables an upstream scheduler (e.g. actions-runner-controller) to causally connect its "schedule" span to the job without re-rooting the runner's deterministic trace: - Read ACTIONS_RUNNER_PARENT_TRACEPARENT / _TRACESTATE from the environment. - Parse the traceparent per W3C rules (validate hex/length, reject all-zero and the forbidden "ff" version, tolerate unknown future versions). - Attach it as a span LINK on the job span (cross-trace causality), preserving tracestate and carrying the inbound sampled flag rather than overriding it. - The runner's own trace/span IDs are unchanged, so API-reconstruction merge still works. The deterministic IDs are never advertised as W3C-random. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
otel-explorer now identifies runner spans by the standard instrumentation scope (github.actions.runner) and resource service.name, so the bespoke source=runner attribute is no longer needed on every span. Also drop github.runner.name and github.runner.id, which duplicated cicd.worker.name / cicd.worker.id; group and ephemeral stay under github.runner.* (no semconv equivalent). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Builds linux-arm64 runner from source and overlays onto the official actions-runner image; bakes CMD run.sh. For deploying native-OTel runner into k8s/ARC. See otel-explorer examples/arc/OTEL-RUNNER-STATUS.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Actions service denies ephemeral/JIT runners below the current minimum version (AccessDenied -> exit 7), which ARC reads as Outdated -> teardown loop. PR#4366 is on 2.333.0; report 2.335.1 via build ARG so the source stays pristine. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…esult
Two fixes found comparing the runner's exported trace across Tempo/Jaeger:
1. The job span was parented to NewSpanID(run_id) (a 'workflow span') that the
runner never emits -> a dangling parentSpanId. The runner owns the job, not
the workflow run, so the job is now the root; the authoritative run span comes
from the GitHub API path (merged downstream), and an upstream scheduler is
still attached as a span LINK.
2. action-resolution ('Resolve <action>') spans set task.name but no
cicd.pipeline.task.run.result, unlike every other task span. The caller now
passes success/failure.
L0: JobSpan_IsRoot_NoDanglingParent + RecordSpan result assertion; 34/34 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…up job) Action downloads run inside a step's execution context (action resolution happens during 'Set up job'), but RecordSpan hard-parented them to the job span, so they rendered as siblings of the steps instead of children. Parent them to the owning step span when the caller supplies it (RecordSpan parentStepName/parentStepNumber); fall back to the job span otherwise. Exposes IExecutionContext.StepDisplayName/StepOrder. L0: RecordSpan_ParentsToStep_WhenStepGiven; 35/35 pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The runner emitted only run-level metrics (cicd.pipeline.run.duration/errors); job- and step-level timing was trace-only. Add a cicd.pipeline.task.duration histogram with one observation per job and per step, tagged cicd.pipeline.task.name, cicd.pipeline.task.run.result, and type (job|step), so you get RED-style metrics at job/step granularity (e.g. p95 step duration, failures by step). L0: TaskMetrics_DurationPerStepAndJob; 36/36 pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n.attempt - P0: wrap FlushAsync body in try/catch so a serialization/OOM failure can never escape into CompleteJobAsync and fail the job (the export must never affect it). - Security: stop propagating OTEL_EXPORTER_OTLP_HEADERS (collector credential) into step child-process env, where any user step could read and exfiltrate it. Only TRACEPARENT + endpoint are propagated. - Visibility: add cicd.pipeline.run.attempt to all metric data points so re-runs don't collapse onto one series. L0: StepPropagationEnv_DoesNotLeakAuthHeaders + Metrics_CarryRunAttempt; 38/38 pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Move run/task duration + errors out of the reserved cicd.* semconv namespace into the vendor github.pipeline.* namespace (cicd.pipeline.task.duration was an invented name in a reserved namespace). - Replace empty-bounds single-bucket histograms with explicit duration buckets so the collector can aggregate real percentiles (mirrors the spanmetrics connector). - run.attempt attribute -> github.run_attempt; task 'type' -> github.record_type. L0: DurationHistogram_HasExplicitBucketsNotEmptyBounds; 43/43 pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ess, log schemaUrl - Network ops (action download, container pull/create) now emit SpanKind=CLIENT (3) via a RecordSpan spanKind param; process/step spans stay INTERNAL. - Namespace the ad-hoc 'type' span+metric attribute as github.record_type. - Handle OTLP partial success: a 200 with rejectedSpans/DataPoints/LogRecords is now detected and logged (was silently discarded). - Logs gain scope version, schemaUrl, and observedTimeUnixNano (parity with traces/metrics). - Refresh the class doc to the job-is-root model + point at docs/otel-id-contract.md. L0: RecordSpan_HonorsClientSpanKind, DescribePartialSuccess_FlagsRejectedItems, Logs_HaveSchemaUrlAndObservedTime; 46/46 pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e tests Split the gzip + retry + partial-success + HttpClient out of the 1400-line exporter into OTelHttpTransport (constructor-injectable HttpMessageHandler). Closes the 'zero HTTP-transport tests' blocker: OTelHttpTransportL0 uses a fake handler to verify URL routing, application/json + gzip Content-Encoding, gunzip->original OTLP, auth-header attach, transient retry (503->200), no-retry on 4xx, partial-success delivery, and best-effort behavior on an expired deadline. 51 OTel L0 tests pass (46 exporter + 5 transport). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…y, no behavior change) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…8/500 The OTLP/HTTP failures table limits retryable responses to 429, 502, 503, 504; all other 4xx/5xx MUST NOT be retried. IsTransientStatus previously also retried 408 and 500 — a hard-MUST spec violation that doubles load on a collector that is deliberately rejecting. Align the set with the spec, cite the table in the code, cover 408/500 with L0 tests, and update the ADR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Retrying a fixed 200 ms after an explicit 429/503 ignores the server's signal (OTLP spec SHOULD honor Retry-After) and lets fleets retry in lockstep. Now: an in-deadline Retry-After is waited out before the single retry; one longer than the flush deadline could honor skips the retry entirely; without the header the delay is a jittered 100-400 ms. The deliberate one-retry bound (vs the spec's exponential-backoff SHOULD) is documented in the ADR transport section. Delay is injectable so the L0 timing tests don't sleep for real. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
OTLP exporter spec SHOULD: send a User-Agent identifying the exporter and its version so collector operators can attribute and police traffic from runner fleets. Sends GitHubActionsRunner-OTLP-Exporter/<runner version>. Also documents why gzip stays always-on with no opt-out knob: the OTLP spec requires every server component to support gzip request bodies, so a compression escape hatch would be config for a non-conformant server. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…UTF-8 bytes One POST per signal meant a buffer-cap job (~18-40 MB uncompressed) could exceed a collector's decompressed-body limit (OTel Collector default: 20 MiB) and take a 413 — non-retryable per OTLP — silently dropping the whole signal, exactly on the big jobs where telemetry matters most. FlushAsync now sends spans/logs in batches of MaxItemsPerPost (1,000, ~4 MB uncompressed) under the same single 4 s flush deadline, verified by an L0 test that captures the wire requests. The Build* serializers also stop round-tripping the payload through a UTF-16 string (bytes -> string -> bytes before gzip): they return the Utf8JsonWriter bytes directly, so flush no longer triple-materializes large payloads. docs/otel-benchmarks.md's 'motivates gzip + chunking' note is now implemented; ADR transport section updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
VarUtil returns "Linux"/"macOS"/"Windows" and "X86"/"X64"/"ARM"/"ARM64"; the semconv enums are lowercase (Apple is darwin — there is no macOS value) and x86/amd64/arm32/arm64. Raw OS name is preserved in free-form os.name. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The registered semconv key is process.exit.code (type int), not process.exit_code. RecordSpan attributes are now IDictionary<string, object> so bool/int/long values keep their OTLP type (intValue) instead of being coerced to strings; the exit code also no longer goes through current-culture ToString() (a negative code under a non-ASCII NegativeSign culture produced an unparseable value). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lly conforms to 1.29.0 lacks vcs.provider.name, vcs.owner.name, vcs.repository.name, cicd.worker.id/name, cicd.system.component, cicd.pipeline.run.url.full, cicd.pipeline.task.run.result and cicd.pipeline.result, all of which this exporter emits; a wrong schema_url poisons schema-aware version transformations. Audited every emitted key against the v1.34.0 registry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The job span is recorded last, so at the 10k cap the drop-newest policy evicted exactly the trace root, orphaning every already-buffered step span (they all parent to its deterministic ID). It now bypasses the cap (one per job, worst case cap+1). ADR documents the reservation; the crash-before- flush loss mode was already documented there as an accepted trade-off. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ding it Host step processes inherit the worker's environment via ProcessInvoker, so the raw collector credential reached every untrusted step (printenv) — the exact leak the StepPropagationEnv comment promises can't happen. Read it once in Initialize, then remove it (CommandSettings precedent), including when export is disabled (credential without endpoint is still a credential). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A basic-auth-in-URL collector (https://user:token@...) embeds a credential in the endpoint. It now never reaches step processes (StepPropagationEnv hands out a stripped base URL) and is registered with the SecretMasker so diag/transport log lines mask it — matching the existing posture of withholding OTLP headers from steps. ADR documents that propagation is host-perspective (localhost is unreachable from container jobs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… INSECURE The insecure flag disables ALL server-certificate validation, not the 'skip verification for self-signed collectors' the ADR implied — with ACTIONS_RUNNER_OTLP_HEADERS on every POST, a MITM could harvest the collector credential. Now: a PEM CA bundle (mirroring OTEL_EXPORTER_OTLP_CERTIFICATE) is the supported primitive for self-signed/private-CA collectors (hostname still checked, bad bundle fails closed), and enabling INSECURE logs an explicit Trace.Warning naming the MITM exposure — never silent. ADR documents both honestly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er throw into the job) Record*/FlushAsync were already best-effort, but these entry points were unguarded — and the first GetService<IOTelTraceExporter>() runs before JobRunner's try block, so an unexpected throw (e.g. handler factory, bad CA bundle) surfaced straight into job execution. Initialize failure now disables export (fail-closed); the others log-and-continue like every other hook. L0 failure-injection test covers the Initialize path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dropped Every failure path was Trace.Info in per-job _diag files — effectively silent at fleet scale (an expired collector credential just made traces stop). FlushAsync now counts delivered vs failed POSTs and emits a single greppable Trace.Warning summary (failed POSTs + cap-drop counts) only when something was lost; a total flush failure warns likewise. Still swallowed — the job is never affected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r values Reviewed adding a name filter/length floor for ACTIONS_RUNNER_OTLP_HEADERS values registered with the SecretMasker and rejected it: a heuristic would leave short tokens or non-standard auth headers unmasked (fail-unsafe), while over-masking a short non-secret value is only cosmetic and matches the add-mask command's fail-safe convention. Comment records the decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rray/bytes) The hand-rolled encoder coerced every non-bool/int attribute to stringValue. OTLP AnyValue defines doubleValue, arrayValue and bytesValue; the first caller to record a double would silently get a string on the wire, breaking numeric queries downstream. Map each CLR type to its wire type; only genuinely unsupported types stringify (masked), and that arm is documented. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…only exception event exception.type was set to the literal CI conclusion "failure", which is not an exception class; backends that group errors by exception.type would bucket every failed step in one meaningless group. Omit the faked type (semconv allows message-only exception events; no message, no event) and surface the error message as span status.message per the Recording Errors guidance. error.type stays as the classifier. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e run cicd-spans: pipeline-run spans SHOULD be SERVER (with cicd.pipeline.result); task-run spans SHOULD be INTERNAL. The job span carries the cicd.pipeline.task.* attribute set but was emitted as SERVER, straddling both models. Emit it INTERNAL like the steps; the SERVER run span belongs to the API-side consumer. Documented in docs/otel-id-contract.md so span kind is a reliable discriminator. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ric honestly github.pipeline.run.errors was the registered semconv metric cicd.pipeline.run.errors wearing a vendor name (same instrument, unit and required attrs cicd.pipeline.name + error.type; a failed job is 'an error encountered in a pipeline run') — adopt the semconv name so standard CI/CD dashboards pick it up. github.pipeline.run.duration is NOT renamed to the semconv twin: the worker observes one job's wall time, not the pipeline run's, so cicd.pipeline.run.duration (whole-run duration grouped by cicd.pipeline.run.state) would be semantically wrong (a 10-job run would emit 10 bogus run-duration points, no queue time). Semconv has no task-level duration metric, so it stays vendor-namespaced — renamed to github.pipeline.job.duration to say what it measures. Rationale in the ADR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ction steps - Move TRACEPARENT/OTEL_* injection into a shared Handler helper (AddOTelPropagationToEnvironment) and call it from ScriptHandler, NodeScriptActionHandler and ContainerActionHandler, so uses: (JS and Docker) action steps nest their telemetry too, matching the ADR's 'each step's env' claim. Composite run: steps already flow through ScriptHandler. - Precedence: workflow wins. Inject each key only when the step env does not already define it, so a step that sets its own TRACEPARENT/OTEL_* keeps them (the ADR's no-clobber promise, now enforced). - ADR: state the precedence rule and the handler coverage explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…Exporter The global ServiceLocator fallback in TestHostContext.GetService silently instantiated the real default for ANY unregistered service, removing the deliberate missing-mock seam for every L0 test. Restore the throw and keep the fallback only for an explicit allowlist (IOTelTraceExporter — resolved lazily on many production paths, inert unless its env vars are set). Adds TestHostContextL0 guarding both behaviors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Flush-before-complete is deliberate: the completion report triggers ephemeral-runner teardown (ARC deletes the pod), so flushing after it races pod deletion and drops the job's telemetry. Say so at both call sites and in the ADR trade-off, including the bounded worst case (4 s deadline, hung collector) vs the healthy-path cost (ms). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nchmarks - New L0 Flush_HangingCollector_BoundedByOverallDeadline: a collector that accepts TCP then stalls (the case nothing fast-fails on) — asserts a real FlushAsync returns within the 4 s overall deadline (< 6 s with slack). - Benchmarks doc: add a measured failure-mode table (refused < 1 s, hung ~4 s), a collector-down threshold row, and replace the n=3 'no measurable overhead' claim with the bounded statement it supports. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…icitly Every other Constants.Runner.Features gate defaults off; this one is deliberately ?? true (endpoint opt-in is the real gate, ?? false would kill the feature on GHES/self-hosted where the flag is never provisioned). Name the deviation, and offer the conventional-polarity alternative (negative flag actions_runner_otel_export_disabled) for maintainer sign-off rather than leaving it implicit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ADR now reflects every shipped behavior from the hardening batches: - child spans (action download, container pull/create, run: process with typed process.exit.code) and their INTERNAL/CLIENT kinds - semconv enum mapping for os.type/host.arch - typed OTLP AnyValue serialization + exporter User-Agent - guarded entry points and the one end-of-job Warning loss summary - no-length-floor masking policy for OTLP header values - OTelHttpTransport named as the wire component releaseNote.md gains the feature bullet (opt-in endpoint, deterministic IDs, bounded best-effort flush, propagation precedence, kill switch). otel-id-contract.md audited against the batches: no ID-affecting change, already accurate (span kinds, vectors, TRACEPARENT derivation). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…C listener) The runner emits cicd.worker.name and cicd.worker.id as resource attributes (one worker per process). The ARC listener emits them as span attributes because one listener process serves many runners. Adds a new § with a table and a TraceQL query pattern covering both scopes, so consumers can query `span.cicd.worker.name || resource.cicd.worker.name` without missing either side of a merged trace. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
0daba49 to
248f5d1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Opt-in, best-effort native OpenTelemetry export from the runner — no API polling, no new NuGet dependencies (hand-built OTLP/JSON over the runner's proxy-aware HTTP handler). Enabled by setting
ACTIONS_RUNNER_OTLP_ENDPOINT. Emits, from the existing job/step lifecycle:service.*,host.arch,os.type,github.runner.{name,id,group,ephemeral}).Span IDs use a deterministic, name-based contract (run id, run attempt, job/step display names — not the feature-flagged numeric job id) so they merge with a GitHub-API trace reconstruction. Design + usage:
docs/adrs/4366-native-otel-export.md. Consumer-side companion: stefanpenner/otel-explorer#54.Changes
IOTelTraceExporter : RunnerService(renamed from a staticOTelStepTracer), resolved viaHostContext.GetService<>()— matches the runner's DI/service convention.Lifecycle hooks:
JobRunner.RunAsync(resource),ExecutionContext.InitializeJob(job identifiers + flag),ExecutionContext.Complete(job/step spans),JobRunner.CompleteJobAsync(flush).Gating:
enabled = ACTIONS_RUNNER_OTLP_ENDPOINT set && (Constants.Runner.Features.RunnerOtelExport ?? true)— operator env opt-in + server-side kill switch (defaults on so self-hosted/GHES isn't blocked when the flag isn't provisioned).Proxy-aware: export goes through
HostContext.CreateHttpClientHandler()(honors runner proxy config).Secret masking: every exported string runs through
HostContext.SecretMasker.Best-effort: failures are logged via
Traceand swallowed; never affect the job.Env var names are constants in
Constants.Variables.Agent:ACTIONS_RUNNER_OTLP_ENDPOINT{endpoint}/v1/tracesACTIONS_RUNNER_OTLP_HEADERSkey=valueauth headersACTIONS_RUNNER_OTLP_INSECUREtrueto skip TLS verificationDeterministic ID contract (mirrored in otel-explorer)
Test plan
RunnerWebProxyL0.IsNotUseRawHttpClientHandlersatisfied;dotnet formatclean; no new dependencies--lintreports no semconv issuesNotes
action-dedup-case-fix).🤖 Generated with Claude Code