Skip to content

Add optional native OpenTelemetry export for runner, jobs, and steps#4366

Draft
stefanpenner wants to merge 60 commits into
actions:mainfrom
stefanpenner:otel-step-traces
Draft

Add optional native OpenTelemetry export for runner, jobs, and steps#4366
stefanpenner wants to merge 60 commits into
actions:mainfrom
stefanpenner:otel-step-traces

Conversation

@stefanpenner

@stefanpenner stefanpenner commented Apr 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Opt-in, best-effort native OpenTelemetry export from the runner — no API polling, no new NuGet dependencies (hand-built OTLP/JSON over the runner's proxy-aware HTTP handler). Enabled by setting ACTIONS_RUNNER_OTLP_ENDPOINT. Emits, from the existing job/step lifecycle:

  1. Runner info as the OTLP Resource (service.*, host.arch, os.type, github.runner.{name,id,group,ephemeral}).
  2. A job span parented to the workflow span.
  3. A step span per step parented to the job span, with sub-second timing, action refs, and CI/CD semconv attributes.

Span IDs use a deterministic, name-based contract (run id, run attempt, job/step display names — not the feature-flagged numeric job id) so they merge with a GitHub-API trace reconstruction. Design + usage: docs/adrs/4366-native-otel-export.md. Consumer-side companion: stefanpenner/otel-explorer#54.

Changes

  • IOTelTraceExporter : RunnerService (renamed from a static OTelStepTracer), resolved via HostContext.GetService<>() — matches the runner's DI/service convention.

  • Lifecycle hooks: JobRunner.RunAsync (resource), ExecutionContext.InitializeJob (job identifiers + flag), ExecutionContext.Complete (job/step spans), JobRunner.CompleteJobAsync (flush).

  • Gating: enabled = ACTIONS_RUNNER_OTLP_ENDPOINT set && (Constants.Runner.Features.RunnerOtelExport ?? true) — operator env opt-in + server-side kill switch (defaults on so self-hosted/GHES isn't blocked when the flag isn't provisioned).

  • Proxy-aware: export goes through HostContext.CreateHttpClientHandler() (honors runner proxy config).

  • Secret masking: every exported string runs through HostContext.SecretMasker.

  • Best-effort: failures are logged via Trace and swallowed; never affect the job.

  • Env var names are constants in Constants.Variables.Agent:

    Env var Required Description
    ACTIONS_RUNNER_OTLP_ENDPOINT Yes OTLP/HTTP base URL; traces POSTed to {endpoint}/v1/traces
    ACTIONS_RUNNER_OTLP_HEADERS No Comma-separated key=value auth headers
    ACTIONS_RUNNER_OTLP_INSECURE No true to skip TLS verification

Deterministic ID contract (mirrored in otel-explorer)

trace = md5("{run_id}-{run_attempt}")
job   = md5("job-{run_id}-{run_attempt}-{job_name}")[:8]              parent -> bigendian(run_id)  (workflow)
step  = md5("step-{run_id}-{run_attempt}-{job_name}-{step_name}")[:8] parent -> job

Test plan

  • L0 unit tests for the exporter: golden + cross-language IDs; enable/disable; feature-flag on/off; step semconv + failure status; job span linkage; runner-info resource; secret masking; unreachable endpoint never throws
  • Full L0 suite green (961 tests); RunnerWebProxyL0.IsNotUseRawHttpClientHandler satisfied; dotnet format clean; no new dependencies
  • End-to-end on a real self-hosted runner: native spans captured and merged with an API-reconstructed trace (server-side queue time + runner-native per-step precision), each span once; --lint reports no semconv issues

Notes

  • OTel-only PR; the earlier action-dedup case-sensitivity commits were split to a separate branch (action-dedup-case-fix).

🤖 Generated with Claude Code

@stefanpenner stefanpenner changed the title Add optional OTel trace export for step-level timing Add optional native OpenTelemetry export for runner, jobs, and steps Jun 17, 2026
@stefanpenner stefanpenner force-pushed the otel-step-traces branch 2 times, most recently from 60ab864 to 4c1b74f Compare June 22, 2026 15:15
stefanpenner added a commit to stefanpenner/otel-explorer that referenced this pull request Jun 22, 2026
The self-hosted runner's native OpenTelemetry (actions/runner#4366) emits a
trace whose root is the JOB span (SpanKind=SERVER), carrying run-level
attributes (cicd.pipeline.run.id, vcs.*) and task-level attributes
(cicd.pipeline.task.*), with step spans as its children. There is no separate
workflow-run span above the job, and the job-root's parent ref points at a
run span the runner never exports (so the ref dangles).

The span classifier only treated a truly-parentless span as a run and never
marked cicd task spans as leaves, so it counted the job-root as a plain job
and its step children as jobs. The summary reported Runs: 0.

Recognize the runner contract in a shared classifier (classifySpans):
  - a trace-root span (no parent, or dangling parent ref) with
    cicd.pipeline.run.id is the run; if it also has cicd.pipeline.task.name it
    is the collapsed job-as-root, counted as both run and job.
  - its cicd.pipeline.task.* children are steps.
  - cicd.pipeline.task.run.result drives the success-rate math.

Used by CombinedMetricsFromSpans (markdown summary), RunsFromSpans (timeline
reconstruction), and CalculateSummary (terminal report) so all three agree.
The GitHub-API path and multi-level pipelines are unchanged: a dangling parent
alone no longer promotes a span to a run; only the cicd.pipeline.run.id signal
does.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
stefanpenner and others added 23 commits July 2, 2026 07:52
When ACTIONS_RUNNER_OTLP_ENDPOINT is set, the runner emits OTel spans
for each step as they complete. Spans use deterministic IDs (MD5-based)
compatible with otel-explorer's GitHub Actions trace view, allowing
runner step spans to merge into workflow traces with zero configuration.

Attributes follow OTel CI/CD semantic conventions (cicd.pipeline.*) and
include GitHub-specific fields (github.action, github.step_type, etc.).

Spans are batched per job and flushed via OTLP/HTTP JSON before job
completion is reported to the server. Export is best-effort — failures
are silently swallowed to never impact job execution.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Broadens the step-only exporter into OTelTracer, which emits a full
job subtree natively from the runner:

- Runner info as the OTLP Resource (service.*, host.arch, os.type,
  github.runner.{name,id,group,ephemeral}), set in JobRunner.
- Job span (ExecutionContext.Complete, Job branch), parented to the
  workflow span (NewSpanID(run_id)).
- Step spans parented to the job span, with CI/CD semconv task attrs
  (cicd.pipeline.task.name/run.id/run.result/run.url.full), error.type
  and ERROR status on failure, and typed attribute values.

Spans use a name-based deterministic ID contract derived only from
identifiers the runner always has (run id, run attempt, job display
name, step name), captured once at job init so step->job linkage stays
consistent. IDs match otel-explorer's reconstruction so runner spans
merge into the GitHub Actions trace. Cross-language golden values are
pinned in OTelTracerL0.

Still opt-in via ACTIONS_RUNNER_OTLP_ENDPOINT; export is best-effort.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds Constants.Runner.Features.RunnerOtelExport (actions_runner_otel_export)
as a kill switch on top of the ACTIONS_RUNNER_OTLP_ENDPOINT operator opt-in:

  enabled = endpoint configured && (feature flag ?? true)

The flag defaults to true so the endpoint opt-in keeps working on
self-hosted / GHES where the flag isn't provisioned, while letting GitHub
disable export fleet-wide without a runner redeploy. Captured once at job
init via Global.Variables.GetBoolean, matching the runner's feature-flag
convention.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Usage, configuration, ID contract, lifecycle, and the env-var + feature-flag
gating for the native OpenTelemetry exporter.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ACTIONS_RUNNER_OTLP_HEADERS: comma-separated key=value headers sent with
  the export, for authenticated collectors (Honeycomb, etc.).
- Run every exported string (span names + attribute/resource values) through
  HostContext.SecretMasker, matching how the runner scrubs all other telemetry
  sent off-box.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Define the OTLP env var names as constants in Constants.Variables.Agent
  (OtlpEndpoint/OtlpHeaders/OtlpInsecure) instead of inlining the literals,
  matching the runner's convention for runner env vars.
- Add an L0 test asserting export to an unreachable endpoint never throws
  (best-effort guarantee).
- dotnet format.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Converts the static OTelTracer into IOTelTraceExporter : RunnerService,
resolved via HostContext.GetService<>() at the call sites — matching the
runner's DI/service convention. This unlocks:

- Proxy-aware export via HostContext.CreateHttpClientHandler() instead of a
  raw HttpClientHandler (satisfies RunnerWebProxyL0.IsNotUseRawHttpClientHandler).
- Failure logging via Trace.Info (best-effort, never throws) instead of a
  silent catch.
- Secret masking via HostContext.SecretMasker directly (no injected delegate).

TestHostContext registers a default (disabled) exporter so the many tests that
exercise a real ExecutionContext don't each need to wire it up. Drops the
no-op demo harness. Full L0 suite (961) green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rdening

- Skip composite/embedded sub-steps (no !IsEmbedded guard before): they
  flatten under the job and collide by display name with no API counterpart.
  Only top-level steps are emitted now.
- Serialize OTLP with System.Text.Json instead of hand-built string concat:
  a control character in a step name no longer invalidates the whole batch
  (still zero new NuGet deps).
- Flush: 2s budget (was 5s) so a slow collector can't delay job completion;
  check the HTTP status and dispose the response/content; log non-2xx.
- Register OTLP header values with the SecretMasker (commonly auth tokens).
- Accept an endpoint that already includes /v1/traces (no double path).
- ToUnixNano treats Unspecified DateTimeKind as UTC (no local-offset skew).
- Write _featureEnabled under the lock.

Adds L0 tests for the embedded-step skip and control-char JSON safety.
Full L0 suite (963) green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- TestHostContext.GetService now instantiates the ServiceLocator default when
  no mock is registered, mirroring HostContext. Removes the per-test wiring
  (and the earlier Worker reference in the harness) for the OTel exporter and
  any future leaf service exercised through real production code.
- Step span IDs include the 1-based step number, so two top-level steps sharing
  a display name get distinct IDs (matches the API's step.number, still merges).
- ADR: document the feature-flag default-on as a deliberate decision, and that
  re-runs need no special handling (current-attempt parenting matches the API).

Full L0 suite (963) green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… logs

- VCS + actor context on every span (vcs.revision, vcs.ref.head.name,
  github.actor, cicd.pipeline.run.url.full); job span carries throttling delay.
- RecordSpan: generic child span (parented to job) for finer-grained timing;
  wired to action download (action_download spans).
- Trace-context propagation (actions#4): inject W3C TRACEPARENT + OTEL_EXPORTER_OTLP_*
  + OTEL_RESOURCE_ATTRIBUTES into each step's env (opt-in via
  ACTIONS_RUNNER_OTLP_PROPAGATE) so in-job tools/actions emit spans parented to
  the step. Namespaced runner vars (ACTIONS_RUNNER_OTLP_*) avoid clobbering these.
- OTel Logs signal: step errors/warnings emitted as log records correlated to
  the step span (trace_id/span_id), flushed to {endpoint}/v1/logs, masked.

New L0 tests for each. Full suite 969 green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…schema_url

Audited against the OTel attribute registries:
- cicd.pipeline.task.run.result: map skipped -> "skip" (spec enum is
  success|failure|cancellation|skip|timeout|error, not "skipped").
- VCS: emit vcs.ref.head.revision (spec name) instead of vcs.revision; add
  vcs.provider.name=github, vcs.owner.name, vcs.repository.name.
- Resource: add cicd.worker.name/id and service.instance.id alongside
  github.runner.*.
- Declare conformance: schemaUrl on resource/scope spans + scope version.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Exception span events (semconv) on failed steps/jobs: exception.type +
  exception.message from the step's first error annotation.
- PR context on pull_request: vcs.change.id (PR number parsed from github.ref)
  and vcs.ref.base.name (base branch).
- cicd.pipeline.task.type inferred (build|test|deploy) from step/action name.

L0 tests for each. Full suite 974 green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On job completion, export OTLP metrics to {endpoint}/v1/metrics:
- cicd.pipeline.run.duration (histogram, seconds) tagged cicd.pipeline.name +
  cicd.pipeline.result.
- cicd.pipeline.run.errors (monotonic counter) emitted only on failure, tagged
  error.type. Best-effort, same 2s budget as the other signals.

L0 tests for the histogram + error-counter and the success (no-errors) case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the runner's OTel resource Kubernetes/ARC-aware the spec-native way:

- Parse the standard OTEL_RESOURCE_ATTRIBUTES env var (comma-separated,
  percent-encoded key=value) and merge it into the resource. This is how a
  deployment attaches k8s.pod.name / k8s.namespace.name / k8s.node.name (e.g.
  ARC via the Kubernetes Downward API) with no runner-specific env names.
  Explicitly-set keys take precedence.
- Add cicd.system.component=agent (semconv): the runner is the CI/CD agent.

Preferring the standard env var over bespoke K8S_* names keeps the runner
emitting portable, any-backend telemetry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MD5.HashData throws on a FIPS-enabled host. Switch the trace/span ID helpers to
SHA-256 truncated to the ID length (16 bytes trace, 8 bytes span); the scheme is
otherwise unchanged and stays byte-identical to otel-explorer's contract. Golden
L0 values + the cross-language check updated to SHA-256.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Enables an upstream scheduler (e.g. actions-runner-controller) to causally
connect its "schedule" span to the job without re-rooting the runner's
deterministic trace:

- Read ACTIONS_RUNNER_PARENT_TRACEPARENT / _TRACESTATE from the environment.
- Parse the traceparent per W3C rules (validate hex/length, reject all-zero and
  the forbidden "ff" version, tolerate unknown future versions).
- Attach it as a span LINK on the job span (cross-trace causality), preserving
  tracestate and carrying the inbound sampled flag rather than overriding it.
- The runner's own trace/span IDs are unchanged, so API-reconstruction merge
  still works. The deterministic IDs are never advertised as W3C-random.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
otel-explorer now identifies runner spans by the standard instrumentation scope
(github.actions.runner) and resource service.name, so the bespoke source=runner
attribute is no longer needed on every span. Also drop github.runner.name and
github.runner.id, which duplicated cicd.worker.name / cicd.worker.id; group and
ephemeral stay under github.runner.* (no semconv equivalent).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Builds linux-arm64 runner from source and overlays onto the official
actions-runner image; bakes CMD run.sh. For deploying native-OTel runner
into k8s/ARC. See otel-explorer examples/arc/OTEL-RUNNER-STATUS.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Actions service denies ephemeral/JIT runners below the current minimum
version (AccessDenied -> exit 7), which ARC reads as Outdated -> teardown loop.
PR#4366 is on 2.333.0; report 2.335.1 via build ARG so the source stays pristine.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…esult

Two fixes found comparing the runner's exported trace across Tempo/Jaeger:

1. The job span was parented to NewSpanID(run_id) (a 'workflow span') that the
   runner never emits -> a dangling parentSpanId. The runner owns the job, not
   the workflow run, so the job is now the root; the authoritative run span comes
   from the GitHub API path (merged downstream), and an upstream scheduler is
   still attached as a span LINK.

2. action-resolution ('Resolve <action>') spans set task.name but no
   cicd.pipeline.task.run.result, unlike every other task span. The caller now
   passes success/failure.

L0: JobSpan_IsRoot_NoDanglingParent + RecordSpan result assertion; 34/34 pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…up job)

Action downloads run inside a step's execution context (action resolution happens
during 'Set up job'), but RecordSpan hard-parented them to the job span, so they
rendered as siblings of the steps instead of children. Parent them to the owning
step span when the caller supplies it (RecordSpan parentStepName/parentStepNumber);
fall back to the job span otherwise. Exposes IExecutionContext.StepDisplayName/StepOrder.

L0: RecordSpan_ParentsToStep_WhenStepGiven; 35/35 pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The runner emitted only run-level metrics (cicd.pipeline.run.duration/errors);
job- and step-level timing was trace-only. Add a cicd.pipeline.task.duration
histogram with one observation per job and per step, tagged
cicd.pipeline.task.name, cicd.pipeline.task.run.result, and type (job|step),
so you get RED-style metrics at job/step granularity (e.g. p95 step duration,
failures by step).

L0: TaskMetrics_DurationPerStepAndJob; 36/36 pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n.attempt

- P0: wrap FlushAsync body in try/catch so a serialization/OOM failure can never
  escape into CompleteJobAsync and fail the job (the export must never affect it).
- Security: stop propagating OTEL_EXPORTER_OTLP_HEADERS (collector credential) into
  step child-process env, where any user step could read and exfiltrate it. Only
  TRACEPARENT + endpoint are propagated.
- Visibility: add cicd.pipeline.run.attempt to all metric data points so re-runs
  don't collapse onto one series.

L0: StepPropagationEnv_DoesNotLeakAuthHeaders + Metrics_CarryRunAttempt; 38/38 pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
stefanpenner and others added 29 commits July 2, 2026 07:53
- Move run/task duration + errors out of the reserved cicd.* semconv namespace into
  the vendor github.pipeline.* namespace (cicd.pipeline.task.duration was an invented
  name in a reserved namespace).
- Replace empty-bounds single-bucket histograms with explicit duration buckets so the
  collector can aggregate real percentiles (mirrors the spanmetrics connector).
- run.attempt attribute -> github.run_attempt; task 'type' -> github.record_type.

L0: DurationHistogram_HasExplicitBucketsNotEmptyBounds; 43/43 pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ess, log schemaUrl

- Network ops (action download, container pull/create) now emit SpanKind=CLIENT (3)
  via a RecordSpan spanKind param; process/step spans stay INTERNAL.
- Namespace the ad-hoc 'type' span+metric attribute as github.record_type.
- Handle OTLP partial success: a 200 with rejectedSpans/DataPoints/LogRecords is now
  detected and logged (was silently discarded).
- Logs gain scope version, schemaUrl, and observedTimeUnixNano (parity with traces/metrics).
- Refresh the class doc to the job-is-root model + point at docs/otel-id-contract.md.

L0: RecordSpan_HonorsClientSpanKind, DescribePartialSuccess_FlagsRejectedItems,
Logs_HaveSchemaUrlAndObservedTime; 46/46 pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e tests

Split the gzip + retry + partial-success + HttpClient out of the 1400-line exporter
into OTelHttpTransport (constructor-injectable HttpMessageHandler). Closes the
'zero HTTP-transport tests' blocker: OTelHttpTransportL0 uses a fake handler to
verify URL routing, application/json + gzip Content-Encoding, gunzip->original OTLP,
auth-header attach, transient retry (503->200), no-retry on 4xx, partial-success
delivery, and best-effort behavior on an expired deadline.

51 OTel L0 tests pass (46 exporter + 5 transport).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…y, no behavior change)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…8/500

The OTLP/HTTP failures table limits retryable responses to 429, 502, 503,
504; all other 4xx/5xx MUST NOT be retried. IsTransientStatus previously
also retried 408 and 500 — a hard-MUST spec violation that doubles load on
a collector that is deliberately rejecting. Align the set with the spec,
cite the table in the code, cover 408/500 with L0 tests, and update the ADR.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Retrying a fixed 200 ms after an explicit 429/503 ignores the server's
signal (OTLP spec SHOULD honor Retry-After) and lets fleets retry in
lockstep. Now: an in-deadline Retry-After is waited out before the single
retry; one longer than the flush deadline could honor skips the retry
entirely; without the header the delay is a jittered 100-400 ms. The
deliberate one-retry bound (vs the spec's exponential-backoff SHOULD) is
documented in the ADR transport section. Delay is injectable so the L0
timing tests don't sleep for real.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
OTLP exporter spec SHOULD: send a User-Agent identifying the exporter and
its version so collector operators can attribute and police traffic from
runner fleets. Sends GitHubActionsRunner-OTLP-Exporter/<runner version>.

Also documents why gzip stays always-on with no opt-out knob: the OTLP
spec requires every server component to support gzip request bodies, so a
compression escape hatch would be config for a non-conformant server.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…UTF-8 bytes

One POST per signal meant a buffer-cap job (~18-40 MB uncompressed) could
exceed a collector's decompressed-body limit (OTel Collector default:
20 MiB) and take a 413 — non-retryable per OTLP — silently dropping the
whole signal, exactly on the big jobs where telemetry matters most.
FlushAsync now sends spans/logs in batches of MaxItemsPerPost (1,000,
~4 MB uncompressed) under the same single 4 s flush deadline, verified by
an L0 test that captures the wire requests.

The Build* serializers also stop round-tripping the payload through a
UTF-16 string (bytes -> string -> bytes before gzip): they return the
Utf8JsonWriter bytes directly, so flush no longer triple-materializes
large payloads. docs/otel-benchmarks.md's 'motivates gzip + chunking'
note is now implemented; ADR transport section updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
VarUtil returns "Linux"/"macOS"/"Windows" and "X86"/"X64"/"ARM"/"ARM64";
the semconv enums are lowercase (Apple is darwin — there is no macOS value)
and x86/amd64/arm32/arm64. Raw OS name is preserved in free-form os.name.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The registered semconv key is process.exit.code (type int), not
process.exit_code. RecordSpan attributes are now IDictionary<string, object>
so bool/int/long values keep their OTLP type (intValue) instead of being
coerced to strings; the exit code also no longer goes through
current-culture ToString() (a negative code under a non-ASCII NegativeSign
culture produced an unparseable value).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lly conforms to

1.29.0 lacks vcs.provider.name, vcs.owner.name, vcs.repository.name,
cicd.worker.id/name, cicd.system.component, cicd.pipeline.run.url.full,
cicd.pipeline.task.run.result and cicd.pipeline.result, all of which this
exporter emits; a wrong schema_url poisons schema-aware version
transformations. Audited every emitted key against the v1.34.0 registry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The job span is recorded last, so at the 10k cap the drop-newest policy
evicted exactly the trace root, orphaning every already-buffered step span
(they all parent to its deterministic ID). It now bypasses the cap (one per
job, worst case cap+1). ADR documents the reservation; the crash-before-
flush loss mode was already documented there as an accepted trade-off.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ding it

Host step processes inherit the worker's environment via ProcessInvoker, so
the raw collector credential reached every untrusted step (printenv) — the
exact leak the StepPropagationEnv comment promises can't happen. Read it
once in Initialize, then remove it (CommandSettings precedent), including
when export is disabled (credential without endpoint is still a credential).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A basic-auth-in-URL collector (https://user:token@...) embeds a credential
in the endpoint. It now never reaches step processes (StepPropagationEnv
hands out a stripped base URL) and is registered with the SecretMasker so
diag/transport log lines mask it — matching the existing posture of
withholding OTLP headers from steps. ADR documents that propagation is
host-perspective (localhost is unreachable from container jobs).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… INSECURE

The insecure flag disables ALL server-certificate validation, not the
'skip verification for self-signed collectors' the ADR implied — with
ACTIONS_RUNNER_OTLP_HEADERS on every POST, a MITM could harvest the
collector credential. Now: a PEM CA bundle (mirroring
OTEL_EXPORTER_OTLP_CERTIFICATE) is the supported primitive for
self-signed/private-CA collectors (hostname still checked, bad bundle
fails closed), and enabling INSECURE logs an explicit Trace.Warning
naming the MITM exposure — never silent. ADR documents both honestly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er throw into the job)

Record*/FlushAsync were already best-effort, but these entry points were
unguarded — and the first GetService<IOTelTraceExporter>() runs before
JobRunner's try block, so an unexpected throw (e.g. handler factory,
bad CA bundle) surfaced straight into job execution. Initialize failure
now disables export (fail-closed); the others log-and-continue like every
other hook. L0 failure-injection test covers the Initialize path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dropped

Every failure path was Trace.Info in per-job _diag files — effectively
silent at fleet scale (an expired collector credential just made traces
stop). FlushAsync now counts delivered vs failed POSTs and emits a single
greppable Trace.Warning summary (failed POSTs + cap-drop counts) only when
something was lost; a total flush failure warns likewise. Still swallowed —
the job is never affected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r values

Reviewed adding a name filter/length floor for ACTIONS_RUNNER_OTLP_HEADERS
values registered with the SecretMasker and rejected it: a heuristic would
leave short tokens or non-standard auth headers unmasked (fail-unsafe),
while over-masking a short non-secret value is only cosmetic and matches
the add-mask command's fail-safe convention. Comment records the decision.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rray/bytes)

The hand-rolled encoder coerced every non-bool/int attribute to
stringValue. OTLP AnyValue defines doubleValue, arrayValue and
bytesValue; the first caller to record a double would silently get a
string on the wire, breaking numeric queries downstream. Map each CLR
type to its wire type; only genuinely unsupported types stringify
(masked), and that arm is documented.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…only exception event

exception.type was set to the literal CI conclusion "failure", which is
not an exception class; backends that group errors by exception.type
would bucket every failed step in one meaningless group. Omit the faked
type (semconv allows message-only exception events; no message, no
event) and surface the error message as span status.message per the
Recording Errors guidance. error.type stays as the classifier.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e run

cicd-spans: pipeline-run spans SHOULD be SERVER (with
cicd.pipeline.result); task-run spans SHOULD be INTERNAL. The job span
carries the cicd.pipeline.task.* attribute set but was emitted as
SERVER, straddling both models. Emit it INTERNAL like the steps; the
SERVER run span belongs to the API-side consumer. Documented in
docs/otel-id-contract.md so span kind is a reliable discriminator.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ric honestly

github.pipeline.run.errors was the registered semconv metric
cicd.pipeline.run.errors wearing a vendor name (same instrument, unit
and required attrs cicd.pipeline.name + error.type; a failed job is 'an
error encountered in a pipeline run') — adopt the semconv name so
standard CI/CD dashboards pick it up.

github.pipeline.run.duration is NOT renamed to the semconv twin: the
worker observes one job's wall time, not the pipeline run's, so
cicd.pipeline.run.duration (whole-run duration grouped by
cicd.pipeline.run.state) would be semantically wrong (a 10-job run
would emit 10 bogus run-duration points, no queue time). Semconv has no
task-level duration metric, so it stays vendor-namespaced — renamed to
github.pipeline.job.duration to say what it measures. Rationale in the
ADR.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ction steps

- Move TRACEPARENT/OTEL_* injection into a shared Handler helper
  (AddOTelPropagationToEnvironment) and call it from ScriptHandler,
  NodeScriptActionHandler and ContainerActionHandler, so uses: (JS and
  Docker) action steps nest their telemetry too, matching the ADR's
  'each step's env' claim. Composite run: steps already flow through
  ScriptHandler.
- Precedence: workflow wins. Inject each key only when the step env does
  not already define it, so a step that sets its own TRACEPARENT/OTEL_*
  keeps them (the ADR's no-clobber promise, now enforced).
- ADR: state the precedence rule and the handler coverage explicitly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…Exporter

The global ServiceLocator fallback in TestHostContext.GetService silently
instantiated the real default for ANY unregistered service, removing the
deliberate missing-mock seam for every L0 test. Restore the throw and keep
the fallback only for an explicit allowlist (IOTelTraceExporter — resolved
lazily on many production paths, inert unless its env vars are set).

Adds TestHostContextL0 guarding both behaviors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Flush-before-complete is deliberate: the completion report triggers
ephemeral-runner teardown (ARC deletes the pod), so flushing after it
races pod deletion and drops the job's telemetry. Say so at both call
sites and in the ADR trade-off, including the bounded worst case
(4 s deadline, hung collector) vs the healthy-path cost (ms).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nchmarks

- New L0 Flush_HangingCollector_BoundedByOverallDeadline: a collector that
  accepts TCP then stalls (the case nothing fast-fails on) — asserts a real
  FlushAsync returns within the 4 s overall deadline (< 6 s with slack).
- Benchmarks doc: add a measured failure-mode table (refused < 1 s,
  hung ~4 s), a collector-down threshold row, and replace the n=3
  'no measurable overhead' claim with the bounded statement it supports.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…icitly

Every other Constants.Runner.Features gate defaults off; this one is
deliberately ?? true (endpoint opt-in is the real gate, ?? false would
kill the feature on GHES/self-hosted where the flag is never provisioned).
Name the deviation, and offer the conventional-polarity alternative
(negative flag actions_runner_otel_export_disabled) for maintainer
sign-off rather than leaving it implicit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ADR now reflects every shipped behavior from the hardening batches:
- child spans (action download, container pull/create, run: process with
  typed process.exit.code) and their INTERNAL/CLIENT kinds
- semconv enum mapping for os.type/host.arch
- typed OTLP AnyValue serialization + exporter User-Agent
- guarded entry points and the one end-of-job Warning loss summary
- no-length-floor masking policy for OTLP header values
- OTelHttpTransport named as the wire component

releaseNote.md gains the feature bullet (opt-in endpoint, deterministic
IDs, bounded best-effort flush, propagation precedence, kill switch).
otel-id-contract.md audited against the batches: no ID-affecting change,
already accurate (span kinds, vectors, TRACEPARENT derivation).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…C listener)

The runner emits cicd.worker.name and cicd.worker.id as resource attributes
(one worker per process). The ARC listener emits them as span attributes
because one listener process serves many runners.

Adds a new § with a table and a TraceQL query pattern covering both scopes,
so consumers can query `span.cicd.worker.name || resource.cicd.worker.name`
without missing either side of a merged trace.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant