Add optional native OpenTelemetry export for runner, jobs, and steps

.dockerignore

@@ -0,0 +1,9 @@
+_layout
+_dotnetsdk
+_downloads
+_work
+_package
+_diag
+ghalistener
+**/bin/Debug
+**/obj

docs/otel-benchmarks.md

@@ -0,0 +1,93 @@
+# Native OTel export — performance & memory benchmarks
+
+Overhead of the native OpenTelemetry export (PR #4366). Goal: prove the
+instrumentation is cheap enough to run on every job and bounded in memory.
+
+## Micro-benchmark (emit + serialize)
+
+`src/Test/L0/Worker/OTelTraceExporterBenchmark.cs` — reuses the exporter's internal
+hooks; measures per-step record cost, end-of-job OTLP/JSON serialize cost, payload
+size, and allocations. Run:
+
+```
+dotnet test src/Test/Test.csproj -c Release \
+  --filter "FullyQualifiedName~OTelTraceExporterBenchmark" \
+  --logger "console;verbosity=detailed"
+```
+
+Results (Release, net8.0, Apple M-series; representative):
+
+| spans  | emit (ns/step) | alloc (B/step) | retained (B/span) | serialize | payload (B/span) |
+|-------:|---------------:|---------------:|------------------:|----------:|-----------------:|
+| 10     | ~6,500         | ~4,200         | ~5,700            | 0.09 ms   | 1,829            |
+| 100    | ~2,800         | ~4,120         | ~4,280            | 0.83 ms   | 1,806            |
+| 1,000  | ~2,300         | ~4,110         | ~4,150            | 11.2 ms   | 1,807            |
+| 10,000 | ~3,500         | ~4,150         | ~3,230            | 97 ms     | 1,811            |
+
+### Reading it
+- **Emit: ~2–6 µs per step.** Negligible next to real step durations (seconds–minutes).
+  Cost is 3× SHA-256 (deterministic IDs) + ~30 attribute sets per span.
+- **Memory: ~4 KB allocated and retained per span**, held until the single end-of-job
+  flush. Linear and freed at job end:
+  - 50 steps ≈ 0.2 MB · 200 ≈ 0.8 MB · 1,000 ≈ 4 MB · **10,000 ≈ ~32 MB**
+  - This is the case for a per-buffer cap (see below): a pathological job (100k
+    steps/annotations) would retain hundreds of MB.
+- **Serialize: ~10 µs/span** (one `Utf8JsonWriter` pass at flush). 11 ms @ 1k, ~100 ms @ 10k.
+- **Payload: ~1.8 KB/span** of OTLP/JSON, gzipped and chunked into POSTs of at most
+  `MaxItemsPerPost` (1,000) spans/logs each (→ ~4 MB uncompressed per request even
+  at the 10k buffer cap, safely under the OTel Collector's 20 MiB default body limit).
+
+## Macro-benchmark (job wall-time, ON vs OFF) — measured
+
+A 50-trivial-step workflow (`run: ":"`) on the local self-hosted runner, OTel ON
+(healthy collector) vs OFF (endpoint unset → exporter disabled), job wall-time from
+the GitHub job `started_at`→`completed_at`:
+
+| | run 1 | run 2 | run 3 |
+|---|---|---|---|
+| **OTel ON**  | 18 s | 25 s | 18 s |
+| **OTel OFF** | 18 s | 25 s | 25 s |
+
+The two distributions **fully overlap**: job-to-job variance (~7 s, from runner
+orchestration/scheduling) dwarfs any OTel signal. **Caveat: n=3 per arm.** With
+samples this small the honest claim is bounded, not statistical: any ON/OFF delta
+is below the ~7 s orchestration noise floor, consistent with the micro numbers
+(µs/step emit + a single bounded, gzipped flush POST at job end). The hard
+guarantee comes from the failure-mode bounds below, not from these three runs.
+
+## Failure modes (collector down) — measured
+
+The job-end flush is the only place a bad collector can cost wall time. Both
+failure classes are measured in L0 (run on every CI pass, not just benchmarked
+once), timing a real `FlushAsync` against a real local socket:
+
+| collector failure | test | measured job-end delay |
+|---|---|---|
+| Connection refused (down, NXDOMAIN-like fast-fail) | `Flush_UnreachableEndpoint_DoesNotThrow` | **< 1 s** (fail + one 200 ms retry) |
+| Accepts TCP, then stalls (blackhole / slow-loris) | `Flush_HangingCollector_BoundedByOverallDeadline` | **~4 s**, asserted < 6 s |
+
+The stall case is the worst case: nothing fast-fails, so the 4 s overall flush
+deadline (`flushCts.CancelAfter(4 s)`, one deadline across all three signals
+including the retry) is what bounds it — and the test asserts that bound holds
+against a genuinely hung socket. DNS failure and blackholed IPs collapse to the
+same two classes: either the socket op errors fast (row 1) or the cancellation
+deadline fires (row 2). Flush is best-effort and swallowed — it never fails the
+job; the loss is reported in one end-of-job Warning summary.
+
+## Sampling / scale model
+
+Decision: **server-controlled**. The runner emits every span/metric/log
+unconditionally (gated only by the server feature flag + the operator's endpoint
+opt-in). Sampling and metric-cardinality control are the **collector's / server's**
+responsibility (tail sampling, cardinality limits, rate limiting) — not the runner's.
+This keeps the runner simple, preserves tail visibility (all failures), and matches
+how export is already gated. Memory is bounded at the source by the per-buffer caps
+(see code: `MaxBufferedSpans/Logs/TaskMetrics`).
+
+## Acceptable-overhead thresholds (proposed)
+- Disabled (common case): job wall-time delta < 0.1 %; RSS delta within noise.
+- Enabled, healthy collector: added job-end wall-time < ~100 ms.
+- Enabled, collector down: added job-end wall-time ≤ ~4 s (the flush deadline;
+  measured above, asserted in L0).
+- Per-step emit < ~50 µs. Serialize < ~10 ms for 200 spans.
+- Peak RSS delta a few KB/span (cap buffers so a runaway job can't OOM the worker).

docs/otel-id-contract.md

@@ -0,0 +1,153 @@
+# OpenTelemetry deterministic ID contract
+
+**Status**: Normative. This document is the source of truth for the deterministic
+trace/span IDs the runner's native OTLP exporter emits (see
+[ADR 4366](adrs/4366-native-otel-export.md)). It is owned by this repository.
+
+Any independent consumer that reconstructs a GitHub Actions trace from the public
+API (timeline / REST) MUST compute IDs by this spec to merge with runner-emitted
+spans. [`stefanpenner/otel-explorer`](https://github.com/stefanpenner/otel-explorer)
+is **one example consumer** of this published contract, not a byte-compatibility
+obligation: this document, not that repo, defines the contract.
+
+The reference implementation is `OTelTraceExporter` in
+`src/Runner.Worker/OTelTraceExporter.cs` (functions `NewTraceID`, `NewJobSpanID`,
+`NewStepSpanID`, `NewSpanIDFromString`, `NewSpanID`).
+
+## Why deterministic IDs
+
+Two systems describe the same run from different vantage points:
+
+- the **runner**, post-hoc, from its `TimelineRecord`s (sub-second step timing,
+  runner-host context), and
+- the **GitHub API path**, which reconstructs the run/workflow span and
+  server-side queue time.
+
+Deriving every ID from stable identifiers both sides already know means the two
+sets of spans **dedupe and merge into one trace** with no shared state, no
+coordination, and no W3C propagation between them. The same property makes
+**re-runs** trivial: each attempt is its own trace.
+
+## Algorithm
+
+```
+id = truncate( SHA-256( UTF-8( input_string ) ), N bytes )  → lowercase hex
+```
+
+- **SHA-256**, not MD5. MD5 throws on a FIPS-enabled host; SHA-256 gives the same
+  deterministic-from-`run_id` behavior. Only the **leading bytes** of the digest
+  are used.
+- **Trace ID**: leading **16 bytes** → **32 hex chars**.
+- **Span ID**: leading **8 bytes** → **16 hex chars**.
+- Hex is lowercase, zero-padded per byte (`x2`).
+- `run_attempt` of `0` (or absent) is normalized to `1` before hashing.
+- IDs are intentionally **non-random**, so the W3C trace-context Level-2
+  randomness flag is never set.
+
+## Input strings
+
+| ID | Bytes | Input string (UTF-8) |
+|----|-------|----------------------|
+| Trace ID | 16 | `"{run_id}-{run_attempt}"` |
+| Job span | 8 | `"job-{run_id}-{run_attempt}-{job_name}"` |
+| Step span | 8 | `"step-{run_id}-{run_attempt}-{job_name}-{step_number}-{step_name}"` |
+| Child span | 8 | `"{span_type}-{run_id}-{run_attempt}-{job_name}-{name}-{start_unix_nano}"` |
+
+Field definitions:
+
+- **`run_id`** — GitHub `github.run_id` (the workflow run id).
+- **`run_attempt`** — `github.run_attempt`, 1-based (`0`/absent → `1`).
+- **`job_name`** — the job's **display name** (matches the GitHub API `job.name`),
+  not the `github.job` context key.
+- **`step_number`** — the **1-based** top-level step number, matching the GitHub
+  API `step.number`. It disambiguates two top-level steps that share a display
+  name while keeping the ID mergeable with the API trace.
+- **`step_name`** — the step's display name.
+- **`span_type`**, **`name`**, **`start_unix_nano`** — for generic child spans
+  (e.g. action download). The start time is included so repeated same-named
+  operations don't collide. Child spans are runner-only; the API path has no
+  counterpart, so they are not a merge point.
+
+### Workflow / run span ID (API-path only)
+
+The trace also contains a **run/workflow span** whose ID is the **big-endian
+8-byte encoding of `run_id`** (`NewSpanID(run_id)`), *not* a hash. The runner
+**does not emit this span** and **does not parent its job span to it** — the job
+span is the trace root (`parentSpanId` omitted). The encoding is retained in the
+runner and published here only so the **API path** can emit the run span into the
+same trace; the runner's job span merges by sharing the trace ID.
+
+```
+run/workflow span = bigendian_uint64(run_id)   → 16 hex
+```
+
+## Trace shape
+
+```
+trace  sha256("{run_id}-{run_attempt}")[:16]
+└─ (run/workflow span  bigendian(run_id))      ← emitted by the API path, not the runner
+   └─ job span   sha256("job-…")[:8]            ← runner: trace ROOT (no parent)
+      ├─ step span   sha256("step-…-{n}-…")[:8]
+      │  └─ child span  sha256("{type}-…")[:8]
+      └─ …
+```
+
+An inbound scheduler `traceparent` (if present) is attached to the job span as a
+**link**, not a parent — it does not change any ID above.
+
+### Span kinds
+
+Per the CI/CD span conventions (cicd-spans), **task-run spans are `INTERNAL`**
+and **pipeline-run spans are `SERVER`** (and carry `cicd.pipeline.result`). The
+runner's job and step spans are task runs (`cicd.pipeline.task.*`), so the
+runner emits them as `INTERNAL`; the `SERVER` pipeline-run span belongs to the
+API path with the run/workflow span above. Span kind is therefore a structural
+discriminator: `SERVER` = the run, `INTERNAL` = tasks.
+
+## Attribute scope: cicd.worker.*
+
+`cicd.worker.name` and `cicd.worker.id` identify the runner host that executed
+a job. Their OTel **scope** differs between the runner and the ARC listener:
+
+| Component | Scope | Rationale |
+|-----------|-------|-----------|
+| Runner (`OTelTraceExporter.cs`) | **Resource** attribute | One worker process per runner host; the identity is constant for the process lifetime. |
+| ARC listener (`cmd/ghalistener/metrics/otel.go`) | **Span** attribute | One listener process serves many runners; the worker identity varies per job span. |
+
+Both carry the same key names and value semantics. A TraceQL query that must
+match spans from either component should check both scopes:
+
+```
+span.cicd.worker.name != "" || resource.cicd.worker.name != ""
+```
+
+## On enumerability of trace IDs
+
+Trace IDs derive from the **public, enumerable `run_id`** (a known, guessable
+property of a run). This is **acceptable for CI traces**: the data is workflow
+metadata, the collector is operator-controlled, and the merge property requires a
+shared, reproducible trace ID. It is a deliberate trade-off, not an oversight — do
+not treat a runner trace ID as a secret.
+
+## Conformance vectors
+
+Golden values from the L0 suite (`src/Test/L0/Worker/OTelTraceExporterL0.cs`). A
+conforming implementation MUST reproduce these exactly.
+
+| Function | Input | Output |
+|----------|-------|--------|
+| Trace ID | `run_id=99999, run_attempt=1` → `sha256("99999-1")[:16]` | `acad1e2a107636235fd56bb742499bd0` |
+| Job span | `99999, 1, "build"` → `sha256("job-99999-1-build")[:8]` | `81606d47848a59c0` |
+| Step span | `99999, 1, "build", 3, "Run tests"` → `sha256("step-99999-1-build-3-Run tests")[:8]` | `7a4c67339b7bb8a7` |
+| Run/workflow span | `bigendian(99999)` | `000000000001869f` |
+| Run/workflow span | `bigendian(42)` | `000000000000002a` |
+
+Derived check — the propagated `TRACEPARENT` for that step is
+`00-acad1e2a107636235fd56bb742499bd0-7a4c67339b7bb8a7-01`
+(`00-{traceId}-{stepSpanId}-01`).
+
+Normalization checks:
+
+- `run_attempt = 0` produces the same IDs as `run_attempt = 1`.
+- The same inputs always produce the same IDs; different `run_attempt` produces a
+  different trace ID.

otel-runner.Dockerfile

@@ -0,0 +1,39 @@
+# Build an ARC-compatible runner image carrying the native-OTel runner
+# (actions/runner#4366). We build the linux layout from source, then overlay it
+# onto the official actions-runner image (keeping its runner user, k8s hooks,
+# and entrypoint).
+#
+#   docker build -f otel-runner.Dockerfile -t otel-runner:dev .
+#   kind load docker-image otel-runner:dev --name gha-runner
+
+# ---- builder: compile the runner layout from source (linux/arm64 native) ----
+FROM mcr.microsoft.com/dotnet/sdk:8.0-noble AS builder
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends curl git unzip ca-certificates \
+ && rm -rf /var/lib/apt/lists/*
+ENV DOTNET_CLI_TELEMETRY_OPTOUT=1 DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
+WORKDIR /runner
+COPY . .
+WORKDIR /runner/src
+# The reported runner version must be >= GitHub's current minimum, else the
+# Actions service denies the ephemeral/JIT runner (AccessDenied -> exit code 7),
+# which ARC interprets as "Outdated" and tears the runner down in a loop.
+# PR#4366 branches off 2.333.0; bump the reported version to a current one.
+ARG RUNNER_VERSION_OVERRIDE=2.335.1
+RUN printf '%s' "${RUNNER_VERSION_OVERRIDE}" > runnerversion
+# dev.sh bootstraps its own pinned dotnet SDK and downloads externals (node).
+RUN ./dev.sh layout Release
+
+# ---- runtime: official ARC runner image + our instrumented binaries ----
+FROM ghcr.io/actions/actions-runner:latest
+# Merge the source-built layout over the official runner home. This replaces
+# bin/externals/run.sh/config.sh/env.sh with our OTel build while keeping the
+# official image's k8s hooks, run-helper.sh, safe_sleep.sh, and entrypoint.
+COPY --from=builder --chown=runner:runner /runner/_layout/. /home/runner/
+
+# Bake the runner launch command into the image. ARC normally sets
+# `command: [/home/runner/run.sh]` on the pod, but doing so in chart values
+# triggers an "Outdated" hash-reconcile loop in gha-runner-scale-set 0.14.1.
+# Baking it here lets the values stay command-free (loop-safe).
+WORKDIR /home/runner
+CMD ["/home/runner/run.sh"]

releaseNote.md

@@ -27,6 +27,7 @@
 * Add new env var to allow single-prefix multiline logs on stdout by @nuclearpidgeon in https://github.com/actions/runner/pull/4424
 * Bump Microsoft.DevTunnels.Connections from 1.3.39 to 1.3.48 by @dependabot[bot] in https://github.com/actions/runner/pull/4441
 * Bump System.Formats.Asn1 and System.Security.Cryptography.Pkcs by @dependabot[bot] in https://github.com/actions/runner/pull/4369
+* Native OpenTelemetry export (opt-in) by @stefanpenner in https://github.com/actions/runner/pull/4366 — with `ACTIONS_RUNNER_OTLP_ENDPOINT` set, the runner emits OTLP/HTTP traces, metrics, and logs per job: job/step/child spans with deterministic (SHA-256) IDs, `cicd.*`/`vcs.*` semconv attributes (schema 1.34.0), annotation logs, and duration/error metrics. Best-effort and bounded: secret-masked output, capped buffers, gzip + chunked POSTs with a single bounded retry, 4 s job-end flush deadline, one Warning summary on export loss. Optional `ACTIONS_RUNNER_OTLP_PROPAGATE=true` injects `TRACEPARENT`/`OTEL_*` into step env (workflow-set values always win). Server kill switch: `actions_runner_otel_export`. See docs/adrs/4366-native-otel-export.md and docs/otel-id-contract.md.
 
 ## New Contributors
 * @GitPaulo made their first contribution in https://github.com/actions/runner/pull/4383

src/Runner.Common/Constants.cs

@@ -180,6 +180,7 @@ public static class Features
                 public static readonly string BatchActionResolution = "actions_batch_action_resolution";
                 public static readonly string UseBearerTokenForCodeload = "actions_use_bearer_token_for_codeload";
                 public static readonly string OverrideDebuggerWelcomeMessage = "actions_runner_override_debugger_welcome_message";
+                public static readonly string RunnerOtelExport = "actions_runner_otel_export";
             }
 
             // Node version migration related constants
@@ -312,6 +313,20 @@ public static class Agent
                 public static readonly string ActionArchiveCacheDirectory = "ACTIONS_RUNNER_ACTION_ARCHIVE_CACHE";
                 public static readonly string SymlinkCachedActions = "ACTIONS_RUNNER_SYMLINK_CACHED_ACTIONS";
                 public static readonly string EmitCompositeMarkers = "ACTIONS_RUNNER_EMIT_COMPOSITE_MARKERS";
+
+                // Native OpenTelemetry trace export (see docs/adrs/4366-native-otel-export.md).
+                public static readonly string OtlpEndpoint = "ACTIONS_RUNNER_OTLP_ENDPOINT";
+                public static readonly string OtlpHeaders = "ACTIONS_RUNNER_OTLP_HEADERS";
+                // Path to a PEM CA bundle trusted for the collector connection (the safe
+                // primitive for self-signed collectors; mirrors OTEL_EXPORTER_OTLP_CERTIFICATE).
+                public static readonly string OtlpCertificate = "ACTIONS_RUNNER_OTLP_CERTIFICATE";
+                public static readonly string OtlpInsecure = "ACTIONS_RUNNER_OTLP_INSECURE";
+                public static readonly string OtlpPropagate = "ACTIONS_RUNNER_OTLP_PROPAGATE";
+                // Inbound W3C trace context from an upstream scheduler (e.g. ARC): the job
+                // span gets a span LINK to this context (cross-trace causality), keeping the
+                // runner's deterministic IDs intact.
+                public static readonly string OtlpParentTraceparent = "ACTIONS_RUNNER_PARENT_TRACEPARENT";
+                public static readonly string OtlpParentTracestate = "ACTIONS_RUNNER_PARENT_TRACESTATE";
             }
 
             public static class System