Add optional OpenTelemetry trace export for job lifecycle

[stefanpenner]

Summary

• Adds an OpenTelemetry trace recorder that implements the existing listener.MetricsRecorder interface alongside the Prometheus exporter
• When configured with an OTLP endpoint, emits three child spans per completed job: runner.queue, runner.startup, runner.execution
• Uses a CompositeRecorder to fan out to both Prometheus and OTel when both are enabled
• Zero behavior change when otel_endpoint is not set

Motivation

GitHub Actions workflows can be reconstructed as OpenTelemetry traces (workflow → job → step), but there's a visibility gap between "job was queued" and "step started executing." ARC has the timestamps that explain this gap — QueueTime, ScaleSetAssignTime, RunnerAssignTime, FinishTime — but currently only exposes them as Prometheus histogram aggregates.

This PR emits those timestamps as individual trace spans, giving per-job visibility into:

• Queue wait — time before ARC acquires the job
• Runner startup — time for pod creation and runner registration
• Execution — time spent running the job

Trace correlation

Spans use deterministic IDs (TraceID = MD5(runID-attempt), SpanID = BigEndian(jobID)) that are compatible with tools like otel-explorer which reconstruct workflow traces from the GitHub API. ARC's runner spans automatically merge into the same trace as the workflow/job/step spans — no correlation configuration needed.

Configuration

{
  "otel_endpoint": "otel-collector.monitoring:4318",
  "otel_insecure": true
}

Or via Helm values:

listenerTemplate:
  spec:
    containers:
      - name: listener
        env:
          - name: OTEL_ENDPOINT
            value: "otel-collector.monitoring:4318"

Files changed

File	Change
cmd/ghalistener/metrics/otel.go	OTelRecorder implementing listener.MetricsRecorder
cmd/ghalistener/metrics/composite.go	CompositeRecorder fan-out wrapper
cmd/ghalistener/metrics/otel_test.go	Tests for recorder, composite, and deterministic IDs
cmd/ghalistener/main.go	Wire OTel recorder alongside Prometheus
cmd/ghalistener/config/config.go	Add otel_endpoint and otel_insecure fields
go.mod / go.sum	Add OTel SDK dependencies

Test plan

• All existing metrics tests pass
• New tests: 3-span emission, missing timestamps, common attributes, run attempt override, no-op methods, composite delegation, deterministic ID stability
• go build ./cmd/ghalistener/ compiles clean
• Integration test with real ARC deployment + OTel Collector
• Verify spans appear in Tempo/Jaeger alongside workflow traces

🤖 Generated with Claude Code

[murphpdx]

I would love to have this!

[austinpray-mixpanel]

I would also love this