Collect self-hosted worker logs for debug archives

README.md

@@ -423,6 +423,25 @@ shows up as a distinct series.
   attempts; spikes indicate flapping workers.
 - `oz_worker_info{version,backend,worker_id}` (gauge, value `1`): build and
   runtime metadata, useful for joining other series by labels.
+- `oz_worker_debug_archive_requests_total{backend,ownership,outcome,reason}`
+  (counter): debug-archive log requests this worker owned. Requests for
+  executions another instance ran are silent and are not counted here.
+- `oz_worker_debug_archive_snapshot_duration_seconds{backend}` and
+  `oz_worker_debug_archive_snapshot_bytes{backend}` (histograms): cost and
+  size of producing a log snapshot.
+- `oz_worker_debug_archive_truncations_total{backend}` (counter): snapshots
+  that dropped bytes to stay within their bound.
+- `oz_worker_debug_archive_uploads_total{result}` and
+  `oz_worker_debug_archive_upload_duration_seconds{result}`: snapshot upload
+  outcomes and latency.
+- `oz_worker_debug_archive_requests_in_flight` (gauge): requests currently
+  being snapshotted or uploaded, bounded by `debugLogCapture.maxConcurrentUploads`.
+- `oz_worker_debug_archive_capture_bytes` (gauge): disk currently reserved by
+  direct-execution captures and request snapshots.
+- `oz_worker_cleanup_grace_entries` (gauge) and
+  `oz_worker_cleanup_grace_results_total{backend,result}` (counter):
+  executions retained past terminal state and the backend cleanups performed
+  when their grace expired.
 
 ### Sample dashboards / alerts
 
@@ -441,6 +460,109 @@ Direct mappings for the questions enterprise operators most commonly ask:
 - **Reconnect storms:**
   `sum(rate(oz_worker_websocket_reconnects_total[5m])) > 0.1`
 
+## Debug archive log collection
+
+Warp can assemble a debug archive for a cloud-agent run. For self-hosted
+executions the logs live inside your infrastructure, so Warp asks the worker
+that actually ran the execution for a bounded snapshot instead of reaching into
+your Docker daemon, Kubernetes cluster, or host.
+
+When Warp requests logs for an execution, the worker that ran it snapshots the
+backed-up output, uploads it directly to a short-lived Warp-signed destination,
+and reports the result. Every other worker process silently ignores the request.
+Collection never blocks, delays, or fails a running agent: if logs cannot be
+captured or uploaded, the archive is simply marked partial.
+
+### Sensitive data
+
+A snapshot contains whatever the execution wrote to stdout and stderr, which
+can include prompts, source code, identifiers, and secrets a process printed.
+Treat a debug archive as sensitive. The worker itself never logs captured
+bytes, upload destinations, signed headers, local capture paths, or upload
+response bodies.
+
+### Supported backends
+
+| Backend | Source |
+| --- | --- |
+| Docker | The execution container's stdout and stderr, covering the entrypoint script and the client process it starts. |
+| Kubernetes | Every init and regular container in the execution's pods, including current and best-effort previous logs after a restart. |
+| Direct | A bounded on-disk capture of setup, agent, and teardown stdout/stderr. |
+| Command | Not supported. The dispatch command hands the task to an opaque runtime with no log API, so the worker reports the source unavailable rather than passing off dispatch output as the agent's log. |
+
+Docker and Kubernetes report one merged stream per container, so those records
+are labeled `combined` rather than attributing a line to the entrypoint or the
+client. Only direct execution owns distinct handles, so only it labels
+`setup`, `agent`, and `teardown` phases.
+
+### Sizing the cleanup grace for failure capture
+
+The worker retains an execution's log source for the cleanup grace it already
+resolves for `--idle-on-complete`, in this order:
+
+1. the run's `idle_timeout_minutes`
+2. the worker's `idle_on_complete` (`worker.idleOnComplete` in the chart)
+3. the Oz default of 45 minutes
+
+There is deliberately no separate archive retention setting: one clock governs
+both how long the agent stays available for follow-ups and how long its logs
+stay retrievable.
+
+Collection triggered by a failure has to reach the worker after the terminal
+event propagates to Warp, and the request itself allows up to 30 minutes. If
+you want reliable archives for failed runs, keep the grace comfortably longer
+than that; a shorter grace still works but produces a partial archive when it
+lapses first.
+
+**Kubernetes:** `kubernetesBackend.ttlSecondsAfterFinished` must not be shorter
+than the effective grace. The Job TTL controller deletes a finished Job's pods,
+and once they are gone their logs are gone with them regardless of the worker's
+own retention. A successful Job is now deleted at the grace deadline rather
+than immediately at task completion.
+
+**Worker replacement:** ownership is process-local. If a worker pod is replaced
+while a preserved Job keeps running, the replacement does not inherit the old
+process's ownership, and a later request for that execution yields a partial
+archive rather than an incorrect upload.
+
+### Capture bounds
+
+Each execution's snapshot is capped at 64 MiB. Output above the cap keeps the
+first and last portions with an explicit gap marker, so both the early setup
+context and the terminal failure survive.
+
+The Helm chart mounts a dedicated 1 GiB ephemeral volume for the capture root
+and renders the matching bounds:
+
+```yaml
+debugLogCapture:
+  enabled: true
+  sizeLimit: 1Gi
+  directory: /var/lib/oz/debug-logs
+  maxTotalBytes: 1073741824
+  maxExecutionBytes: 67108864
+  maxConcurrentUploads: 2
+```
+
+Outside Kubernetes, set the same bounds under `debug_log_capture` in the config
+file; an unset value uses the default above, and an unwritable root or an
+invalid bound disables archive capture without affecting task execution.
+
+The volume is scratch space, not storage: nothing in it survives worker
+replacement, and it does not extend the cleanup grace.
+
+### Compatibility
+
+Self-hosted logs in debug archives require a worker built with this protocol.
+An older worker ignores the request and Warp records the source as unavailable,
+so upgrading is safe and never required for ordinary task execution. Each
+authenticated connection reports its build version so Warp can show exactly
+which worker ran an execution; a worker that reports no version still executes
+tasks normally and is shown as not reported.
+
+The Kubernetes backend needs no additional permissions: the chart's existing
+namespace-scoped `get pods/log` grant is sufficient.
+
 ## License
 
 Copyright © 2026 Warp

charts/oz-agent-worker/templates/configmap.yaml

@@ -13,6 +13,14 @@ data:
     {{- if .Values.worker.idleOnComplete }}
     idle_on_complete: {{ .Values.worker.idleOnComplete | quote }}
     {{- end }}
+    debug_log_capture:
+      {{- if .Values.debugLogCapture.enabled }}
+      directory: {{ .Values.debugLogCapture.directory | quote }}
+      {{- end }}
+      {{- /* int64 keeps large byte counts out of YAML scientific notation. */}}
+      max_total_bytes: {{ .Values.debugLogCapture.maxTotalBytes | int64 }}
+      max_execution_bytes: {{ .Values.debugLogCapture.maxExecutionBytes | int64 }}
+      max_concurrent_uploads: {{ .Values.debugLogCapture.maxConcurrentUploads | int }}
     backend:
       kubernetes:
         namespace: {{ default .Release.Namespace .Values.kubernetesBackend.namespace | quote }}

charts/oz-agent-worker/templates/deployment.yaml

@@ -98,6 +98,10 @@ spec:
             - name: config
               mountPath: /etc/oz-agent-worker
               readOnly: true
+            {{- if .Values.debugLogCapture.enabled }}
+            - name: debug-log-capture
+              mountPath: {{ .Values.debugLogCapture.directory | quote }}
+            {{- end }}
           {{- with .Values.worker.livenessProbe }}
           livenessProbe:
             {{- toYaml . | nindent 12 }}
@@ -114,6 +118,14 @@ spec:
         - name: config
           configMap:
             name: {{ include "oz-agent-worker.fullname" . }}-config
+        {{- if .Values.debugLogCapture.enabled }}
+        # Scratch space for debug-archive log capture. It is explicitly
+        # ephemeral: nothing here survives worker pod replacement, and it does
+        # not change the execution cleanup-grace contract.
+        - name: debug-log-capture
+          emptyDir:
+            sizeLimit: {{ .Values.debugLogCapture.sizeLimit | quote }}
+        {{- end }}
       {{- with .Values.worker.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}

charts/oz-agent-worker/values.yaml

@@ -91,9 +91,15 @@ kubernetesBackend:
   # How long finished task Jobs that the worker leaves in place are retained before
   # the Kubernetes Job TTL controller deletes them (and their pods). This covers
   # failed task Jobs, which are kept for post-mortem debugging, and Jobs orphaned by
-  # worker disruption. Successful task Jobs are deleted immediately by the worker, so
-  # this does not keep them around. Defaults to 24h (86400s). When worker.cleanup=false
-  # the worker sets no TTL, so these Jobs remain indefinitely.
+  # worker disruption.
+  #
+  # A successful task Job is deleted by the worker at its execution's cleanup-grace
+  # deadline (task idle_timeout_minutes, then worker.idleOnComplete, then the Oz
+  # default) rather than at task completion, so its logs remain readable for a debug
+  # archive. Keep this TTL at least as long as the effective grace: if the controller
+  # deletes the pods first, a debug archive collected for that run is partial.
+  # Defaults to 24h (86400s). When worker.cleanup=false the worker sets no TTL, so
+  # these Jobs remain indefinitely.
   ttlSecondsAfterFinished: 86400
   workspaceSizeLimit: ""
   unschedulableTimeout: "30s"
@@ -116,6 +122,30 @@ kubernetesBackend:
   #       memory: 64Mi
   preflightResources: {}
 
+# Bounds for debug-archive log capture. The worker uses this space for
+# direct-execution output captures and for the request-scoped snapshot it
+# uploads; Docker and Kubernetes executions read provider logs on demand and
+# keep nothing here.
+#
+# There is deliberately no retention duration: how long an execution's logs stay
+# retrievable is the existing idle-on-complete cleanup grace (task
+# `idle_timeout_minutes`, then `worker.idleOnComplete`, then the Oz default).
+debugLogCapture:
+  # Mounts a dedicated ephemeral volume for the capture root. Disable only if
+  # the container filesystem already provides suitable scratch space.
+  enabled: true
+  # Size of the ephemeral volume. Keep it at or above maxTotalBytes so the
+  # worker's own accounting, not the volume, is the binding limit.
+  sizeLimit: 1Gi
+  # Mount path for the capture volume, used as the worker's capture root.
+  directory: /var/lib/oz/debug-logs
+  # Process-local disk budget shared by captures and request snapshots.
+  maxTotalBytes: 1073741824
+  # Per-execution retention bound. May not exceed the protocol's 64 MiB ceiling.
+  maxExecutionBytes: 67108864
+  # Concurrent snapshot/upload handlers per worker process.
+  maxConcurrentUploads: 2
+
 # Metrics export uses the OpenTelemetry autoexport package, so the operator
 # selects the exporter via standard OpenTelemetry environment variables. When
 # `metrics.enabled` is false the worker emits no metrics, matching the

internal/common/cleanup_grace_test.go

@@ -0,0 +1,100 @@
+package common
+
+import (
+	"testing"
+	"time"
+
+	"github.com/warpdotdev/oz-agent-worker/internal/types"
+)
+
+func taskWithIdleTimeout(minutes int) *types.Task {
+	return &types.Task{
+		ID:                  "task-1",
+		AgentConfigSnapshot: &types.AmbientAgentConfig{IdleTimeoutMinutes: &minutes},
+	}
+}
+
+func TestResolveCleanupGracePrecedence(t *testing.T) {
+	tests := []struct {
+		name           string
+		task           *types.Task
+		idleOnComplete string
+		want           time.Duration
+	}{
+		{
+			name:           "task idle_timeout_minutes wins",
+			task:           taskWithIdleTimeout(90),
+			idleOnComplete: "10m",
+			want:           90 * time.Minute,
+		},
+		{
+			name:           "worker idle_on_complete is next",
+			task:           &types.Task{ID: "task-1"},
+			idleOnComplete: "10m",
+			want:           10 * time.Minute,
+		},
+		{
+			name:           "oz default is the fallback",
+			task:           &types.Task{ID: "task-1"},
+			idleOnComplete: "",
+			want:           DefaultIdleOnComplete,
+		},
+		{
+			name:           "a zero worker override disables retention",
+			task:           &types.Task{ID: "task-1"},
+			idleOnComplete: "0s",
+			want:           0,
+		},
+		{
+			name:           "a non-positive task timeout falls through to the worker override",
+			task:           taskWithIdleTimeout(0),
+			idleOnComplete: "5m",
+			want:           5 * time.Minute,
+		},
+		{
+			name:           "an unparseable worker override falls back to the default",
+			task:           &types.Task{ID: "task-1"},
+			idleOnComplete: "not-a-duration",
+			want:           DefaultIdleOnComplete,
+		},
+		{
+			name:           "a nil task uses the worker override",
+			task:           nil,
+			idleOnComplete: "15m",
+			want:           15 * time.Minute,
+		},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			if got := ResolveCleanupGrace(tc.task, tc.idleOnComplete); got != tc.want {
+				t.Fatalf("grace = %v, want %v", got, tc.want)
+			}
+		})
+	}
+}
+
+func TestResolveCleanupGraceMatchesTheEmittedIdleOnCompleteFlag(t *testing.T) {
+	// The retention window and the flag the agent receives must stay in step:
+	// an operator tuning one is tuning both.
+	task := taskWithIdleTimeout(30)
+	args := AugmentArgsForTask(task, nil, TaskAugmentOptions{IdleOnComplete: "10m"})
+
+	var emitted string
+	for i, arg := range args {
+		if arg == "--idle-on-complete" && i+1 < len(args) {
+			emitted = args[i+1]
+		}
+	}
+	if emitted != "30m" {
+		t.Fatalf("emitted --idle-on-complete %q, want 30m", emitted)
+	}
+
+	emittedDuration, err := time.ParseDuration(emitted)
+	if err != nil {
+		t.Fatalf("the emitted flag %q is not a duration: %v", emitted, err)
+	}
+	if grace := ResolveCleanupGrace(task, "10m"); grace != emittedDuration {
+		t.Fatalf("cleanup grace = %s, want it to match the emitted flag %s", grace, emitted)
+	}
+}

internal/common/task_utils.go

@@ -4,10 +4,15 @@ import (
 	"encoding/json"
 	"fmt"
 	"strings"
+	"time"
 
 	"github.com/warpdotdev/oz-agent-worker/internal/types"
 )
 
+// DefaultIdleOnComplete mirrors the oz CLI's default --idle-on-complete value,
+// used when neither the task nor the worker sets one.
+const DefaultIdleOnComplete = 45 * time.Minute
+
 // TaskAugmentOptions contains settings translated into oz CLI flags for every task.
 // Add new CLI overrides here rather than as extra parameters.
 type TaskAugmentOptions struct {
@@ -148,6 +153,28 @@ func shareAccessLevelForEmission(access types.AccessLevel) string {
 	}
 }
 
+// ResolveCleanupGrace returns how long the agent stays alive after its
+// conversation finishes, using the same precedence as the emitted
+// --idle-on-complete flag: task idle_timeout_minutes, then the worker's
+// idle_on_complete, then the oz CLI default.
+//
+// The worker reuses this window as its post-terminal ownership and
+// log-retrieval deadline, so a debug-archive request has one cleanup clock to
+// reason about instead of a second archive-specific retention setting. An
+// unparseable worker override falls back to the default rather than dropping
+// retention to zero.
+func ResolveCleanupGrace(task *types.Task, workerIdleOnComplete string) time.Duration {
+	value, ok := resolveIdleOnComplete(task, TaskAugmentOptions{IdleOnComplete: workerIdleOnComplete})
+	if !ok {
+		return DefaultIdleOnComplete
+	}
+	grace, err := time.ParseDuration(value)
+	if err != nil || grace < 0 {
+		return DefaultIdleOnComplete
+	}
+	return grace
+}
+
 func resolveIdleOnComplete(task *types.Task, opts TaskAugmentOptions) (string, bool) {
 	if task != nil &&
 		task.AgentConfigSnapshot != nil &&

internal/config/config.go

@@ -26,6 +26,26 @@ type FileConfig struct {
 	// overrides are no longer needed.
 	IdleOnComplete *string       `yaml:"idle_on_complete"`
 	Backend        BackendConfig `yaml:"backend"`
+	// DebugLogCapture bounds the disk and concurrency used to collect debug
+	// archive logs. It carries no retention duration: how long an execution's
+	// logs stay retrievable is the existing idle_on_complete cleanup grace.
+	DebugLogCapture *DebugLogCaptureConfig `yaml:"debug_log_capture"`
+}
+
+// DebugLogCaptureConfig bounds debug-archive log capture. Values left unset
+// use the worker's built-in defaults.
+type DebugLogCaptureConfig struct {
+	// Directory overrides the root beneath which captures and request
+	// snapshots are written. Empty uses the process temporary directory.
+	Directory string `yaml:"directory"`
+	// MaxTotalBytes is the process-local disk budget shared by direct-execution
+	// captures and request snapshots.
+	MaxTotalBytes *int64 `yaml:"max_total_bytes"`
+	// MaxExecutionBytes bounds one execution's retained output. It may not
+	// exceed the protocol's 64 MiB ceiling.
+	MaxExecutionBytes *int64 `yaml:"max_execution_bytes"`
+	// MaxConcurrentUploads bounds how many snapshot/upload handlers run at once.
+	MaxConcurrentUploads *int `yaml:"max_concurrent_uploads"`
 }
 
 // BackendConfig contains the backend selection.