feat(worker): report container setup phase durations as run client events - #125
Merged
Merged
Conversation
…ents The worker now reports the duration of each Docker setup phase to warp-server. The phases are the image pull, the sidecar preparation, and the container create and start. The worker sends each phase as a run client event with a setup metric payload. It uses the task API key and the workload token from the task assignment. The server records the events in the metric ambient_agents/client/event_latency_ms with the timeline phase oz_run_claimed. This data shows where the time goes between the claim of a run and the container ready event. The reports are best-effort and asynchronous. A report failure does not change task execution. Co-Authored-By: Warp Agent <agent@warp.dev>
…vents The Kubernetes backend now reports setup phase durations to warp-server. The kubelet performs the image pulls and the container starts for this backend, so the worker derives the phases from pod status timestamps instead of direct measurement. The backend times the Job create call and reports it as setup_worker_job_create. A tracker inspects each pod snapshot from the pod watch and the safety poll. It reports setup_worker_pod_schedule from pod creation to the PodScheduled condition, setup_worker_sidecar_prep across the copy-sidecar init containers, setup_worker_setup_command for the setup init container, and setup_worker_container_start from the end of init to the task container start. The tracker reports each phase one time. A failed init container reports its phase with is_error set. Pods without init containers (image-volumes mode) report only the schedule and container start phases. Co-Authored-By: Warp Agent <agent@warp.dev>
Contributor
Author
|
This PR was generated with Warp.   |
Address review feedback on the setup phase reporting. Share the task pod container names between the pod-spec construction in kubernetes.go and the phase tracker. Both sides now use the kubernetesTaskContainerName, kubernetesSetupContainerName, and kubernetesSidecarInitPrefix constants, so a container rename cannot silently break phase reporting. Log a warning when setup event reporting is disabled for a task. The warning states the reason: no server root URL, or no task API key in the assignment env vars. A fleet-wide config or credential change is now visible in the worker logs. Do not record cancellations as setup errors. The new startPhase and startPhaseIf helpers time each phase and skip the report when the context was cancelled, because a cancelled phase is neither a success nor a failure. The Docker and Kubernetes direct-timing sites now use these helpers. Skip the Docker sidecar-prep phase for tasks without sidecars, matching the Kubernetes backend, so zero-sidecar tasks do not emit near-zero samples that skew the phase percentiles. Co-Authored-By: Warp Agent <agent@warp.dev>
captainsafia
marked this pull request as ready for review
August 12, 2026 21:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The worker reports the duration of each setup phase to warp-server as run client events, for the Docker and Kubernetes backends. This closes the observability gap between the
oz_run_claimedandworker_container_readytimeline events. Before this change, that window was opaque: the worker sent no signals between the task claim and the first API call from the container.Changes
setupEventReporter(internal/worker/setup_events.go). It sends one client event for each completed setup phase toPOST /api/v1/agent/runs/{runId}/client-events. It authenticates with the per-task API key and the workload token from the task assignment.setup_worker_image_pull: the image pull, including the no-op pull when the image is cached.setup_worker_sidecar_prep: the sidecar volume preparation, including the sidecar image pull.setup_worker_container_start: the container create call through the container start call.internal/worker/kubernetes_setup_events.go) derives the phases from pod status timestamps delivered by the existing pod watch and safety poll:setup_worker_job_create: the Job create API call, timed directly.setup_worker_pod_schedule: pod creation to thePodScheduledcondition. This surfaces node capacity and autoscaler waits.setup_worker_sidecar_prep: the firstcopy-sidecar-*init container start to the last finish, including the kubelet's sidecar image pulls. Absent in image-volumes mode.setup_worker_setup_command: thesetupinit container run time, when a setup command is configured.setup_worker_container_start: the end of init (or scheduling) to thetaskcontainer start, including the kubelet's task image pull.is_error: true. The Kubernetes tracker reports each phase one time; repeated watch deliveries do not duplicate events, and pod-derived timestamps keep latencies accurate when a watch update arrives late.TaskParams. The field can be nil, and all methods are safe on a nil reporter.github.com/google/uuidto a direct dependency.Where the data lands
The server ingests each event as a
ClientSetupMetricPayloadand needs no changes. The events appear in the metricsambient_agents/client/eventsandambient_agents/client/event_latency_mswithtimeline_phase="oz_run_claimed", next to the existingsetup_*events that the in-container client reports for the later phases.Notes
Testing
go build ./...,go vet ./..., andgo test ./...pass.is_error, the wait for all sidecar init containers, and nil safety.Conversation: https://staging.warp.dev/conversation/bfa633f7-0c7d-45e4-bd78-f43a0f2865be
Run: https://oz.staging.warp.dev/runs/019ff6db-8344-7d14-93bd-f3b596c0aa85
This PR was generated with Oz.