[ARC/DinD] Deferred Node.js setup omits daemon-visible relocation

[romainh-betclic]

Summary

On ARC/DinD, gh-aw normally emits Ensure Node.js is at daemon-visible path after
automatic Node.js setup. This handles Node.js installations or cache hits outside
${RUNNER_TEMP}, which are not guaranteed to be visible to AWF through the DinD daemon.

When runtime setup is deferred until after a named custom checkout, the compiler inserts
Setup Node.js but does not insert the daemon-visible relocation step. This is independent
of the unnamed-checkout detector bug: the reproduction below uses a named checkout, and
Node.js setup is present.

This was re-verified on 2026-07-22 by exporting and building the exact then-current
origin/main commit 2868d6e080ff76e594e79cafbdfacd11bfa53d2b, then compiling the
reproduction below with that binary. The permalinks below pin
9df84ea4ecae5d9604499771694281dc4a9738f1; every referenced source file is identical
between those two commits.

Related issue: #47529

Impact

• Deferred and early Node.js setup receive different ARC/DinD safety handling.
• actions/setup-node may resolve Node.js outside the daemon-visible shared workspace.
• AWF can receive a Node.js path that is valid in the runner container but inaccessible
  through its chroot.
• Compilation succeeds even though the generated ARC bootstrap is incomplete.
• Using compiler-managed checkout masks the issue by selecting the early runtime path.

Environment

• Re-verified at upstream commit: 2868d6e080ff76e594e79cafbdfacd11bfa53d2b
• Engine: Copilot
• Runner: Linux self-hosted ARC with Docker-in-Docker
• Runner topology: arc-dind

Minimal reproduction

Create .github/workflows/repro-arc.md:

---
on:
  workflow_dispatch:

engine: copilot
runs-on: my-arc-runner
runner:
  topology: arc-dind

permissions:
  contents: read
network: defaults

steps:
  - name: Checkout repository
    uses: actions/checkout@v6
    with:
      fetch-depth: 0
      persist-credentials: false
  - name: Prepare context
    run: echo ready
---

# ARC/DinD reproduction

Call `noop`.

Compile it:

gh aw compile repro-arc --strict

Compilation succeeds with no errors (only a non-blocking staleness notice that
actions/checkout@v6 has a newer release).

Inspect the generated agent job:

rg -n "name: (Redirect tool cache|Checkout repository|Setup Node|Ensure Node|Prepare context)" \
  .github/workflows/repro-arc.lock.yml

Expected behavior

The deferred Node.js runtime path should include:

1. ARC/DinD tool-cache redirect
2. Named custom checkout
3. Setup Node.js
4. Ensure Node.js is at daemon-visible path
5. Remaining custom and framework steps
6. Agent execution

The ARC relocation invariant should not depend on whether Node.js setup is emitted early or
deferred after custom checkout.

Actual behavior

The inspect command above outputs at 2868d6e0:

442:      - name: Redirect tool cache and install paths for ARC/DinD
459:      - name: Checkout repository
464:      - name: Setup Node.js
469:      - name: Prepare context

It omits:

Ensure Node.js is at daemon-visible path

Compiler-managed checkout emits both Setup Node.js and the relocation step, so the two
supported checkout configurations produce different ARC safety behavior.

Agent analysis and root cause

1. The compiler documents why relocation is needed

The ARC/DinD prelude explains that setup-node may find a cached Node.js installation
outside ${RUNNER_TEMP} and that such a path is not visible to the DinD daemon:

gh-aw/pkg/workflow/compiler_yaml_runtime_setup.go

Lines 131 to 138 in 9df84ea

	// ARC/DinD: ensure Node.js is at a daemon-visible path.
	// On ARC runners, setup-node may find a pre-cached node at the original tool cache
	// (e.g. /home/runner/_work/_tool/node/...) which is NOT under RUNNER_TEMP and therefore
	// not bind-mounted into the AWF container. This step copies node to the redirected
	// tool cache if needed and sets GH_AW_NODE_BIN for the AWF entrypoint.
	// Only emit when runtime steps (including setup-node) were already emitted above;
	// when they are deferred to after a custom checkout, this step would run before
	// setup-node and could relocate an absent or wrong node binary.

2. Relocation is emitted only for early runtime setup

generateArcDindNodePathStep is called only when runtimeStepsEmittedEarly is true:

gh-aw/pkg/workflow/compiler_yaml_runtime_setup.go

Lines 118 to 140 in 9df84ea

	runtimeStepsEmittedEarly := needsCheckout || !customStepsContainCheckout
	if runtimeStepsEmittedEarly {
	// Case 1 or 3: Add runtime steps before custom steps
	// This ensures checkout -> runtime -> custom steps order
	compilerYamlLog.Printf("Adding %d runtime steps before custom steps (needsCheckout=%t, !customStepsContainCheckout=%t)", len(runtimeSetupSteps), needsCheckout, !customStepsContainCheckout)
	for _, step := range runtimeSetupSteps {
	for _, line := range step {
	yaml.WriteString(line)
	yaml.WriteByte('\n')
	}
	}
	}
	// ARC/DinD: ensure Node.js is at a daemon-visible path.
	// On ARC runners, setup-node may find a pre-cached node at the original tool cache
	// (e.g. /home/runner/_work/_tool/node/...) which is NOT under RUNNER_TEMP and therefore
	// not bind-mounted into the AWF container. This step copies node to the redirected
	// tool cache if needed and sets GH_AW_NODE_BIN for the AWF entrypoint.
	// Only emit when runtime steps (including setup-node) were already emitted above;
	// when they are deferred to after a custom checkout, this step would run before
	// setup-node and could relocate an absent or wrong node binary.
	if isArcDindTopology(data) && runtimeStepsEmittedEarly {
	c.generateArcDindNodePathStep(yaml)

The guard itself is deliberate and documented in the comment above it: emitting the
relocation step at this prelude position on the deferred path would run it before
setup-node and "could relocate an absent or wrong node binary". The bug is that no
equivalent relocation is emitted at the correct position — immediately after the deferred
Setup Node.js — so the deferred path ends with no relocation at all. A fix must not
simply remove the guard; it must emit the step after deferred Node.js setup.

3. The deferred insertion path has no equivalent relocation

With a named custom checkout, emitCustomSteps delegates runtime insertion to
addCustomStepsWithRuntimeInsertion:

gh-aw/pkg/workflow/compiler_yaml_runtime_setup.go

Lines 181 to 185 in 9df84ea

	if customStepsContainCheckout && len(runtimeSetupSteps) > 0 {
	// Custom steps contain checkout and we have runtime steps to insert
	// Insert runtime steps after the first checkout step
	compilerYamlLog.Printf("Calling addCustomStepsWithRuntimeInsertion: %d runtime steps to insert after checkout", len(runtimeSetupSteps))
	c.addCustomStepsWithRuntimeInsertion(yaml, customStepsToEmit, runtimeSetupSteps, data.ParsedTools)

That function inserts runtimeSetupSteps after checkout but has no access to, or call for,
the ARC/DinD Node.js relocation step:

gh-aw/pkg/workflow/compiler_yaml_runtime_setup.go

Lines 376 to 400 in 9df84ea

	if isCheckoutStep {
	// This is a checkout step, copy all its lines until the next step
	i++
	for i < len(lines) {
	nextLine := lines[i]
	nextTrimmed := strings.TrimSpace(nextLine)
	// Stop if we hit the next step
	if strings.HasPrefix(nextTrimmed, "- name:") || strings.HasPrefix(nextTrimmed, "- uses:") {
	break
	}
	// Add the line
	nextIsBS := blockScalarState.update(nextLine)
	appendYAMLLine(yaml, " ", nextLine, nextIsBS)
	i++
	}
	// Now insert runtime steps after the checkout step
	compilerYamlLog.Printf("Inserting %d runtime setup steps after checkout in custom steps", len(runtimeSetupSteps))
	for _, step := range runtimeSetupSteps {
	for _, stepLine := range step {
	yaml.WriteString(stepLine + "\n")
	}
	}

The relocation requirement therefore depends on which compiler branch emitted the same
Node.js setup action.

Current workaround

Use compiler-managed checkout:

checkout:
  fetch-depth: 0

This selects the early runtime path and emits:

Checkout repository
Redirect tool cache and install paths for ARC/DinD
Setup Node.js
Ensure Node.js is at daemon-visible path

Agentic implementation plan

Please implement the following changes.

1. Add a failing ARC/DinD regression test
   (pkg/workflow/checkout_runtime_order_test.go or a focused runtime setup test):

   • Configure runner.topology: arc-dind.
   • Use a named custom checkout so runtime insertion succeeds.
   • Confirm Setup Node.js is followed by
     Ensure Node.js is at daemon-visible path.
   • Confirm remaining custom steps retain their order.

2. Make ARC Node.js relocation reusable
   (pkg/workflow/compiler_yaml_runtime_setup.go):

   • Represent the relocation operation as a reusable generated step or helper.
   • Emit it immediately after generated Node.js setup in both early and deferred paths.
   • Do not emit it when Node.js is not among the generated runtime requirements.
   • Preserve the existing ARC tool-cache redirect ordering.

3. Avoid coupling ARC behavior to checkout ownership
   (pkg/workflow/compiler_yaml_runtime_setup.go):

   • Keep the relocation invariant attached to generated Node.js setup rather than
     runtimeStepsEmittedEarly.
   • Preserve compiler-managed checkout behavior.
   • Preserve non-ARC behavior without adding relocation steps.

4. Cover edge cases

   • Named custom checkout with with: options.
   • Multiple detected runtimes, with relocation emitted only once after Node.js setup.
   • Non-ARC custom runner receives Node.js setup without ARC relocation.
   • Compiler-managed checkout continues to emit the existing ARC bootstrap.
   • Customized actions/setup-node retains current deduplication semantics.

5. Run project validation

   • Run focused tests for checkout/runtime ordering.
   • Run make recompile and inspect generated ARC workflow ordering.
   • Run make agent-finish before completion.

No new dependency or public configuration field should be required.

Acceptance criteria

• ARC/DinD Node.js relocation follows generated Node.js setup in both early and deferred
  runtime paths.
• Named custom checkout and compiler-managed checkout receive equivalent ARC Node.js safety
  handling.
• Relocation is emitted exactly once when generated Node.js setup is present.
• Non-ARC workflows do not receive the ARC relocation step.
• Existing custom-step ordering and runtime deduplication remain unchanged.
• Strict compilation, unit tests, and workflow recompilation pass.

Duplicate research

Searched open and closed issues and pull requests on 2026-07-22 for ARC/DinD,
daemon-visible Node.js, GH_AW_NODE_BIN, deferred runtime setup, and
generateArcDindNodePathStep. No existing issue covers relocation being skipped only on
the deferred custom-checkout path.

Closest related items, all closed and addressing different failure modes:

• gh-aw compiler: ARC/DinD topology requires daemon-visible path redirections #42807 introduced the ARC/DinD tool-cache redirect.
• [awf] Node.js not found in chroot when RUNNER_TOOL_CACHE is /home/runner/work/_tool #35801 covered Node.js outside the daemon-visible path and motivated the existing
  relocation step.
• [aw-failures] Node.js v24.15.0 binary not found in agent container (GH_AW_NODE_BIN path broken) #26876, [aw-failures] [aw] Node.js unreachable inside AWF chroot — Daily News failed with exit 127 (run 27268191506) #38379, and [aw-failures] [aw] Copilot CLI exits 127 — node missing inside AWF chroot (chronic Daily News failure on main) #40074 are runtime Node.js failure reports in this repository's own
  workflows, not deferred-path compiler bugs.