CI: skip the redundant rebuild in the case-opt run job

[sbryngelson]

Description

The Case Opt | Oak Ridge | Frontier (AMD) (gpu-omp) job fails intermittently during cmake --install:

   Installing syscheck...
   ✗ Install failed for syscheck

 Install Failed - Error Details:
 CMake Error at build/staging/gpu-mp-4e60642e2d/cmake_install.cmake:52 (file):
   file INSTALL cannot copy file
   ".../build/staging/gpu-mp-4e60642e2d/syscheck"
   to ".../build/install/gpu-mp-4e60642e2d/bin/syscheck":
   No such file or directory.

Error: Failed to install the syscheck target.
mfc: ERROR > main.py finished with a 143 exit code.
FAIL: viscous_weno5_sgb_acoustic (build or run error)

The exit code 143 is not an external kill. toolchain/main.py calls quit(signal.SIGTERM) on MFCException, so any build or install failure surfaces as 143.

Cause

On frontier_amd the run step submits three concurrent SLURM jobs that share one workspace. run_case_optimization.sh skips its own build block for that cluster, because prebuild-case-optimization.sh already built and installed every binary in a prior SLURM job. It then calls ./mfc.sh run anyway, and run always builds: run() calls build(targets) unconditionally, and __build_target runs configure, build, and install for every non-dependency target on every case.

syscheck, pre_process and post_process hash to a single slug for all five benchmarks, because get_slug hashes case.get_fpp(target) and get_fpp maps only pre_process and simulation to real generators. Everything else gets "! This file is purposefully empty.". The per-shard logs confirm it:

run-...-1-of-3.out  syscheck: gpu-mp-4e60642e2d  pre_process: gpu-mp-0e981924c0  post_process: gpu-mp-01766823fa
run-...-2-of-3.out  syscheck: gpu-mp-4e60642e2d  pre_process: gpu-mp-0e981924c0  post_process: gpu-mp-01766823fa
run-...-3-of-3.out  syscheck: gpu-mp-4e60642e2d  pre_process: gpu-mp-0e981924c0  post_process: gpu-mp-01766823fa

Only simulation gets a per-case slug. So all three shards run cmake --build and cmake --install against the same build/staging and build/install paths, with no synchronization. The pre-build already guards against exactly this, with a marker handshake and the comment "Concurrent shards must not build those shared staging dirs simultaneously". The run phase never got the same treatment.

The collision window is at shard startup. All three jobs left the queue in the same second after roughly 90 minutes pending:

06:34:38.299  === Streaming output for job 5341449 ===
06:34:38.299  === Streaming output for job 5341447 ===
06:34:38.299  === Streaming output for job 5341448 ===

They then walked through an identical startup and reached the shared syscheck install together. That is why the failure lands on a shard's first case, and why the losing shard differs between attempts of the same run: attempt 1 failed on hypo_hll (shard 3), attempt 2 on viscous_weno5_sgb_acoustic (shard 2).

Fix

Pass --no-build to mfc.sh run on the clusters whose binaries were built in a prior SLURM job, which is phoenix and frontier_amd, the same two the in-script build block already skips. --no-build makes is_buildable() return False, so __build_target returns before configure, build and install. The run phase then issues no cmake invocations at all and there is nothing left to race on. run() still loads and validates the case and still calls __generate_input_files, so only the build side is affected.

frontier (CCE) is unchanged: it builds on the compute node inside this script, so it keeps building through run.

Testing

• The pre-built binaries are the ones the run resolves, so skipping the build does not change what executes. Every shard finished its cases within 11 minutes of leaving the queue, against roughly 30 minutes per case for an AMD flang case-opt build, so no shard was building anything. --gbpp and --steps do not enter get_slug.
• bash -n passes. shellcheck reports no new findings; the remaining SC2086 and SC2154 warnings are pre-existing and unrelated.
• Real verification is the Case Opt matrix on this PR, on both Phoenix and Frontier AMD.

Known remaining exposure

The pre-build has the same structural hazard and this PR does not address it. Shards 2 and 3 wait on build/.prebuild-shared-targets-done, then each runs ./mfc.sh run "$case" --dry-run per case, which reinstalls the shared targets into the same paths. It has not been observed to fail, most likely because the marker is polled on a 30 second sleep, so the shards are staggered rather than released together. Worth a follow-up, but it is a different change and should not ride along with this one.

Type of change (delete unused ones)

• Bug fix

Reference

Failing job: https://github.com/MFlowCode/MFC/actions/runs/32684872325/job/97532516750

[Copilot]

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR prevents intermittent failures in the Case Optimization CI job on shared-workspace SLURM clusters by ensuring the run step does not rebuild/reinstall binaries that were already built in a prior job.

Changes:

• Introduces a build_opts flag that becomes --no-build on phoenix and frontier_amd.
• Passes build_opts through to ./mfc.sh run so concurrent shards skip cmake --build/cmake --install.
• Adds in-script documentation explaining the shared-slug collision and resulting install race.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.67%. Comparing base (41df185) to head (dcb50a6).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1766   +/-   ##
=======================================
  Coverage   61.67%   61.67%           
=======================================
  Files          84       84           
  Lines       21619    21619           
  Branches     3196     3196           
=======================================
  Hits        13334    13334           
  Misses       6093     6093           
  Partials     2192     2192           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.