ci: run Frontier CI on the g1 partition, and raise the queue-wait budget to 4h

[sbryngelson]

Stacked on #1763 — base is ci/phoenix-queue-timeout-combined-alloc. Merge that one first; this PR's second commit retunes the budget it introduces. The net diff over #1763 is two files.

Two related fixes for Frontier CI reliability.

1. Run Frontier CI on the g1 partition

Frontier CI jobs are submitted to batch, where they queue behind the rest of the machine. A full matrix routinely sits pending for a long stretch before any node is handed over.

g1 is a dedicated 64-node carve-out (frontier10177-10240). Those nodes belong to g1 only and are not part of batch, so CI starts promptly instead of competing for the general pool. They are otherwise ordinary Frontier hardware — Gres=gpu:8, 112 allocatable cores — so both the GPU (acc/omp) and CPU matrices run unchanged.

This points the CPU and GPU device blocks at g1 for frontier and frontier_amd. Nothing else about the job header changes: --stepmgr, --qos=normal, -N 1, and the 01:59:00 walltime all carry over, comfortably inside g1's 2-day cap.

The QOS deliberately stays normal

g1 carries a partition QOS that is also named g1. slurmctld applies it on its own once a job lands in the partition — requesting it explicitly fails, because CFD154 holds no association with it:

$ sbatch --test-only -p g1 -A CFD154 --qos=g1 -N1 -t 01:59:00 --wrap=hostname
allocation failure: Invalid qos specification

A comment in the script records this, so the next reader doesn't "fix" the partition change by adding --qos=g1.

2. Raise the queue-wait budget to 4h for all clusters

#1763's SLURM_MAX_QUEUE_SECONDS:=5400 was sized against Phoenix's preemptible embers QOS, where the failure it guards against is a job that sits PENDING indefinitely. As a global default it also lands on Frontier, whose batch waits are much longer and fatter — so routine queue pressure surfaces as red CI. That is what this bench-gpu-omp failure is: job 5351071 sat in batch for the full 90 minutes and was cancelled, Elapsed=00:00:00.

One week of MFC CI jobs on Frontier batch (n=1230 that eventually started):

p50	p75	p90	p95	max
1m	22m	96m	176m	466m

A 90m budget sits at roughly p89 and tripped 11% of those jobs — and that undercounts, since a further 162 jobs over three days never started at all.

4h clears p95 with margin, stays well inside the job-level timeout-minutes: 480 that remains the real backstop, and keeps the bound finite so a wedged embers job still can't hold a runner slot forever. SLURM_MAX_QUEUE_SECONDS is still overridable per workflow, and 0 still means wait indefinitely.

This matters even with g1 in place: the g1 QOS caps a user at 16 running nodes (MaxTRESPU=node=16) against a measured peak of 25 concurrent CI jobs, so a full matrix runs in about two waves and wave-2 bench jobs can legitimately wait behind a 01:59:00 allocation.

Verification

I stubbed sbatch on PATH to capture the header the script actually generates, then ran each captured header through sbatch --test-only against the live controller. All eight combinations — frontier/frontier_amd × cpu/gpu × test/build-and-test (the new job type from #1763) — are accepted and schedule immediately onto g1:

#SBATCH -J MFC-test-gpu-none-1-of-2
#SBATCH --account=CFD154
#SBATCH -N 1
#SBATCH -n 8
#SBATCH -p g1
#SBATCH -t 01:59:00
#SBATCH --qos=normal
#SBATCH --stepmgr

sbatch: Job 5351955 to start at 20:06:34 using 56 processors on nodes frontier10193 in partition g1

CPU headers are identical apart from -n 32. Phoenix is untouched — its dynamic select-gpu-partition.sh path is unaffected, and it keeps the same budget semantics, just with a longer default.

https://claude.ai/code/session_01RPGyAS6S7BRDELsANWaBpR

[Copilot]

Pull request overview

This PR updates the Frontier CI SLURM submission header generation to target the dedicated g1 partition (instead of batch) so CI jobs start sooner, and documents the partition QoS behavior to prevent incorrect --qos=g1 additions.

Changes:

• Switch Frontier/Frontier-AMD CPU and GPU job submissions from #SBATCH -p batch to #SBATCH -p g1.
• Expand inline documentation explaining why --qos=normal must remain and why --qos=g1 should not be requested.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.67%. Comparing base (3e3651d) to head (15177d9).

Additional details and impacted files

@@                           Coverage Diff                            @@
##           ci/phoenix-queue-timeout-combined-alloc    #1767   +/-   ##
========================================================================
  Coverage                                    61.67%   61.67%           
========================================================================
  Files                                           84       84           
  Lines                                        21619    21619           
  Branches                                      3196     3196           
========================================================================
  Hits                                         13334    13334           
  Misses                                        6093     6093           
  Partials                                      2192     2192           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.