famd: move Frontier AMD builds to AFAR drop 23.2.1

[sbryngelson]

Moves the famd (Frontier AMD / amdflang) toolchain from the therock-23.1.0 AFAR drop to therock-afar-23.2.1-gfx90a-7.13.0-7357b5084b, the newest drop available on Frontier (module afar/23.2.1-7.13.0-7357b5084b, staged 2026-08-25).

What changed

toolchain/bootstrap/modules.sh — the 23.2.x drop layout differs from 23.1.0:

• include/mpich3.4a2 is gone; only mpich4.3.1 ships, which is the right one for cray-mpich 9.x anyway.
• hipfort moved from include/hipfort + lib/ to lib/llvm/include/hipfort + lib/llvm/lib.
• $OLCF_AFAR_ROOT is now overridable, matching the amdfund block below it, so testing the next drop doesn't need a source edit.

cmake/MFCTargets.cmake — resolve HIP and hipfort out of the drop explicitly. The loaded ROCm module also ships libhipfort-amdgcn.a, and find_library searches CMAKE_PREFIX_PATH before HINTS, so HIPFORT_AMDGCN_LIB was silently resolving to /opt/rocm-7.2.0/lib/libhipfort-amdgcn.a — pairing the drop's flang .mod files with an archive built by a different flang. It linked, but only because the -L$OLCF_AFAR_ROOT/... flags happened to come first.

What deliberately did not change

toolchain/modules stays on amd/7.2.0 rocm/7.2.0. /opt/rocm-7.13.0 (with modules amd/7.13.0 / rocm/7.13.0) is new on Frontier and ships amdflang 23, but it cannot be used here:

• It and the AFAR drop both ship libclang-cpp.so.23.0git and libLLVM.so.23.0git. famd puts the drop's lib/llvm/lib first on LD_LIBRARY_PATH, so the system cc loads the drop's LLVM and dies with malloc(): invalid size (unsorted) compiling a trivial .c, which fails the hipfort dependency configure. amd/7.2.0 has LLVM 22 sonames, so no collision.
• It ships no hipfort.
• Cray's amd-built mpi.mod (both 9.0.1/ofi/amd/6.0 and 9.1.0/ofi/amd/7.0) is rejected by flang 23 with 'mpi.mod' is not a module file for this compiler. Only the drop's own include/mpich4.3.1 works, so the drop has to stay the base.

Also worth noting: the /opt/rocm-7.0.2 entry on LD_LIBRARY_PATH is load-bearing, not stale. cray-mpich's libmpifort_amd.so is built with classic flang and needs libpgmath/libflang/libflangrti/libompstub, which no AFAR drop ships and which ROCm dropped after 7.0.2. Retargeting it at $ROCM_PATH produces libpgmath.so: cannot open shared object file at run time. Added a comment so it doesn't get cleaned up again.

Verification

Frontier, 1 node, gfx90a, --gpu mp, MFC master b2a01f9:

• syscheck PASSED — MPI init/barrier/finalize, omp_get_num_devices() > 0, device selection.
• Test suite: 667/686 passed. All 19 failures are chemistry cases failing on a missing gpu-mp-chem-* binary, which this run never built (--no-build; CI builds that variant in a separate job) — not compiler failures.
• simulation device-LTO link: 1682 s on 23.1.0 → 780–951 s on 23.2.1. Both builds shared a login node so the numbers are approximate, but the ~2x is well outside the noise. This is the cost that drives the concurrent base/chem job split in .github/workflows/common/build.sh; that split is left alone here, but there may be room to simplify it in a follow-up.

[Copilot]

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Moves the Frontier AMD (famd / amdflang) toolchain configuration to the newer AFAR 23.2.1 drop and tightens CMake library resolution to ensure HIP/hipfort are taken from the drop (avoiding ROCm module archive mismatches).

Changes:

• Update famd AFAR root to therock-afar-23.2.1... and adjust mpich/hipfort include+lib paths to the new drop layout.
• Allow $OLCF_AFAR_ROOT override for easier drop testing, and document the pinned ROCm 7.0.2 runtime requirement.
• Change CMake find_library() usage to prefer AFAR-drop HIP and hipfort archives over installed ROCm module libraries.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
toolchain/bootstrap/modules.sh	Switches famd to AFAR 23.2.1, updates include/lib locations, and documents pinned runtime libs.
cmake/MFCTargets.cmake	Adjusts library discovery so HIP/hipfort resolve from AFAR drop before falling back elsewhere.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.67%. Comparing base (b2a01f9) to head (e1a404a).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1768   +/-   ##
=======================================
  Coverage   61.67%   61.67%           
=======================================
  Files          84       84           
  Lines       21619    21619           
  Branches     3196     3196           
=======================================
  Hits        13334    13334           
  Misses       6093     6093           
  Partials     2192     2192           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.