Load-balance, active-box windowing, block-structured AMR - #1628
Open
sbryngelson wants to merge 606 commits into
Open
Load-balance, active-box windowing, block-structured AMR#1628sbryngelson wants to merge 606 commits into
sbryngelson wants to merge 606 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in (“default-off”) family of performance/diagnostic features (load-weight and SFC partition diagnostics, weighted init-time decomposition, rank timing), plus major simulation capabilities (active-box RHS windowing and block-structured AMR) and corresponding post-processing support and documentation/validation updates.
Changes:
- Adds new runtime parameters and toolchain metadata/validation hooks for the experimental performance/AMR feature family.
- Extends the simulation code with new modules for active-box restriction, load-weight diagnostics, SFC partition reporting, rank timing, and AMR integration points (including restart/output plumbing).
- Updates post_process to read/write AMR fine-block overlays and adds/updates golden metadata plus documentation/indexing.
Reviewed changes
Copilot reviewed 82 out of 94 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/params/descriptions.py | Adds user-facing descriptions for new experimental/performance parameters. |
| toolchain/mfc/params/definitions.py | Registers new parameters (AMR, hybrid sensors, load-balance diagnostics) and target applicability. |
| toolchain/mfc/lint_docs.py | Treats new validator checks as non-physics doc checks. |
| tests/F980C769/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ECABA006/golden-metadata.txt | Adds golden metadata for active-box test coverage. |
| tests/DD4CD8F3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/CC4213FD/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BD21A5C0/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BCBA6E74/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ACE05393/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/987D9025/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/852CCB81/golden-metadata.txt | Adds golden metadata for AMR-related golden tests. |
| tests/65C375B4/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/4DADE04B/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/454C565F/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/3A474BEE/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/2FC423D3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/13945217/golden-metadata.txt | Adds golden metadata for moving-IB under AMR test coverage. |
| src/simulation/m_viscous.fpp | Clamps FD coefficient indexing to avoid ghost-region coefficient OOB in IB drag gradient evaluation. |
| src/simulation/m_time_steppers.fpp | Integrates active-box bounds into RK update loops and interleaves AMR fine-stage/subcycle operations. |
| src/simulation/m_start_up.fpp | Wires up new modules (rank timing, active-box, load-weight, SFC partition, AMR) into init/timestep/finalize and restart I/O. |
| src/simulation/m_sfc_partition.fpp | Adds analysis-only SFC tiling + weighted partition prediction and reporting. |
| src/simulation/m_rank_timing.fpp | Adds per-rank wall-time imbalance measurement helpers and reporting. |
| src/simulation/m_load_weight.fpp | Adds per-cell load-weight field construction and rank-level imbalance reporting. |
| src/simulation/m_hypoelastic.fpp | Refactors FD coefficient setup into a callable update routine (supporting AMR grid swaps). |
| src/simulation/m_global_parameters.fpp | Adds AMR working-state mirrors and slot selection helper plus defaults for new parameters. |
| src/simulation/m_data_output.fpp | Adds output/report hooks for load-weight, SFC partition, and rank-time diagnostics. |
| src/simulation/m_checker.fpp | Adds input validation/prohibits for active-box, hybrid sensors, load-balance, and AMR configurations. |
| src/simulation/m_active_box.fpp | Adds active-box initialization/growth and debug envelope checking. |
| src/simulation/m_acoustic_src.fpp | Adds AMR-aware handling of acoustic source support (bounding boxes and overlap abort). |
| src/post_process/m_start_up.fpp | Calls AMR fine-data reader and AMR overlay writer when amr is enabled. |
| src/post_process/m_global_parameters.fpp | Adds default-off amr flag for post_process overlay behavior. |
| src/post_process/m_data_output.fpp | Implements AMR fine-block overlay mesh/variables output (Silo/binary) and multimesh registration. |
| src/common/m_phase_change.fpp | Exposes per-cell Newton iteration count and threads it through relaxation to support load-weighting. |
| src/common/m_global_parameters_common.fpp | Adjusts start_idx lifecycle/allocation and makes load_weight_wrt visible to GPU macros. |
| src/common/m_derived_types.fpp | Introduces a simple t_box type used by new partitioning infrastructure. |
| src/common/m_box.fpp | Adds box/partition arithmetic helpers (equal/weighted splits, box-from-splits). |
| src/common/m_boundary_common.fpp | Skips BC buffer population during AMR fine advance to rely on coarse-driven ghost fill. |
| docs/module_categories.json | Registers new modules under documentation categories. |
| docs/documentation/readme.md | Adds AMR section link to the documentation index. |
| .typos.toml | Adds project-specific abbreviations to the spelling allowlist. |
| D = ((gs_min(lp) - 1.0_wp)*cvs(lp))/((gs_min(vp) - 1.0_wp)*cvs(vp)) | ||
|
|
||
| #ifdef MFC_SIMULATION | ||
| if (relax .and. load_weight_wrt) then |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1628 +/- ##
==========================================
+ Coverage 60.77% 61.98% +1.21%
==========================================
Files 83 93 +10
Lines 20872 25637 +4765
Branches 3101 4206 +1105
==========================================
+ Hits 12685 15892 +3207
- Misses 6121 6993 +872
- Partials 2066 2752 +686 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
Upstream latent gap found during the MHD+AMR investigation (independent of this PR): Also for the record: MHD+AMR was attempted and re-gated on measured evidence rather than assumption — the coarse/fine seam is a continuous O(1) div(B) source that cleaning spreads but cannot remove (details in the amr.md support matrix row and commit ac203b1). |
sbryngelson
changed the title
Terse pass over the PR-ADDED comments in m_ibm, m_checker, m_time_steppers, and m_rhs (pre-existing files - only the PR's own AMR / fine-IB / swap / gate comments were edited; verified no pre-existing comment or PROHIBIT message string was modified). Cut filler and run-ons while preserving every gate rationale, compiler-quirk note (CCE lib-4425 module separation, declare-target/present-table, ab_int device-copy, AMD defaultmap/ROCm), TWIN marker, and trap warning. Comment-only; build + 8 AMR/IB/IGR/subcycle/QBMM goldens bit-identical.
The Example suite caps grids at 25x25 (modify_example_case), but an AMR block must sit >= buff_size cells inside the domain and span at most half of it - there is no room for a valid block at 25x25, and the example's block indices (sized to its own 128-cell grid) fall outside the capped grid, so the CI Example run failed case validation ('amr_block_end must be <= global cell max per axis'). Reproduced locally. The example stays valid/runnable at native resolution and AMR is covered directly by amr_golden_tests; this only removes it from the downsized Example smoke suite (same reason as the existing 2D_reacting_mixing_layer skip).
Member
Author
|
…forced cross-rank remap (l0_migrate_step)
… bit-identical (l0_rebalance_interval)
… (gather at output), all paths bit-identical
…ter, bit-identical on nvfortran OpenACC (2xA100)
…ming, allreduced), reproducible on CPU+GPU
…-identical 2D+3D np=1/2
…le wrap-seam), bit-identical 2D+3D
…_bc -> consistent wrap-seam list), fixes cross-rank deadlock, bit-identical
…windows, halves noise-driven migration churn (bit-identical)
…, lint fix), bit-identity re-verified
…), spike bit-identical 2D+3D [Option 2 foundation]
…p) + l0_slot_off (tile prefix), behavior-neutral [spike-only + AMR np1==np2 byte-identical]
…runs the device pack/unpack P2P path that CI must exercise on Cray/AMD-flang (no other test hits it)
…vf intent(in)->intent(inout) (passed to bidirectional s_l0_copy_block inout dummy; nvfortran allowed it, GNU/Intel reject)
…th (T1/I4b-a) mg:slot measured 57.9 s mean / 90.2 s max at np=8 (5.5% of wall): every 8-16 incremental replica-slot allocs re-staged the whole store through the host. One s_amr_prereserve_stash call per wave grows the store at most once, and s_amr_st_reserve now zeroes only the NEW columns (one full-store host pass saved per growth event). No new device code; store contents byte-identical. Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
…ehind cadence Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
…ttach stash-alloc doc Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
XA_NH header words ([site, blk, bl, bh]) ride ahead of every F1/F2/F3 payload under MFC_DEBUG and are verified before unpacking; zero-width in production. Device kernels untouched (offset via argument slices). Gates: 75/75 AMR goldens (production); debug np=8 probe clean with headers live on all 858 F1 + 1646 F2 messages; a seeded consume-order bug aborted with the full expected-vs-got diagnostic. Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
s_amr_stage_fill_wave: F1+F3 per-stage gathers as one aggregated message per (peer, family) per RK stage - all recvs posted, device packs into pool slices via the existing kernels, one WAITALL, box-major consume through the single amr_cg. Replaces the per-box owner-WAITALL / contributor-flush / F3-blocking-SEND chain. Level>=2 keeps the per-box F2 path (I3); subcycle keeps its sites (I8). Gates: [amr-xa] F1 payload words exact vs baseline (msgs 858->381), F2/F4-F7 byte-identical; live identity headers on every wave transfer (F1 np=8, F3 np=2 with real traffic) + per-message length asserts; seeded offset-shift arm aborts at the header check; adversarial review (grow-helper data-loss bug fixed pre-gate). Outstanding at commit time: full-suite goldens job 383666 (baseline worktree). Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
s_amr_parent_fill_wave(lev): the per-step level>=2 parent gather - previously a pooled ISEND + blocking MPI_RECV per box per stage on the majority of boxes - as one aggregated message per (parent-owner, child-owner) pair per level, levels ascending. Pair lists derive on all ranks from replicated metadata only; reuses the I2a wave's scratch; zero new device kernels. s_amr_fine_stage_fill lost its last caller and is deleted. Regrid chunked, subcycle, and init/static F2 paths unchanged. Also fixes restart leaving amr_num_levels at 1 until the first regrid (the per-level driver needs it truthful; the old per-box loop was immune). Gates: [amr-xa] F2 payload words exact vs baseline, msgs 1646->524, F1 and F4-F7 byte-identical; live identity headers + per-message length asserts on the F2 wave; seeded offset-shift arm aborts at the header check; adversarial review clean (10/10 invariants); local AMR goldens 75/75 incl. multi-level restart np=2 and ppn=4 dynamic regrid. Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
The cross-rank branch of s_amr_fine_fine_halo (one blocking MPI_SENDRECV per pair through two shared seam buffers) becomes one aggregated message per (peer, direction) per call: both owners walk the same replicated pair list ascending, so per-peer offsets agree with no metadata exchange. Same-rank pairs keep the batched device kernel. The shared seam buffers and their tile-grow reconciliation are deleted. Gates: [amr-xa] F6 payload words exact vs baseline (380,849,184), msgs 2646->378; all other families byte-identical; debug identity headers + length asserts clean; seeded header-shift arm aborts; local AMR-75 goldens 75/75.
The last per-box rendezvous chains: the level-1 reflux-face exchange (s_amr_p2p_reflux_faces per box) becomes s_amr_reflux_faces_wave — receives post zero-copy into the freg register host mirrors, owner D2H + multicast ISENDs, one WAITALL, receivers push H2D — and the level>=2 split-ownership freg handoff becomes s_amr_freg_wave. s_amr_reflux_to_parent gains do_xchg so the subcycle path keeps its per-box exchange. Message count is unchanged by design (zero-copy into per-box register slots); the wave removes the O(boxes) rendezvous chain. s_amr_reg_reserve hoists ahead of both waves since the apply can reallocate the registers. Gates: [amr-xa] F5 payload words exact (659,423,232), msgs 6708 unchanged; all other families byte-identical; debug-only companion identity headers + length asserts clean; seeded blk-shift arm aborts at the companion check; local AMR-75 goldens 75/75.
…1 blocks The per-box s_amr_apply_reflux launched up to 3 tiny face kernels per block per step; the launch overhead, not the arithmetic, dominated the reflux-apply phase (10.4 pct of the np8 step post-wave). A host precompute now walks the level-1 slots with the same select_slot + face-flags logic and fills per-slot descriptor arrays; one kernel per face direction corrects the coarse rhs for every block, mirroring the capture-side batching. The per-box form is deleted (single call site); the subcycle path is untouched. Block corrections are disjoint (merge invariant) and a block's x/y/z outside layers are distinct cells, so per-(face, eq, cell) arithmetic and child-sum order are identical. Gates: step-5 full-state output byte-identical to the pre-batch binary on the np8 probe; amr-xa byte-identical; reflux calls/rank 759 to 15; local AMR-75 goldens 75/75.
…npack, overlap carry-forward) The migration half of the regrid budget was host-staged end to end: full-slot device-host round trips per owned old block at stash creation, serial host cast loops for the MPI pack and unpack, a full-slot push per received replica, and a host overlap carry-forward. All four now run where the store is authoritative: a device cons-to-stor stash kernel, device pack/unpack kernels whose copyin/copyout stage exactly the packed interior (wire layout byte-identical, message set and amr-xa F4 totals unchanged), and a device overlap kernel with the per-box prolong push hoisted ahead of it (same final device state). The stash no longer touches the host mirror, retiring the two grow-hazard push sites. Gates: step-5 output byte-identical to the previous binary on the np8 probe; amr-xa byte-identical; probe rg:mig -41 pct, unpack 74 to 11 ms/call; AMR-75 75/75. Also documents a new compiler trap: amdflang silently drops target regions nested in Fortran block constructs from the device image (INVALID_SYMBOL_NAME at first launch) - kernels must live in ordinary subroutines.
…shes s_prolong_one_var and the alphas/species closure prolongs were host loops writing the cons host mirror, forcing a full-slot H2D push per built box at three call sites (rebuild, startup populate, persistent-L2 build). All three are now GPU kernels: the gathered patch's device mirror is pushed once per dispatch (patch-sized, 8x smaller than the slot pushes it replaces), the slot is built in place, and every full-slot push is deleted. minmod was already GPU_ROUTINE-decorated; the shared alpha limiter switch is inlined into its kernel and the helper deleted. With the device-side migration this completes the rebuild's device-resident data path. CPU builds compile the kernels to the identical plain loops (CPU results unchanged); GPU prolong arithmetic moves device-side and gates on golden tolerance. Gates: np8 probe rc=0 with amr-xa byte-exact; AMR-75 75/75.
s_amr_st_reserve preserved live blocks through a growth by pulling the entire store device to host, reallocating, and pushing it back - for each of up to four arrays per event. The round trip's one remaining purpose (carrying the migration stash's host writes across a growth) died with the device-side migration, so growth now stages through a device-mapped temporary: two on-device copies, zero PCIe. The host mirror comes out of a growth undefined, within the existing contract (every host reader pulls its slot first; compaction already leaves the host stale by design). The 5-step np8 probe fell 100.8 to 43.4 s (-57 pct) with byte-identical output - the growth round trips were a large untimed cost in every short run; the operating-point yield is smaller (high-water reached early at int=20). Gates: step-5 output byte-identical; amr-xa unchanged; AMR-75 75/75. Follow-up: the registers REG_GROW macro keeps the same pattern.
Probe verdict recorded in the ledger: the plan walks cost 0.01 ms/call, refuting the plan-caching increment (I6) before it was built; the wave's residue is pack copyouts and the WAITALL. Next comm target re-ranked to ring-clip-on-waves.
…t, restart pad zeroing Adversarial review of the day's four increments returned three findings, all fixed here. (1) The batched reflux apply relies on block corrections being disjoint, but the IB path re-broke the invariant: s_amr_expand_box_over_bodies runs after clustering and the follow-up merge fused only overlapping pairs, so two boxes left with a 1-cell gap had coincident outside coarse cells - an unsynchronized read-modify-write inside one kernel. The IB merge now fuses pairs closer than a 2-cell gap. (2) The device-native store grow transiently held old plus staging = 2x the array on device at exactly the memory high-water mark, a measured OOM class; device staging now applies only up to 32 old columns (where the short-run win concentrates) and larger grows keep the OOM-safe host round trip. (3) Restart pushes full padded columns after writing only interiors, and the host pad bytes are undefined since the device-native grow; the host column is now zeroed (owner-only) before each restart read. Gates: bit-identity vs the pre-fix binary on the np8 probe; AMR-75 75/75 with the IB and restart-roundtrip cases green.
The reverted clip (proven correct; killed by the amdflang whole-image codegen bug, since root-caused with a verified link-flag workaround) is reimplemented on the stage-fill wave: after each pair's box intersection, the slab is clipped against the patch's hollow shell (the open core is provably dead - amr_stepfill_ring_clip.md), yielding up to six sub-slab transfers derived identically on both sides from replicated metadata. The primitives (shell slab decomposition, clip, the debug NaN-poison arm, the shell-only own-box copy) are lifted verbatim from the reverted implementation; pack, unpack, and consume were already generic over slab bounds, so sub-slabs are just more transfers. The pbmv gather keeps its full-box wire contract (qbmm plus non-polytropic runs stay unclipped, as the original deliberately did). Messages stay at the per-peer count; only payload drops. Gates: F1 payload words 1,071,084,168 to 416,141,172 (-61 pct) with message count unchanged and every other family byte-identical; step-5 output byte-identical on the np8 probe; the debug NaN-poison arm runs clean (any consumer read of an unshipped cell aborts); seeded header arm aborts on a shell sub-slab; AMR-75 75/75; wall not in the slow codegen class.
…ror of MFlowCode#1759): the overnight np8 pairs showed the device-migration kernels crossed the AAPointerInfo cap and recompiled the whole module to slow ISA (rhs +70 pct on untouched phases); pi16k-rebuilt HEAD is byte-identical on the 5-step probe
…adopted in-tree; AMReX inter-node bar 1.192x
… (twin of the store grow): the old REG_GROW pulled all 12 creg/freg arrays over PCIe on every doubling; bitcmp byte-identical, AMR-75 75/75
… byte-identical; one shared shell derivation for both wire sides; poison-gated, AMR-75 75/75
…e L0-RHS halo are the new top targets; physics weak-scales at 1.09x
…es are weight-0 in the apply and now never ship (shared s_amr_sibling_face_weights on both wire sides, debug NaN arm); rs:wave/rs:rest/rs:rfp split the top inter-node phase; byte-identical, poison rc=0, AMR-75 75/75
…actly its apply set via the shared s_amr_reflux_faces_for derivation on both wire sides; PCIe pulls cover only the union of shipped faces; debug NaN arm on unshipped faces; F5 wire -49.5 pct (day total -58 pct), byte-identical, poison rc=0, AMR-75 75/75
…ISEND+WAITALL/RECV chain scaled with global block count; np8 5-step probe: F7 words exact, msgs 650->117; bitcmp byte-identical; poison rc=0; AMR-75 75/75
…Phase-2 batched advance): checker+validator gates, retire the 7 AMR bubble golden tests; the mpp_lim pre-conversion rescale and pb/mv side-state forced per-block special cases; qbmm requires bubbles_euler so all AMR QBMM paths are gated too; pb/mv AMR machinery deletion deferred to the deletion list
Lines of Code
|
…np16); residual growth is diffuse wait
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An opt-in, default-off family of performance features and the measurement infrastructure they rest on. With all flags at their defaults the only touched production path is
s_mpi_decompose_computational_domain, refactored through the newm_boxmodule (byte-identical; covered by the existing suite).m_box(partition arithmetic),m_load_weight/load_weight_wrt(per-cell load-weight field + imbalance metric),m_sfc_partition/sfc_partition_wrt(Morton-SFC predicted-imbalance diagnostic),m_load_balance/load_balance(weighted static decomposition at init; AMR-fine-work-aware),m_rank_timing/rank_time_wrt(per-rank compute-time diagnostic).m_active_box/active_box: restricts reconstruction/Riemann/RK windows to a light-cone-grown box around non-ambient flow; strict-subset golden-tested.hybrid_wenoandhybrid_riemann(+hybrid_weno_eps,hybrid_smooth_flux): linear-optimal weights / central-or-Rusanov flux in smooth cells, full WENO/HLLC at flagged discontinuities (Jameson sensor, stencil-dilated, per-level under AMR).m_amr+m_amr_registers: two-level 2:1 refined block hierarchy; conservative restriction and conservative-linear prolongation with physics-specific closures; per-stage flux registers with Berger–Colella refluxing; Berger–Rigoutsos multi-block dynamic regrid; optional dt/2 subcycling; multi-rank (single-owner blocks assigned by Morton-SFC work balancing at each regrid, with migration; blocks may span rank seams via P2P coarse↔fine gather/scatter; same-level seam halo; distributed registers); restart (both IO modes, regridded-layout persistence); AMR-aware post-processing (fine blocks visualizable as Silo overlay domains); GPU-resident fine level on both OpenACC and OpenMP offload.Full algorithm and user documentation:
docs/documentation/amr.md(support matrix enforced at runtime by the checker — unsupported combinations abort with named messages, never silently).AMR physics support matrix (abridged; authoritative table in amr.md)
Supported and golden-tested: single- and multi-fluid (5-eq,
mpp_lim) · 6-eq with per-block pressure relaxation · viscous (refluxed) · phase change (relax) · chemistry incl. species diffusion · Euler–Euler bubbles (polytropic/non-polytropic, mono/polydisperse, QBMM incl. non-polytropic with per-blockpb/mvside-state; dynamic regrid + subcycle) · acoustic sources (coarse-grid support with regrid exclusion) · immersed boundaries (multi-body, static or prescribed-motion, incl. dynamic regrid with body-containment expansion and per-substage guards) · 2D axisymmetric (per-block WENO-coefficient recompute) · stretched grids (exact parent-bisection ghost coordinates + per-swap coefficient recompute) · hybrid WENO/Riemann sensors (per-level) · Lagrangian bubbles (cloud excluded from blocks; two-way coupling on the coarse grid; regrid clips around the moving cloud) ·active_box(blocks contained in the growing window; agrees with plain AMR to ~1e-14) · IGR (restriction-only coupling: fine sigma solve seeded/Dirichlet-bounded by the coarse solve; documented truncation-order seam, exact free-stream) · 1D MHD/RMHD (div(B)=0 by construction in 1D; HLL and HLLD, incl. relativistic).Gated with named aborts (documented rationale): surface tension (seam force imbalance is structural — three fixes attempted and diagnosed in amr.md) · 2D/3D MHD (attempted and measured: the c/f seam is a continuous O(1) div(B) source GLM cleaning cannot remove — needs constrained-transport-class B prolongation/reflux) · hyperelasticity · 3D cylindrical (global azimuthal filter) · force-driven IB (
moving_ibm=2) · STL bodies · Riemann-extrapolation BCs (bc=-4) ·amr_subcycleunder IGR · stretched grids with Lagrangian/IB-regrid (uniform-spacing index conversions).Validation evidence
Known issues (all non-gating or in progress)
continue-on-error): an intermittent post-detected NaN on the two Lagrangian+AMR goldens. Exhaustively unreproducible off GitHub's runners — the exact failing stack (NVHPC 24.3 SDK,-tp=px -Kieee, HPC-X MPI, and the CI docker image itself under apptainer) passes elsewhere, as do native/zen2 builds; 24.5+ green. Documented at the golden definitions.Review guide
The commit history is arc-ordered (active-box → load-weight → SFC → weighted decomposition → rank timing → hybrid → m_box → AMR rungs → physics envelope → CI/GPU hardening); reviewing by arc is much easier than by file. The AMR arc builds stepwise: static hierarchy → restriction/prolongation → fine advance → refluxing → regrid → subcycling → multi-rank → GPU → each physics rung with its own validation. Commit messages carry the validation evidence for their change (measured defects, golden UUIDs, repro details for CI fixes).
All parameters ship default-off with
case_validatorentries, runtime checker gates, andcase.md/amr.mddocumentation.