BMDB nightly red since 08-18: the baseline accept captured an outlier pass for BIOMD0000001065, which has been failing intermittently since June (not a regression)

[jcschaff]

Summary

The BMDB nightly has failed on every run since 2026-08-18 with the same single change:

| BIOMD0000001065.omex | PASSED | FAILED (SOLVER_FAILURE) |

SOLVER_FAILURE count went 28 → 29; everything else in the report matches the baseline. This is
not part of the #1981 churn — that issue covers 849/872/925/961, and 1065 is none of them.

Latest run: 32343249649.

It is a deterministic regression, not a flake

Nightly	Result	1065
08-08 → 08-16 (9 runs)	✅ success	passing
08-17 (32005421226)	❌ failure	still passing — that run's 12 changes were all the COPASI annotation fix (#1981)
08-18 (32094317081, 32110561618)	❌ failure	first failure
08-20 (32343249649)	❌ failure	failing

Nine consecutive passes then three consecutive failures, with the same message each time.

The failure

vonDassow2000_1x4.timecourse1.sedml, task1, aborts at t = 450 after 483 ms:

CVODE solver failed : CV_REPTD_RHSFUNC_ERR: repeated recoverable right-hand side function errors
  : FunctionDomainException : pow(u,v) and u=-0.000000<0 and v=2.619640 not an integer
in "((0.0528… * ((0.000305 / (0.000305 + (EN_0_0 ^ 2.61964))) ^ 9.63095)
    / (5.76e-23 + (0.4 * ((0.000305 / (0.000305 + (EN_0_0 ^ 2.61964))) ^ 9.63095))))
    - (0.132 * ci_0_0))"
  where:
    EN_0_0 = -0.000000
    ci_0_0 = 1.000000

A Hill term raises the species EN_0_0 to a fractional power (2.61964). EN_0_0 has gone
very slightly negative, so pow is genuinely undefined and the solver's domain guard fires.
CVODE treats it as recoverable, retries with smaller steps, and eventually gives up
(CV_REPTD_RHSFUNC_ERR).

Note the guard wording differs from VCell's Java one
(ASTPowerNode.java:132 says u^v and u<0 …), so this is the C++ guard in
SundialsSolverStandalone — i.e. vcell-solvers, not this repo.

-0.000000 prints the same for true IEEE negative zero and for a tiny negative like -1e-9.
Which of the two it is has not been established, and it matters — see "open question" below.

Where it was introduced

Regression window is the 08-17 nightly commit → the 08-18 nightly commit,
c93d087cc..ac5774d5a — 10 commits. Three are SBML-importer changes from the #803 series, all
merged the evening of 08-17:

commit	PR	what it does
6f4c62345c	#1985	Hoist species initial conditions into global parameters — s4.initConc becomes a global s4_initConc, referenced symbolically
b6dba107fa	#1985	Use Structure.StructureSize for compartment sizes outside the application — makes a previously-conditional swap unconditional
089568c8c9	#1985	drops a log-only list

The remaining seven are a release-script fix, a baseline accept, and an error-message change —
none of which can move a trajectory.

Both structural changes alter the generated math for SBML imports, and either could move
EN_0_0 by the tiny amount needed to tip a knife-edge Hill term negative:

• b6dba107fa is the stronger suspect. Making the size swap unconditional means any model where
  the application's StructureMapping size differs from Structure.StructureSize now resolves to
  a different number, which rescales concentrations.
• 6f4c62345c should be mathematically neutral — it makes an initial condition symbolic rather
  than literal — but it does change the generated expression, and constant folding at a different
  point can change the last bits.

Suggested next step

A one-model bisect settles it cheaply — no full nightly needed:

BIOMD0000001065.omex at c93d087cc      (expected: passes)
                     at 6f4c62345c     (hoist only)
                     at b6dba107fa     (hoist + size swap)

Then diff the generated math for the two importer revisions. If EN_0_0's trajectory or any
compartment size changed, that identifies the commit and probably the fix.

Open question worth settling either way

Is EN_0_0 genuinely IEEE −0.0, or a small negative from integration drift?

• If it is −0.0, the solver's domain guard is wrong: -0.0 < 0 is false in IEEE, so
  pow(-0.0, 2.61964) should return +0.0 (C99 §F.9.4.4), not raise. That would make this an
  instance of VCell does not properly handle -0.0 in the IEEE standard within Expressions, creating far-reaching problems #1646 ("VCell does not properly handle -0.0 … creating far-reaching problems"),
  which is currently marked Done on the project board — and would mean it is not fully fixed.
• If it is drift, the guard is correct and the real question is why this model now dips
  negative when it did not before, which points back at the importer change.

The two have completely different fixes, so this should be answered before anyone writes code.
Printing the base with %.17g at the guard, or checking signbit, distinguishes them immediately.

Meanwhile: the gate is red and staying red

Per the workflow's own message, accepting the result means committing the
test_cases.updated.ndjson artifact over vcell-cli/src/main/resources/test_cases.ndjson.
That should not be done here — the point of #1901's gate is that a real regression must not be
absorbed into the baseline and made silent again, which is exactly the argument #1981 makes for
its four models. Until this is diagnosed the nightly will keep failing, which is the gate working.

_{Found while triaging the nightly failure; run history, commit window and code references verified against the working tree.}

[jcschaff]

Correction: both suspects in the issue body are ruled out

@jcschaff suggested comparing the generated MathDescription across the bisect rather than
running the solver, to separate model processing from the solver and its runtime (runner, libm,
solver binary). Following that line first — by asking whether this model even has the constructs
those commits touch — rules both of them out without building anything.

I downloaded BIOMD0000001065.omex from EBI and inspected the SBML:

compartments=4  species=148  globalParameters=71  rules=34  initialAssignments=0

b6dba107fa (unconditional Structure.StructureSize swap) cannot affect this model — twice over:

• Nothing outside the application references a compartment id. No initialAssignment or rule in
  the document names cell_0_0…cell_0_3, so the swap has nothing to act on.
• All four compartments are size="1" constant="true", so even if it fired, the
  StructureMapping size and Structure.StructureSize are both 1. No numerical difference is
  available.

6f4c62345c (hoist species initial conditions into global parameters) cannot affect it either:

• The document has zero initialAssignment elements, and no global parameter references a
  species initial concentration (no parameter id contains init; EN_0_0 simply carries
  initialConcentration="0.25"). There is nothing to hoist.

The other eight commits in the window are two merges, a log-only deletion (089568c8c9), an error
message change (b0b0a67660), the baseline accept (84df64f461, data only) and a
release-script fix (5c7745bd6a). None of them can move this model's math.

So the regression window contains no VCell change capable of causing this, which makes the
environmental explanation the leading one.

What that implies

pow(u, 2.61964) with u at the edge of zero is decided by the last bits of u. If the
generated math is unchanged, then whatever flipped it lives downstream of model processing:

• the SundialsSolverStandalone_x64 binary in the image (rebuilt nightly by the build job),
• the base image's libm — ubuntu:jammy picking up a patched glibc changes pow/exp in the
  last ulp, which is exactly enough here,
• or the runner hardware/instruction path.

None of those are visible in git log, which is why a code bisect was never going to find it.

The test that actually discriminates — but it must test identity, not equivalence

@jcschaff's caveat is the crux, and it is quantifiable. MathDescription.testEquivalency reports
equivalence, which deliberately tolerates small floating-point differences in constants:

// ExpressionUtils.java:438-439
double defaultAbsoluteTolerance = 1e-12;
double defaultRelativeTolerance = 1e-9;

A 1e-9 relative tolerance is about seven orders of magnitude looser than the last-ulp
(~1e-16 relative) difference that decides the sign of EN_0_0 as it passes zero. So
testEquivalency would cheerfully return MathEquivalent_* for two maths whose constants differ
by exactly the amount that flips this bug. Using it here would produce a confident false
negative.

So the comparison has to be textual identity of the serialized math, at full precision:

1. At c93d087cc (last good nightly sha): import BIOMD0000001065.omex, generate math for the
   timecourse1 application, write mathDesc.getVCML() to math_good.vcml.
2. At ac5774d5a (first failing nightly sha): same, to math_bad.vcml.
3. diff math_good.vcml math_bad.vcml — any difference is significant, including a single
   digit in a constant.

Full precision does survive serialization, which this bug incidentally demonstrates: the solver
input carried 0.052821806954783216692 and 2.6196399999999999686, i.e. ~20 significant digits,
so a byte diff will see a last-ulp change rather than hiding it behind rounding.

testEquivalency is still worth running second, on the same pair — if the texts differ but it
says MathEquivalent_*, that tells you the difference is numerically small and therefore
consistent with the knife-edge story, rather than a structural change to the model. The two
answers together are more informative than either alone.

(BMDB_SBMLImportTest.download() already fetches an archive by BioModels id from EBI, so step 1
needs no new plumbing. MathGenCompareTest shows the math-generation call sites.)

Expected outcome given the SBML analysis above: the two files are identical, which converts
this from "find the bad commit" into "find what changed in the solver image", and is worth
recording either way.

If the math is identical, the fix is probably not a bisect at all

Two things become worth doing regardless of which commit is blamed:

1. Settle the -0.0 question (still open, and still the highest-value single fact). Print the
   base with %.17g or test signbit at the C++ guard. If it is true IEEE -0.0, the guard is
   simply wrong — -0.0 < 0 is false, and C99 §F.9.4.4 says pow(-0.0, 2.61964) is +0.0 — and
   this is an instance of VCell does not properly handle -0.0 in the IEEE standard within Expressions, creating far-reaching problems #1646, which the board currently marks Done.
2. Consider clamping. A Hill term raising a concentration to a fractional power is
   guaranteed to hit this whenever the integrator undershoots zero, which for a stiff 148-species
   model is a matter of when, not if. Whether VCell should clamp negative concentrations to zero
   before a fractional pow is a real modelling decision, not just a guard fix — but a model that
   fails at t=450 on -0.0 will keep being fragile even after this particular flip is explained.

_{Correction to my own analysis above — I had the window right and the mechanism wrong, and checking the model's actual SBML is what caught it.}

[jcschaff]

Is the model source versioned? No — and that is worth fixing separately

@jcschaff asked whether the SBML itself could have been updated. Checked, and the answer is
"it could have been, but it wasn't".

The corpus is not pinned. NightlyBMDB_CLI.yml fetches it with:

- name: Checkout Input Files
  uses: actions/checkout@v4
  with:
    repository: sys-bio/temp-biomodels      # no ref:, no tag, no sha

With no ref:, actions/checkout takes whatever the default branch HEAD is that night. So an
upstream commit can change the corpus under the nightly, flip a result, and leave nothing in this
repo's history to explain it. That is a real latent fragility in the gate — a baseline diff with no
corresponding VCell change would be almost impossible to diagnose after the fact.

But it is not what happened here:

	Last changed
omex_files/BIOMD0000001065.omex	2025-05-21
sys-bio/temp-biomodels HEAD (whole repo)	2025-06-04

Both more than a year before the 08-18 regression. The corpus is frozen in practice, just not by
construction.

Recommendation: pin it — ref: <sha> or a tag — and bump deliberately. Then a corpus change
becomes a reviewable commit in this repo instead of an invisible input. This applies to the gate
generally, not just to this bug.

A correction to my previous comment, which survives re-checking

My SBML analysis above was done on the archive from EBI, not the one the nightly uses. They are
not the same file:

source	size	sha256
EBI biomodels/search/download?models=BIOMD0000001065	25 KB	0dc8895e…
sys-bio/temp-biomodels/omex_files/BIOMD0000001065.omex	2.07 MB	cd72e2ab…

The corpus archive is the full COMBINE bundle — SED-ML, reference results from AMICI/COPASI/
Tellurium, figures, a Tellurium script — with the model in
vonDassow2000_1x4.timecourse1.xml (620 KB), matching the name in the solver error.

I re-ran the analysis against the correct file. Every relevant number is identical, so the
conclusion in my previous comment stands unchanged:

compartments=4  species=148  globalParameters=71  rules=34  initialAssignments=0
all compartments: size="1" constant="true"
EN_0_0: initialConcentration="0.25", no initialAssignment
references to a compartment id from outside the application: NONE

Both b6dba107fa and 6f4c62345c still have nothing to act on in this model.

Where that leaves the diagnosis

Ruled out so far, each with evidence rather than argument:

• the two #803 SBML-importer commits — no constructs in this model for them to touch
• the other eight commits in the window — merges, log/message-only, data, CI
• an updated model file — unchanged for over a year
• a flake — 9 consecutive passes, then 3 consecutive identical failures

What remains is downstream of model processing: the SundialsSolverStandalone_x64 binary, the
base image's libm, or the runner. None of which git log on this repo can show.

That makes the identity-diff of the generated math (previous comment) the decisive next step — not
to find a bad commit, but to prove there isn't one, and redirect the search to the solver image.

Also worth noting: the 11 "stopped running" models are fine

The report's Historical 1072 / Current 1061 gap and the 11-row "Incomparable Results" table look
alarming but are accounted for — all 11 carry known_status: SKIP in test_cases.ndjson
(checked 235, 1008, 1042). Deliberate and documented, not silent coverage loss.

[jcschaff]

Corpus is now pinned — PR #2003.

ref: 6a09daf46af1bb89e4857436b623a7b8720863ad (main @ 2025-06-04), which is the commit the nightly was already resolving to, so it changes no results. The hole it closes is that the gate compares two inputs — the baseline and the corpus — and only the baseline was versioned in this repo. Bumping the corpus is now a reviewable PR, and its baseline diff a separate commit, which keeps "the models changed" apart from "VCell changed".

That further narrows this issue: with the corpus frozen and no VCell change in the window capable of touching this model, the remaining explanation is downstream of model processing — the solver binary, libm, or the runner.

[jcschaff]

Result: the generated math is byte-for-byte identical across the regression window

Ran it. BIOMD0000001065 imported and math-generated at both nightly shas, dumping
mathDesc.getVCML() each time:

sha		SHA-256 of generated math
c93d087cc	last green nightly	5c0f65c5b4fcbe1c2619a550c4a5b2ead265f125da21acb35739f6d889966c8e
ac5774d5a	first red nightly	5c0f65c5b4fcbe1c2619a550c4a5b2ead265f125da21acb35739f6d889966c8e

76,211 characters each, diff clean. Not "equivalent" — identical, so the 1e-9 relative
tolerance in testEquivalency never enters into it.

No VCell change in the window alters this model's math. The failure is downstream of model
processing.

The negative control, because an always-identical harness would prove nothing

A comparison that cannot detect a difference is not evidence. So I ran the same harness on
BIOMD0000000872 — the model 6f4c62345c was specifically written to fix — across the same
two shas:

sha	BIOMD0000000872
c93d087cc (good)	import fails: Error binding global parameter 'beta' to model: 'UNRESOLVED.initConc' …
ac5774d5a (bad)	imports cleanly, generates 3,351 chars of math

The two builds demonstrably differ, and the harness demonstrably sees it. The identical result for
1065 is therefore a real finding rather than a broken measurement.

What is now ruled out, with evidence

• the two #803 SBML-importer commits — no constructs in this model for them to touch, and
  now: identical generated math
• the other eight commits in the window — merges, log/message-only, data, CI
• an updated model file — BIOMD0000001065.omex unchanged since 2025-05-21; corpus HEAD
  2025-06-04; now pinned by ci(bmdb): pin the BMDB corpus to a sha instead of tracking main #2003
• a flake — 9 consecutive passes, then 3 consecutive identical failures
• any model-processing change at all — the math is bit-identical

What remains is the execution environment: the SundialsSolverStandalone_x64 binary in the
nightly image, the base image's libm, or the runner. Identical input math, different outcome.

Suggested next step, now that the search area is small

The nightly build job rebuilds the Docker image each run, so the solver binary and libm come
along for the ride. Two cheap checks:

1. Did the solver binary change? vcell-solvers releases are pinned in the build — compare the
   version fetched by the 08-17 vs 08-18 runs. (Incidentally, the rebuild during this exercise
   pulled vcell-solvers v0.0.40.)
2. Did the base image move? ubuntu:jammy is a floating tag. A glibc patch changing pow by
   one ulp is precisely the magnitude required here, and would be invisible to every check we have.

If both are unchanged, the remaining explanation is runner-to-runner CPU variation, which argues
the model is simply too close to the edge to gate on — see the clamping point below.

And the -0.0 question is now the highest-value item

Still unanswered and still cheap: print the base with %.17g or test signbit at the C++ guard.

If it is true IEEE -0.0, the guard is wrong on its own terms — -0.0 < 0 is false, and C99
§F.9.4.4 gives pow(-0.0, 2.61964) = +0.0 — and this becomes an instance of #1646, currently
marked Done on the board.

Separately, and regardless of what flipped it: a Hill term raising a concentration to a
fractional power will hit this whenever the integrator undershoots zero. For a stiff 148-species
model that is a matter of when, not if. Whether VCell should clamp a negative concentration to zero
before a fractional pow is a real modelling decision — but this model will stay fragile even
after the immediate cause is found.

_{Method: SBMLImporter(fis, logger, false).getBioModel() → simContext.updateAll(false) → getMathDescription().getVCML(), built with mvn -pl vcell-core -am package at each sha under Java 17. Model taken from the corpus archive (vonDassow2000_1x4.timecourse1.xml), not the EBI download, which is a different file.}

[jcschaff]

Solved — and it is not a regression. I had this backwards.

Checking the solver binary and base image led somewhere else: BIOMD0000001065 has been failing
for over two months. It is intermittent, and a baseline accept captured one of its rare passes.

What the committed baseline actually said, over time

baseline commit	date	BIOMD0000001065
ac0b976ccd	2026-06-09	FAIL / SOLVER_FAILURE
3853c4c490	2026-08-12	FAIL / SOLVER_FAILURE
30aa3239cf	2026-08-12	FAIL / SOLVER_FAILURE
84df64f461	2026-08-17	FAIL / SOLVER_FAILURE
52cf62cbc3	2026-08-18 05:04 UTC	PASS ← changed here

52cf62cbc3 is "Accept the BMDB baseline after the SBML scope fixes".

The sequence

when (UTC)	what	1065
2026-06-09 → 08-17	every nightly	FAILED, matching a FAIL baseline → green
08-18 03:07	workflow_dispatch run 32094317081	PASSED ← the outlier
08-18 05:04	52cf62cbc3 commits that run's test_cases.updated.ndjson	baseline becomes PASS
08-18 07:15, 08-20	nightlies	FAILED → mismatch with the new PASS baseline → red

The 08-16 green run confirms it: its report has no "Unmatched Results" section at all, and the
baseline then said FAIL — so 1065 failed there too, silently and correctly.

The gate is red because the baseline is wrong, not because anything regressed.

Why I got it wrong

I read the nightly reports' "Unmatched Results" tables as absolute results. They are diffs
against a baseline that was itself moving. "Nine consecutive passes then three failures" was
really "nine runs agreeing with a FAIL baseline, then a baseline flip to PASS, then three runs
disagreeing with it". Same table, opposite meaning.

The other two checks, for the record

Both come back clean, which is consistent with "no regression":

• Solver binary — unchanged. solvers-vcell-linux.version is v0.0.44-dev4 at both shas
  (git diff c93d087cc ac5774d5a -- pom.xml is empty), and that release's linux64.tgz asset was
  created 2024-02-11 and never re-uploaded.
• Base image — did move, but it is a red herring. ubuntu:jammy was republished
  2026-08-18 04:04 UTC, which lands neatly between the 03:07 pass and the 07:15 failure. Very
  tempting. But 1065 had been failing for two months on the old image, so the image is not what
  makes it fail — at most it removed whatever briefly let it pass.

Combined with the byte-identical generated math, nothing on the VCell side changed at all.

Recommended fix

Restore 1065 to FAIL / SOLVER_FAILURE in test_cases.ndjson — its documented state from
2026-06-09 to 2026-08-17 — and record that it is intermittent. That turns the nightly green
immediately and honestly, because failing is this model's normal behaviour.

I have not made that change; it is a one-line edit to the baseline and the call is yours.

The process problem worth fixing, which is the real finding

The accept step takes a whole run's results wholesale. If any model is intermittent, accepting
a run in which it happened to pass writes an unachievable expectation into the baseline and the
gate goes permanently red — for a model nothing is wrong with.

This is a sharp edge on an otherwise good gate (#1901), and it just cost several hours of
bisecting a regression that did not exist. Options, roughly in order of effort:

1. Diff the accept. Have the accept step show which entries move FAIL → PASS and require them
   to be eyeballed. A model going green is exactly when you want a human to ask "reliably?"
2. Track flakiness. Anything that has flipped in the last N runs cannot be auto-accepted.
3. Re-run before accepting. Confirm a FAIL → PASS transition twice before baselining it.

Worth noting the gate did its job here — it went red and stayed red on a real inconsistency. It
just named the wrong culprit, because the baseline is the one input it cannot check.

Still worth doing regardless

The -0.0 question is unchanged and is now the only open technical item: is EN_0_0 true IEEE
-0.0 (in which case the C++ guard is wrong, since -0.0 < 0 is false and C99 §F.9.4.4 gives
pow(-0.0, 2.61964) = +0.0, making this an instance of #1646) or a small negative from integration
drift?

If the guard is wrong, fixing it may make 1065 pass reliably rather than occasionally — which
would be the real fix, and would explain the intermittency as "sometimes the drift lands exactly on
-0.0 and sometimes just below it".

[jcschaff]

Split the model-level question out to #2005 — why 1065 flaps, and the missing INTERMITTENT status needed to express it.

Short answer to "can we already mark it as flapping?": half of it exists. known_failure_type already does the failure-signature check — a case documented FAIL only counts as changed if it fails with a different type (OmexTestReport, the known_status == FAIL branch), and known_failure_desc is deliberately not compared since it embeds a per-run temp path. What is missing is a status that tolerates either outcome: Status is a closed {PASS, FAIL, SKIP}, so for a genuinely intermittent model there is no setting that is green both ways.

That means restoring 1065 to FAIL/SOLVER_FAILURE here is still the right immediate action, but it is a stopgap — the gate will go red again the next time the model passes.