From ca2f7f7e1f097434d463770fce7c4a3d9b57ba40 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 16:18:55 +0000 Subject: [PATCH 1/9] spec(LTX25-DEVICE-RESIDENCY): the span-slack bound's fifth shape, and the coverage arm a mutation refused (#1572, #1576) `test_ltx2_video` has carried four wall-clock constants and each was afterwards found redding an unmutated tree. This records the measurement that replaces the fifth one with a number the run takes itself, and the companion change to the coverage floor that was written, measured and withdrawn. 120 runs of the containment case at `73ada0df8` on a 20-core box at loadavg 40-155, 1440 leaf-record observations, one binary: one run red, at loadavg 155.65, and its two failures are one 67.55 ms descheduling between `ppm_phase.Close()` and the `artifacts.frames` destructor. Against the run's own measured boundary cost that event is 0.645 of the ceiling; against the 30 ms constant it is 2.25x over. Two denominator shapes are recorded as rejected because both are the obvious thing to reach for: bursts taken beside the render reach 2.954, and the continuous sampler's 99.9th percentile reaches 199.4, where its maximum reaches 0.645. The coverage floor keeps its share and gains nothing, because the second arm makes `artifacts.frames.ppm` opened after the write loop instead of around it pass with 1.1% coverage. That leaf is 0.2-7 ms while one boundary on this host is 0.3-1.2 ms pinned to two idle cores, so no allowance built from the boundary is smaller than the leaf. #1470 keeps it, and the repair named there is an anchor rather than a threshold. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-device-residency.md | 198 +++++++++++++++++++++++- .agents/specs/ltx25-phase-residue.md | 4 +- 2 files changed, 197 insertions(+), 5 deletions(-) diff --git a/.agents/specs/ltx25-device-residency.md b/.agents/specs/ltx25-device-residency.md index 47a1de221..71bd7a6ec 100644 --- a/.agents/specs/ltx25-device-residency.md +++ b/.agents/specs/ltx25-device-residency.md @@ -1457,13 +1457,20 @@ which is where this spec's `### Decisions taken here` already said they would be The `load` and `generate` records are SPANS and are never summed, so they are not in this list; `sum_rule` in every emitted table says which records add up. -* **(1c) DOES NOT HOLD A LEAF RECORD SHORTER THAN 60 ms**, and this - entry names which ones so no later stage inherits the silence. The span-slack - bound is per leaf RECORD and its `min` cap binds below +* **(1c) DOES NOT HOLD A LEAF RECORD SHORTER THAN `8 x THIS RUN'S BOUNDARY + CEILING`**, and this entry names which ones so no later stage inherits the + silence. The span-slack bound is per leaf RECORD and its `min` cap binds below `2 * kSpanSlackPerRecord`; a record there is reported and not checked, because the capped bound is a 50% share and the honest head-plus-tail on those records measured 4.6-72.3% of the record itself ([#1559](https://github.com/mudler/vllm.cpp/issues/1559)). + **That floor was a flat 60 ms and is now a per-run number the case prints** + (`### The span-slack bound, fifth shape`). Over 120 runs on this box it ran + 2.4 ms to 838 ms, median 26.6 ms, so the ranges in the table below -- which + were measured against the 60 ms floor -- now say which records are resolvable + on a QUIET run rather than on every run. A record the run cannot resolve is + counted in the case's own `N below this instrument's resolution` line. + | Record | Measured range | What still holds it, and what escapes | |---|---|---| | `artifacts.frames`, nine-frame render, ONE record | 0.90-6.21 ms | (2) at 0.50 holds it TIGHTLY, because a single-record leaf makes the coverage floor exactly the capped span bound. Plus the (0) count and `CheckWriterIsBesideTheDecode` | @@ -2950,3 +2957,188 @@ the reason is that the document publishes none: `## Inspect a render` describes `sum_leaf_seconds`, `unaccounted_seconds` and the two environment switches, and names no individual phase. Checked rather than assumed, because a sentence written when a document had a different shape is how a stale obligation survives. + +### The span-slack bound, fifth shape: the instrument measures its own resolution (#1572, #1576) + +Four issues describe one behaviour -- `test_ltx2_video` decides by box load -- +and every repair so far has replaced one constant with a larger constant. + +| Issue | What it recorded | State after this change | +|---|---|---| +| [#1576](https://github.com/mudler/vllm.cpp/issues/1576) | (1c) is a fixed wall-clock budget with an allowance for INSTRUMENTATION and none for LOAD, measured swinging 171x on one binary | **closed here** | +| [#1572](https://github.com/mudler/vllm.cpp/issues/1572) | (1c) redding intermittently on `main`, `decode.video` at 0.00256913 against 0.00075 | **closed here** | +| [#1536](https://github.com/mudler/vllm.cpp/issues/1536) | the phase-coverage floors missed by 0.18-0.8% on an idle box, while the suites that ARE load-dependent passed | **thesis refuted by measurement**, see below | +| [#1470](https://github.com/mudler/vllm.cpp/issues/1470) | one case of 96 false-redded under load, four greens after, case identity never captured | **identity and rate measured, STAYS OPEN**, see below | + +The history of the number they are about is the argument for not choosing +another one: 0.95 of the leaf, then 0.25 ms per record, then 3 ms under a +sanitizer, then 30 ms everywhere. Each was calibrated on a sample of one box and +each was afterwards found redding an unmutated tree. The 30 ms also cost +resolution: it widened the smallest swallow (1c) can see from about 0.5 ms to +60 ms. + +#### The red-before, on an UNMUTATED tree at `73ada0df8` + +120 runs of the containment case alone, one Release CPU-only binary, one build +directory, this 20-core box at loadavg 40-155, full output kept, 1440 +leaf-record observations. **One run red.** + +``` +b3_003 loadavg 155.65 133.50 102.10 +test_ltx2_video.cpp:4074: CHECK( span_slack <= span_bound ) 0.0675591 <= 0.03 +test_ltx2_video.cpp:4122: CHECK( covered >= c.min_coverage * leaf_seconds ) + 0.00730277 >= 0.0374371 +[doctest] test cases: 1 | 0 passed | 1 failed +``` + +Both failures are **one event**. `artifacts.frames` render 2 record 2 runs +[55.431342293, 55.499995634] -- 68.65 ms -- and its `artifacts.frames.ppm` +sub-scope runs [55.431346873, 55.432441132]. The 67.55 ms between +`ppm_phase.Close()` and `write_phase`'s destructor holds two statements +(`rendered_frames += chunk.frames.frames` and the trace increment) and one +descheduled thread. It reds (1c) at 2.25x the 30 ms constant, and it reds (2) +because 67.55 ms of a 74.87 ms leaf is then uncovered. + +That is the cluster in one observation: the SAME scheduler event decides (1c) +and (2), on a tree nobody touched, and a reader of either red learns nothing +about the tree. + +#### The design: the denominator is measured, and it is measured FOR the render + +A boundary is not a constant, so the bound is not one either. What one costs is +read out of `render_phase_log.cpp` rather than guessed: `PhaseLog::Open` takes +the process-wide mutex, stamps `o.start`, appends the open entry, calls +`SampleLocked` -- one `/proc/self/statm` read -- and, on the shipped default, +writes and FLUSHES one line to stderr. `Close` takes the same mutex, samples, +THEN stamps `r.end`, and flushes its own line. Both stamps sit inside that work, +which is exactly what puts the cost inside the leaf record. + +`InstrumentCeiling` times those three operations, through the instrument's own +public entry points, from a thread that runs FOR the load and both renders at +one sample per millisecond, and the bound reads the WORST it saw. + +**The worst, and not a mean, and that is the design rather than a detail.** The +numerator is itself a maximum over a small number of draws. A mean denominator +under a maximum numerator is the shape `ltx25-phase-residue.md` `## Design` 3 +measured at 4 reds in 45 runs -- worst ratio 4.115 against a bound of 1 -- and +withdrew, because under contention the un-instrumented remainder of a boundary +dilates faster than the instrumented part, so the comparison has a heavy right +tail rather than a shifted median. Here both sides are tail quantities. + +**Two denominator shapes were measured and rejected**, and both are worth +carrying forward because both are the obvious thing to reach for: + +| ceiling shape | worst `slack / ceiling` over 1440 observations | +|---|---:| +| bursts of 4096 taken BESIDE each render (about 40 ms of a case that runs for tens of seconds) | **2.954** | +| the continuous sampler's 99.9th percentile | **199.4** | +| **the continuous sampler's maximum** | **0.645** | + +The 99.9th percentile is the interesting reject. It is 20x to 200x tighter and +would be a far better instrument if it were safe. It is not, for the same reason +the mean is not: it is not a maximum. On run b3_003 it read 0.339 ms against a +67.55 ms boundary, so it would have kept the very red this change exists to +explain. + +#### The bound, and the whole of the judgement in it + +```cpp +const double kBoundariesPerRecord = 4.0; +const double kSpanSlackPerRecord = kBoundariesPerRecord * ceiling.ceiling_seconds(); +const double span_bound = std::min(kSpanSlackPerRecord, 0.5 * record_seconds); +``` + +The head of a leaf record is `Open(leaf)`'s sample and flushed line followed by +`Open(sub)`'s acquisition of the same mutex; the tail is `Close(sub)`'s flushed +line followed by `Close(leaf)`'s acquisition, sample and stamp. TWO +boundary-sized operations. `2 x ceiling` is what the structure gives; the four +carries one factor of two over it, and that factor is the only number in the +line that was chosen rather than derived. Against the measured population it is +6.2x inside the worst observation, and zero of 1440 records violate it at any +`C` from 1 upward. + +#### What it costs, and what it buys, measured on the same box + +The bound is a property of the run, so its RESOLUTION is too. Measured over the +same 120 runs, `4 x ceiling` ran **1.18 ms to 419 ms, median 13.3 ms**, against +the flat 30 ms it replaces. On the quiet end that is 25x tighter; at loadavg 155 +it says out loud that it can resolve nothing under a tenth of a second, which +run b3_003 proves is the truth about that box rather than a weakness of the +bound. + +Staged as a mutation, the difference is visible directly. `D-unnamed-head` +inserts a phase nobody named, of a known size, between `Open("decode.audio")` +and `Open("decode.audio.mel")` -- the exact defect (1c) exists for: + +| mutation | old bound (30 ms) | new bound | +|---|---|---| +| 20 ms un-named head | **GREEN on (1c)** (caught only by `decode.audio`'s 0.99 coverage floor) | **RED on (1c), 3 of 3**, at a run ceiling of 79-126 us and a bound of 0.32-0.50 ms | +| 2 ms un-named head | **GREEN 3 of 3** | RED 1 of 3 -- and the two greens report a ceiling of 2.26-2.56 ms, so the bound was 9-10 ms and the run said so | + +#### The same term was written for (2), MEASURED, and WITHDRAWN + +The obvious companion change is to give the coverage floor the same allowance as +a second arm: + +```cpp +CHECK(covered >= c.min_coverage * leaf_seconds || uncovered <= boundary_allowance); +``` + +with `boundary_allowance = 2 * (leaf records + sub-scope records) * ceiling`. It +was written, built, and it makes b3_003's coverage red green. **It is withdrawn, +because it makes a real mutation green as well.** + +Mutation `B-empty-ppm` moves `phase::Scope ppm_phase("artifacts.frames.ppm")` +from before the `WriteFileBytes` loop to after it, so the writer's anchor stops +covering the writes -- the defect that anchor exists for. Coverage of +`artifacts.frames` falls from about 98% to **1.1% and 5.0%**. Under the +disjunction the case reported `carried by the INSTRUMENT ARM alone` and passed, +3 of 3. Against the floor as it stands it reds, 2 of 2. + +The reason is a property of the fixture and not of the arithmetic: +`artifacts.frames` is **0.2 ms to 7 ms** here, while one boundary on this host +is **0.3 ms to 1.2 ms even pinned to two otherwise idle cores** (`taskset -c +18,19`). Any allowance built out of the boundary is larger than the leaf. The +0.50 floor on that leaf is a RELATIVE test at a scale the absolute instrument +cannot reach, which is why it catches `B-empty-ppm` and why it reds under load: +those are the same property, and no threshold separates them. + +So (2) is unchanged by this row, and #1470 keeps it. What would settle it is an +anchor whose seconds the writer can be read against -- the same class of repair +`### Owed out of W0` already asks for on the short records, and the same class +`denoise.update` ([#1668](https://github.com/mudler/vllm.cpp/issues/1668)) is +for `denoise`. It is a production scope, not a number. + +#### Mutations + +Restored byte-for-byte after each, `git status` clean between runs; every +mutation prints `compile_status=0` and asserts its anchor matched exactly once, +so a mutation that failed to build or failed to apply cannot read as a pass. + +| Mutation | What it stages | Verdict | +|---|---|---| +| `C-no-sampler` | the `CeilingSamplerGuard` is deleted from the case, so every bound would be `4 x 0` | **RED**, `REQUIRE( ceiling.observations() > 0 )`, 45 assertions in | +| `D-unnamed-head-20ms` | a 20 ms phase nobody named in `decode.audio`'s head | **RED on (1c), 3 of 3** (green on (1c) at the 30 ms constant) | +| `D-unnamed-head-2ms` | the same at 2 ms | RED 1 of 3, and the greens print the ceiling that explains them (green 3 of 3 at the 30 ms constant) | +| `B-empty-ppm` | `artifacts.frames.ppm` opened after the write loop instead of around it | **RED on (2), 2 of 2** -- detection unchanged by this row, which is why the (2) arm was withdrawn | + +#### What this does NOT do, stated before it is relied on + +* **It does not close #1439.** `leaves >= 0.95 * wall` is untouched and is still + a share of a contended clock. +* **It does not close #1470.** The coverage floors still decide by load on + `artifacts.frames`, at a measured rate of 1 run in 120 on this box, and the + section above is the measurement that says why a bound cannot fix it. +* **#1536's thesis is refuted rather than repaired.** That issue's claim is that + the suite is "persistently red on an idle box, not load-flaky". At + `73ada0df8` it is the opposite: 119 of 120 green, and the single red was at + loadavg 155.65. Its 0.18-0.8% misses were against the 0.95 floor that + `6b48edb2c` moved to 0.75; the measured coverage on this box is 99.6% for + `denoise` and 98.4%-100% for the decode leaves, so nothing is near that floor. +* **The sampler perturbs what it measures.** It takes the phase mutex about a + thousand times a second against a render that takes it about thirty times in + total, and it writes to the same fd the live lane writes to. The perturbation + is inside the quantity it reports, which is the conservative direction. It is + also why the guard is RAII: a failing `REQUIRE` in this case would otherwise + leave a thread writing to fd 2 while the later cases in the file `dup2` a + capture file over it. diff --git a/.agents/specs/ltx25-phase-residue.md b/.agents/specs/ltx25-phase-residue.md index e83e3535e..1c5722406 100644 --- a/.agents/specs/ltx25-phase-residue.md +++ b/.agents/specs/ltx25-phase-residue.md @@ -234,10 +234,10 @@ once, at `tests/vllm/multimodal/test_ltx2_video.cpp:4325`, asking for it. | [#1569](https://github.com/mudler/vllm.cpp/issues/1569) | a gate on `WriteJson`'s clock ORDERING, **measured green under its own mutation**. Restoring the old order left the conservation case GREEN 10 of 10, at `wall 0.0608987s, unaccounted 0.000534223s, table charge 0.000301655s`, because the copy and sort of a three-record table are nanoseconds. Gating it needs a table with enough records for the sort to be measurable | | [#1570](https://github.com/mudler/vllm.cpp/issues/1570) | an upper bound on the instrument's own share of a leaf. `uncovered <= 2 * leaf_instrument` is stricter than the floor it replaces only while `leaf_instrument` stays small, and nothing bounds it. Moving the DiT `Tick` out of `Evaluate` would charge ~110 flushed writes to `denoise` and widen the gate while printing a small number | | [#1571](https://github.com/mudler/vllm.cpp/issues/1571) | a per-gap decomposition IN the emitted table. The 92% region above was found with a scratch script; a reader of `phase-log.json` still cannot see it without one, and the same investigation will be re-derived the next time the residue moves | -| [#1572](https://github.com/mudler/vllm.cpp/issues/1572) | assertion (1c)'s span slack reds intermittently on `main` — `decode.video` at `0.00256913` against a `0.00075` bound, 3.4x. Pre-existing from `6b48edb2c` and not this row's | +| [#1572](https://github.com/mudler/vllm.cpp/issues/1572) | assertion (1c)'s span slack reds intermittently on `main` — `decode.video` at `0.00256913` against a `0.00075` bound, 3.4x. Pre-existing from `6b48edb2c` and not this row's. **DISCHARGED** by `LTX25-DEVICE-RESIDENCY`'s fifth shape of that bound, which replaces the constant with `4 x` the boundary cost the run itself measured: `.agents/specs/ltx25-device-residency.md` `### The span-slack bound, fifth shape` | | [#1619](https://github.com/mudler/vllm.cpp/issues/1619) | **the `merge=union` driver duplicates a row, MEASURED on this row's own merges.** Both sides appended before the same trailing anchor rather than at the true end, so the driver concatenated two regions that each carried `#1546` and the resolved index held it TWICE, byte-identical, at 538 lines where the correct union is 537. `git merge-tree` called that merge clean and `check-issue-index-append-only.py` passed it, because a duplicate is an ADDITION and that checker only collects removals. `check-agent-record.py` did NOT pass it -- a claim #1556's spec made and this row REFUTED by reproduction: regenerating the raw driver output and running that same tree's checker returns rc=1 with `issue #1546 listed twice`, and the refusal has existed since `8dd6508da` (2026-08-09), before the merge. So the blind gate is exactly one checker, not two, and the gap is narrower than #1556 recorded. The de-duplication half is CONDITIONAL, and the condition is what #1556's spec omitted: the checker reds a repair only when the DUPLICATE IS ALREADY IN THE BASE. Measured at three pairings -- `--base e2a9e035d` against the real canonical 537-line file rc=0, against a synthetic 537 rc=0, and `--base --head <537 de-dup>` rc=1. It diffs `merge-base..HEAD`, so when the base predates the duplicate the addition and the removal CANCEL and it passes. Since `origin/main` is preflight's base, and is the shape this branch used, the gate does NOT red someone who repairs driver output before committing it -- only someone repairing a corruption that already landed. The same range property is why relocating a base-reachable row DOES red it: moving row `#168` to the end gives rc=1 and a `removed:` line naming it. So "de-duplicating in place FAILS the checker", as #1556's spec put it, is false unqualified and true once the duplicate is base-reachable. #1556's spec added that the same driver dropped `#838` on a later re-merge, making this a recurring class; that is WITHDRAWN as unreproducible. Re-running `git merge-file --union` at every later merge where `#838` was on a side leaves it present in all of them, and `git log -S` finds it absent from no committed state -- mechanically a union driver cannot drop a line that is an addition on one side. If it ever went missing, that points at a wholesale take-ours resolution rather than at the driver | | [#1439](https://github.com/mudler/vllm.cpp/issues/1439) | **NOT closed by this row, and it must not be.** See `## Risks and decisions` D4 | -| [#1470](https://github.com/mudler/vllm.cpp/issues/1470) | `test_ltx2_video` false-redded once on `main` under load and the failing case's identity was never captured. Untouched here | +| [#1470](https://github.com/mudler/vllm.cpp/issues/1470) | `test_ltx2_video` false-redded once on `main` under load and the failing case's identity was never captured. Untouched by THIS row, and **the identity and the rate are now measured** by `LTX25-DEVICE-RESIDENCY`: 1 red in 120 runs of the containment case at loadavg 40-155, on `artifacts.frames` render 2, where a 67.55 ms descheduling between `ppm_phase.Close()` and the leaf's destructor left 67.55 ms of a 74.87 ms leaf uncovered. It reds `covered >= 0.50 * leaf_seconds`. The same row also measured, and WITHDREW, the obvious repair: an instrument-relative second arm on that floor makes mutation `B-empty-ppm` — the writer's anchor opened after the write loop instead of around it, coverage 98% to 1.1% — pass. `artifacts.frames` is 0.2-7 ms on this fixture and one boundary on that host is 0.3-1.2 ms pinned to two idle cores, so no allowance built from the boundary is smaller than the leaf. The repair is an anchor, not a threshold | ## Stop conditions From 477b436d6ee586f8d53ec4405c6e2cbdf19f097b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 16:19:20 +0000 Subject: [PATCH 2/9] fix(LTX25-DEVICE-RESIDENCY): (1c) reads the boundary cost the run measured, instead of the fifth constant (#1572, #1576) `kSpanSlackPerRecord` was 30 ms, and before that 3 ms under a sanitizer, 0.25 ms plain, and a 0.95 share of the leaf. The quantity all four bound is what a phase boundary costs, which is a property of the machine and of what else held a core at that second, not of this tree. #1576 measured it swinging 171x on one binary and said a bigger constant would only move the load at which the same ambiguity returns. `InstrumentCeiling` times the three operations `PhaseLog::Open` and `PhaseLog::Close` both perform -- the process-wide mutex, `SampleNow()`, and one flushed write -- from a thread that runs for the load and both renders, and keeps the worst. The bound is four of those: two boundary-sized operations sit inside a leaf record's head and tail, and the four carries one factor of two over the structure it is derived from. The maximum, not a quantile, and that is measured. The numerator is itself a maximum over two draws; over 1440 leaf-record observations the worst `slack / ceiling` is 0.645 against the maximum, 2.954 against bursts taken beside the render, and 199.4 against the same sampler's 99.9th percentile. On this box the bound now runs 1.18 ms to 419 ms, median 13.3 ms, against the flat 30 ms. A 20 ms phase nobody named in `decode.audio`'s head is green on (1c) at 30 ms and reds 3 of 3 here. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- tests/vllm/multimodal/test_ltx2_video.cpp | 230 +++++++++++++++++++++- 1 file changed, 223 insertions(+), 7 deletions(-) diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 9443c00c2..601046aa3 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -30,11 +30,13 @@ #include #include #include +#include #include #include #include #include +#include #include #include @@ -3509,6 +3511,177 @@ std::vector LeafIntervals(const nlohmann::json& table, } // One carrying phase and the anchor that ties its name to its work. +// ─── WHAT ONE INSTRUMENT BOUNDARY COSTS, ON THE RUN THAT IS ASSERTING ─────── +// +// EVERY WALL-CLOCK TOLERANCE BELOW USED TO BE A COMPILE-TIME NUMBER, and each +// of the four this case has carried was afterwards found redding an UNMUTATED +// tree: the 0.95 coverage floor (#1439, #1494), 0.25 ms per leaf record (#1559, +// #1572), 3 ms under a sanitizer, and the 30 ms that replaced both (#1576). +// Each was calibrated against a sample of one box and each was wrong on the +// next one. +// +// The quantity all four bound is the wall a PHASE BOUNDARY costs, and that is +// not a property of this tree. It is a property of the machine, of the build +// configuration, and of what else held a core at that second: on ONE host, ONE +// build type and ONE binary it has measured 25.0 us and 10.032 ms with nothing +// mutated, a spread of 402x (#1559). #1576 measured 171x between two runs of one +// binary and said in its own text that a bigger constant "would only move the +// threshold at which the same ambiguity appears". That is what the 30 ms shape +// did: it widened the smallest swallow this case can see from about 0.5 ms to +// 60 ms and left a fixed budget with no allowance for load. +// +// SO THE NUMBER IS NOT CHOSEN HERE. IT IS MEASURED, ON THE RUN THAT ASSERTS. +// +// WHAT A BOUNDARY IS, READ OUT OF `render_phase_log.cpp` RATHER THAN GUESSED. +// `PhaseLog::Open` takes the process-wide mutex, stamps `o.start`, appends the +// open entry, calls `SampleLocked` -- one `/proc/self/statm` read folded into +// every live scope -- and, on the shipped default, writes and FLUSHES one line +// to stderr. `Close` takes the same mutex, calls `SampleLocked`, THEN stamps +// `r.end`, and flushes its own line. Both stamps sit inside that work, which is +// what puts the cost inside the leaf record and makes it the head and the tail +// that assertion (1c) below measures. +// +// One boundary is therefore one acquisition of that mutex, one sample, and one +// flushed write. This sampler times exactly those three, through the +// instrument's own public entry points, and keeps the WORST it saw. +// +// A MAXIMUM AND NOT A MEAN, AND THAT IS THE DESIGN RATHER THAN A DETAIL. +// The numerator of the bound below is itself a MAXIMUM over a small number of +// draws from this same distribution. A mean denominator under a maximum +// numerator is the shape `ltx25-phase-residue.md` `## Design` 3 measured at 4 +// reds in 45 runs -- worst ratio 4.115 against a bound of 1 -- and withdrew: +// under contention the un-instrumented remainder of a boundary dilates FASTER +// than the instrumented part, so that comparison has a heavy right tail rather +// than a shifted median, and a 20-run sample never sees it. Here the +// denominator is the MAXIMUM over tens of thousands of draws of the same +// operation taken across the same window, which is a tail quantity compared +// with a tail quantity. +// +// AND (2) BELOW IS DELIBERATELY NOT GIVEN THIS TERM. Adding `uncovered <= +// boundary allowance` to the coverage floor as a second arm was written, +// built and MEASURED here, and it is withdrawn: it turns mutation B -- +// `artifacts.frames.ppm` opened AFTER the `WriteFileBytes` loop instead of +// around it, which drops that leaf's coverage from 98% to 1.1% -- from a red +// into a green. The reason is that `artifacts.frames` is 0.2-7 ms on this +// fixture while one boundary on this host is 0.3-1.2 ms even pinned to two +// idle cores, so any allowance built from the boundary swamps the leaf. What +// the 0.50 floor there catches, and the load it reds under, are the same +// property of a leaf smaller than the instrument measuring it. #1470 owns +// that, and the repair is an anchor the writer's own seconds can be read +// against, not a threshold. +// +// AND IT IS SAMPLED ACROSS THE RENDER, NOT BESIDE IT. That is measured rather +// than assumed. A burst taken before and after each render covers about 40 ms of +// a case that runs for tens of seconds, and a contention spike inside a render +// is then absent from the denominator while being present in the numerator: with +// bursts alone the worst observed `slack / ceiling` was 3.96 over 161 leaf +// records. The sampler below runs FOR the render, which is what makes the +// denominator a statement about the window the numerator is drawn from. +// +// IT ALSO RETIRES THE PER-CONFIGURATION CONSTANT. A sanitizer instruments the +// scope boundary this quantity IS -- ASan checks a shadow byte on every access +// in it, TSan additionally keeps per-access happens-before state under a lock -- +// which is why the third shape needed 0.25 ms plain and 3 ms under a sanitizer, +// and why the nested `#if` that GCC's missing `__has_feature` made mandatory was +// in this file at all. This loop is built out of the same operations and is +// compiled into the same binary, so it inherits the same instrumentation. +class InstrumentCeiling { + public: + void Start() { + if (running_) return; + running_ = true; + stop_.store(false); + worker_ = std::thread([this]() { Loop(); }); + } + void Stop() { + if (!running_) return; + stop_.store(true); + worker_.join(); + running_ = false; + std::fputs("\n", stderr); + } + + // THE WORST boundary this run measured, and the two order statistics beside + // it that a reader of a red needs. Readable WHILE the sampler runs, which is + // what `mu_` is for: the assertions run in the middle of the case, and + // stopping the sampler to read it would make the denominator a statement + // about a window the numerator was not drawn from. + // + // A QUANTILE IS NOT ENOUGH AND THAT IS MEASURED, not assumed. The 99.9th + // percentile of the same population is 20x to 200x tighter and would be a far + // better instrument if it were safe. It is not: over 1440 leaf-record + // observations the worst `slack / p999` is **199.4**, against **0.645** for + // the maximum, and on the one run that redded the 30 ms constant p999 read + // 0.339 ms against a 67.55 ms boundary. The numerator is a maximum over two + // draws, so the denominator has to be a maximum too. + double ceiling_seconds() const { return Quantile(1, 1); } + double p999_seconds() const { return Quantile(999, 1000); } + double median_seconds() const { return Quantile(1, 2); } + size_t observations() const { + std::lock_guard lock(mu_); + return seconds_.size(); + } + + private: + double Quantile(size_t num, size_t den) const { + std::vector copy; + { + std::lock_guard lock(mu_); + copy = seconds_; + } + if (copy.empty()) return 0.0; + const size_t index = std::min(copy.size() - 1, (copy.size() * num) / den); + std::nth_element(copy.begin(), copy.begin() + static_cast(index), copy.end()); + return copy[index]; + } + + void Loop() { + while (!stop_.load()) { + const std::chrono::steady_clock::time_point before = std::chrono::steady_clock::now(); + // The three things `Open` and `Close` BOTH do. `SampleNow()` is the public + // entry point for the mutex plus the `/proc/self/statm` read; the flushed + // write is `EmitLocked`, which the live lane performs on every boundary on + // the shipped default. + vllm::multimodal::phase::SampleNow(); + std::fputs(".", stderr); + std::fflush(stderr); + const std::chrono::steady_clock::time_point after = std::chrono::steady_clock::now(); + const double elapsed = std::chrono::duration(after - before).count(); + { + std::lock_guard lock(mu_); + seconds_.push_back(elapsed); + } + // A CADENCE AND NOT A SPIN. One sample per millisecond draws tens of + // thousands of observations across a render and costs about 1% of one + // core; a spin would draw more and would change the contention it is + // measuring. + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + } + + std::atomic stop_{false}; + std::thread worker_; + bool running_ = false; + mutable std::mutex mu_; + std::vector seconds_; +}; + +// Process-wide: the bounds read the boundary cost the WHOLE run measured. +InstrumentCeiling& RunCeiling() { + static InstrumentCeiling ceiling; + return ceiling; +} + +// RAII, and the reason is not tidiness. A failed `REQUIRE` inside the case +// throws, and a sampler left running would keep writing to fd 2 while the LATER +// cases in this file `dup2` a capture file over it -- so one failure here would +// become three unrelated ones there, in a file whose whole subject is +// attributing a red. +struct CeilingSamplerGuard { + CeilingSamplerGuard() { RunCeiling().Start(); } + ~CeilingSamplerGuard() { RunCeiling().Stop(); } +}; + struct Carrying { std::string leaf; // the name that claims the seconds std::vector parts; // the nested sub-scopes sitting on the work @@ -3707,8 +3880,11 @@ void CheckCarryingPhase(const nlohmann::json& table, const Carrying& c) { // limit of the instrument at this geometry, and `### Owed out of W0` is where // the leaf-by-leaf version lives. // - // AND THE RESOLUTION IS NOW 30 ms RATHER THAN 0.25 ms, which is a real - // loss and is the price of a bound that survives a two-core contended runner. + // THE FOURTH SHAPE'S RESOLUTION WAS 30 ms RATHER THAN 0.25 ms, which was a + // real loss and was the price of a bound that survives a two-core contended + // runner. The fifth shape below buys most of it back by measuring the number + // instead of choosing it: on this box the same bound ran 0.54 ms to 419 ms, + // median 9.7 ms, because it is now a property of the RUN. // The 0.25 ms constant was inside its own honest distribution: at head // 6b48edb2c, unmutated, it reddened 3 of 33 runs on a shared // 20-core box and 2 of 10 under an added eight-way spin load. @@ -3784,7 +3960,11 @@ void CheckCarryingPhase(const nlohmann::json& table, const Carrying& c) { // contended. The same probe on a 20-core box read 94 us / 569 us / 895 us in // the same order, so the ordering is the build and not the box.) // - // A NORMALISED BOUND WOULD BE BETTER AND THERE IS NO NORMALISER. The old 0.95 + // A NORMALISED BOUND WOULD BE BETTER AND THERE IS NO NORMALISER IN THE TABLE. + // That is what the five candidates below establish, and it was read as though + // it said there is none anywhere. The fifth shape MEASURES one, from outside + // the table, by running the instrument's own boundary operations beside the + // render -- see the note above `InstrumentCeiling`. The old 0.95 // coverage floor tolerated instrumentation ACCIDENTALLY, by being a ratio of two // quantities that inflate together: when TSan makes everything 8x slower both // `covered` and `leaf_seconds` grow and the ratio survives. The span slack is an @@ -3810,7 +3990,10 @@ void CheckCarryingPhase(const nlohmann::json& table, const Carrying& c) { // instead, the raw quantity is tight: 20.1-93.6 us plain, 128-569 us under ASan, // 52.1-895 us under TSan -- about 4.7x, 4.4x and 17.2x. So configuration is the // variable that actually explains this quantity, and a per-configuration - // constant is the most stable bound available rather than a fallback. + // constant is the most stable bound available FROM THE TABLE rather than a + // fallback. Every candidate in that column is a ratio to something the render + // produced; none of them is a measurement of the instrument, which is what the + // fifth shape adds and why it is not a sixth row of this table. // // WHY NOT A SINGLE FLAT 4 ms COVERING ALL THREE: it would be ~30x slack on the // plain lane, where the honest value is ~121 us and the assertion currently @@ -3848,7 +4031,9 @@ void CheckCarryingPhase(const nlohmann::json& table, const Carrying& c) { // bounded -- and it is strictly tighter than the sum on every leaf that has // more than one record. // - // AND THERE IS ONE CONSTANT NOW, NOT ONE PER BUILD CONFIGURATION. The third + // AND THERE WAS ONE CONSTANT, NOT ONE PER BUILD CONFIGURATION -- until the + // fifth shape retired the constant itself. The paragraph is kept because its + // measurement is why a per-build number was not enough either. The third // shape carried 0.25 ms plain and 3 ms under either sanitizer, on the measured // ground that a sanitizer instruments the scope boundary this quantity IS. That // ground is still true and it is no longer the biggest term. Within the PLAIN @@ -3866,7 +4051,29 @@ void CheckCarryingPhase(const nlohmann::json& table, const Carrying& c) { // `||` form compiled on the TSan leg only because `||` short-circuits before // reaching it, which is the kind of green that means nothing. Anyone // reintroducing a per-configuration constant needs the nested form again. - const double kSpanSlackPerRecord = 0.03; // 30 ms, every build configuration + // AND THE BOUND IS FOUR OF THIS RUN'S OWN BOUNDARIES, NOT A CONSTANT. + // + // The head of a leaf record is `Open(leaf)`'s own sample and flushed line + // followed by `Open(sub)`'s acquisition of the same mutex; the tail is + // `Close(sub)`'s flushed line followed by `Close(leaf)`'s acquisition, sample + // and stamp. TWO boundary-sized operations, one at each end, both read out of + // `render_phase_log.cpp` in the note above `InstrumentCeiling` rather than + // assumed. `2 x ceiling` is the bound the structure gives; the four here + // carries one factor of two over it, and that factor is the whole of the + // judgement in this line. + // + // NOTHING ABOUT IT IS A BOX CONSTANT ANY MORE, which is what closes #1572 and + // #1576. On a quiet host the ceiling is tens of microseconds and this bound is + // far tighter than any number this case has carried; on a contended one it + // widens with the thing that made the quantity widen, and a record it can no + // longer resolve is REPORTED rather than decided. + const InstrumentCeiling& ceiling = RunCeiling(); + REQUIRE_MESSAGE(ceiling.observations() > 0, + "no `InstrumentCeiling` sampler ran before this check, so every bound below is " + "zero and this case would be asserting against an unmeasured number. The " + "`CeilingSamplerGuard` belongs in the case, around the renders"); + const double kBoundariesPerRecord = 4.0; + const double kSpanSlackPerRecord = kBoundariesPerRecord * ceiling.ceiling_seconds(); // THE COVERAGE FLOOR (2) IS WHAT HOLDS A SINGLE-RECORD LEAF THE CAP BELOW // SWALLOWS -- and ONLY a single-record leaf, which is the scope this line // used to leave out. Where the cap binds, this bound degrades to "head plus @@ -3997,7 +4204,10 @@ void CheckCarryingPhase(const nlohmann::json& table, const Carrying& c) { "phase"); } MESSAGE(" " << c.leaf << " span slack: " << span_checked << " of " << leaves.size() - << " leaf record(s) checked at " << kSpanSlackPerRecord << "s, " + << " leaf record(s) checked at " << kSpanSlackPerRecord << "s (" + << kBoundariesPerRecord << " x this run's worst boundary of " + << ceiling.ceiling_seconds() << "s, median " << ceiling.median_seconds() + << "s over " << ceiling.observations() << " samples), " << span_unresolvable << " below this instrument's resolution; worst " << worst_span_slack << "s"); @@ -4538,6 +4748,12 @@ void CheckRenderPhases(const nlohmann::json& table, } // namespace TEST_CASE("ltx2 video: the three carrying phases contain their work and the load keeps its order") { + // THE INSTRUMENT MEASURES ITSELF FIRST, and it does it FOR the load and both + // renders rather than beside them. Every wall-clock bound below is a multiple + // of what this sampler observes; the note above `InstrumentCeiling` is the + // argument, and the `REQUIRE` inside `CheckCarryingPhase` is what stops the + // bounds from silently becoming zero if this line is ever deleted. + const CeilingSamplerGuard sampling; Workspace ws; // `max_phase = 0` is a LOAD extra and is what keeps this on one recipe phase: // the default `distilled_two_stage` kind would need the latent spatial From 0108da9dfa3e4f48872656cc2f9fd8581abd7715 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 21:29:18 +0000 Subject: [PATCH 3/9] record(LTX25-DEVICE-RESIDENCY): the fifth and sixth shapes of the span-slack bound, both measured and both WITHDRAWN (#1727, #1728) Nothing in this change touches (1c). `tests/vllm/multimodal/test_ltx2_video.cpp` is byte-identical to `main` again, and #1572, #1576 and #1536 stay open. What lands is the record of two further shapes that were built, measured and withdrawn, so that the next person does not build them again. THE FIFTH SHAPE replaced the 30 ms constant with `4 x` the worst boundary a 1 kHz sampler observed across the whole case. Its derivation is right and a fresh review confirmed it operation-by-operation against `render_phase_log.cpp`. Its ESTIMATOR is not: that maximum is drawn from 3,764 to 27,975 draws while the numerator it bounds is a maximum over TWO, and maxima do not compare across sample sizes. Over 24 unmutated runs the sampler's MEDIAN operation measured 18.4-22.9 us, stable to 1.24x across loadavg 25 to 82, and its MAXIMUM 47x to 785x that median. One 200 ms descheduling of the sampler thread turned a real 20 ms un-named phase from red 9 of 9 into red 0 of 4, with the whole case GREEN 2 of 4. THE SIXTH SHAPE drew the denominator from the record's OWN window, stamping each draw with `PhaseLog::Elapsed()`. It detects better than anything before it: a 2 ms un-named phase reds (1c) 3 of 3 against 1 of 4 on the fifth shape and 0 of 4 at the constant, and the 20 ms defect with the sampler stall reds 3 of 3. And then it was validated the way this cluster's own rule demands -- 45 consecutive runs of the containment case, one pinned binary, an unmutated tree, loadavg 21.8 to 61.5 -- and it reddened 10 of 45. Five of those are the honest head-and-tail exceeding four of the record's own local worst boundary; the other five are a leaf whose longest record was not resolvable. The first mechanism is the one that decides it, and it is the same one `ltx25-phase-residue.md` `## Design` 3 records for the withdrawn residue bound: the part of a boundary the instrument cannot measure dilates faster under contention than the part it can, it sits in the numerator and in no denominator a sampler can build, and tightening the denominator onto the record's own window moves the bound into the honest distribution's tail. So both are withdrawn rather than repaired by a larger multiplier, which AGENTS.md refuses and #1668 names as the thing not to re-propose. The next traceable hypothesis is recorded and NOT attempted here: a defect is reproducible within a run and a scheduling event is not, so a bound that reds only when the same leaf exceeds it in BOTH of the fixture's renders would keep every detection above while dropping every red measured here, each of which hit one render and not the other. It owes its own red-first evidence, its own mutation set and its own 45-run validation. #1727 and #1728 are filed against the sampler as constraints on any re-proposal: two of its three terms are unfalsifiable by the gate, its estimator is unmeasured under the sanitizers and in the full-suite configuration where #1576's 171x swing was measured, and on a ROCm runner it would drive `hipMemGetInfo` at 1 kHz under the process-wide phase mutex. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/issue-index.md | 2 + .agents/specs/ltx25-device-residency.md | 198 +++++++++++++++++++ tests/vllm/multimodal/test_ltx2_video.cpp | 230 +--------------------- 3 files changed, 207 insertions(+), 223 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index f9eb147e3..28963a7d3 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -629,3 +629,5 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1691](https://github.com/mudler/vllm.cpp/issues/1691) | `DOCS-MODELS-HUMAN` | **`docs/QUICKSTART.md` said no container lane had ever been published, and three had.** The page carried `The container lanes have never been published, so no tag below resolves against the registry yet` and, on its `docker run` line, `The package carries only a stage tag today, which is a build artifact and not a lane`. Both were true when written and false on 2026-08-22: `ghcr.io/mudler/vllm.cpp` is public and carries `main-cpu`, `main-cuda` and `main-vulkan`, each a `linux/amd64` + `linux/arm64` manifest, published 07:30 UTC that day. `docs/guides/container-images.md` already documented the `:main-` tags, so the two pages disagreed. VERIFIED by execution rather than by reading the registry listing: `docker run --rm ghcr.io/mudler/vllm.cpp:main-cpu --version` answered `vllm.cpp 0.0.3 c-abi=23` at digest `sha256:7f88301ea282dad778748929e7aa6869d2418c8d295eef0e7900cca8310d06e5`, and the same image with a mounted `Qwen/Qwen3-0.6B` returned tokens through `/v1/completions` on host `mudler-ubuntu-box` (x86_64, Docker 29.1.2). The image also parses the `vllm_cpp` weight-residency document and echoes `mmap=on prefault=off expert_stream=on expert_stream_slots=4000`, which is what lets the Qwen3.8 2.4T page carry a container form of its recipe. FIXED IN FLOW: the false note is corrected, the `docker run` line names a tag that resolves, and the executed-row table gains its first real row. **This does NOT close [#1281](https://github.com/mudler/vllm.cpp/issues/1281)**: `:latest` still does not exist, `--model org/repo` is still blocked by [#1511](https://github.com/mudler/vllm.cpp/issues/1511) so the executed row mounts a local directory, and no GPU-lane row was run. The executed-row obligation stays owned by #1281 | bug | | [#1690](https://github.com/mudler/vllm.cpp/issues/1690) | `DOC-README-ABI-LANDING-SOURCE` | **`check-doc-checkpoint.py`'s `LANDING_SOURCE_FILES` omitted `include/vllm.h`, so a commit that bumps `VLLM_ABI_VERSION` could never repair the README claim it invalidated.** Found by the review of [PR #1655](https://github.com/mudler/vllm.cpp/pull/1655). The README `## Use it as a library (C API)` block quotes the ABI version out of the header; the header was in `USER_USAGE_FILES` (so an ABI change owes `docs/USAGE.md`) but not in `LANDING_SOURCE_FILES`, and the README rule refuses a claim change unaccompanied by a landing source. The claim could therefore be invalidated but not repaired by the same edit, which is how the README reached `VLLM_ABI_VERSION 21` against a header reading `23`, alongside a stale "46 exported functions" for a header declaring 47. That second half is stale by one rather than by six -- an earlier review figure of 51/52 swept in the `#define VLLM_API` visibility block and counted `vllm_*` identifiers that are typedefs and struct fields rather than exported functions -- so the case for deleting the count is that a live count of one file stored in another goes stale on any ABI addition, not that it is badly wrong. The set's own criterion already admitted it -- the checker's comment says every member is "something the README QUOTES" -- and the header was the only such source missing. FIXED IN FLOW: `include/vllm.h` is added to the set, red-before/green-after pinned by `test_the_c_abi_header_is_a_landing_source`, with `test_the_c_abi_header_permits_but_does_not_demand_readme` proving no new README obligation and the pre-existing no-class tests still green | bug | | [#1692](https://github.com/mudler/vllm.cpp/issues/1692) | `KERNEL-ACCEL-PROVIDER-DECLINE-EXACT` | **[#1584](https://github.com/mudler/vllm.cpp/issues/1584) is FIXED by this row, and its gate is CPU-only: both production call sites need a GPU and neither arm was executed.** The fix adds `GetOpFallbackUncounted`, sharing one body with `GetOpFallback` so resolution order, the reference-tier install, the drain and every throw stay identical and only the `fetch_add` is conditional -- in its existing position, BEFORE the "nothing below" check, because a decline that throws is still a decline. The two SHAPE-GATED providers that hoist the lookup into a function-local static switch to it: `cuda_attention_cross.cu` `BlockedFallback` and `metal_mlx_provider.mm` `MlxFallback`. **Candidate 1 of #1584 -- drop the count from `GetOpFallback` -- was REJECTED and measured rather than argued:** it edits five per-call sites (`vulkan_ops.cpp:950,1067,1488,1509`, `tenstorrent_ops.cpp:1341`), one already gated on an exact value at `test_vulkan_backend.cpp:2901` (`after.declines == before.declines + 1`), and its failure mode for a future caller is silent UNDER-counting, the Risk 4 the seam exists to expose. Net blast radius is ZERO backends, not the four #1584 estimated; exactly two providers change what they report, by one. #1555's `WarmDeclineOnce` workaround is REMOVED. **What this issue owns is the missing EXECUTION:** `test_ops_attention_cross` on CUDA (20 cases, ALL 20 SKIP on a CPU-only build -- 32 assertions, every one the skip guard, so that suite gives the change no coverage here), `test_metal_backend` on a `VLLM_CPP_MLX` build, and the `.agents/reachability.md` mutation on `BlockedFallback()` / `MlxFallback()`. The CUDA COMPILE is covered by the `-DVLLM_CPP_CUDA=ON` `vllm`-target job in `.github/workflows/ci.yml`; the Metal `.mm` file is compiled by NO job in this repository, because MLX needs `MLX_ROOT`. Red-before/green-after and the mutation table in [op-provider-decline-exact.md](specs/op-provider-decline-exact.md) | bug | +| [#1727](https://github.com/mudler/vllm.cpp/issues/1727) | `LTX25-DEVICE-RESIDENCY` | **(1c)'s boundary sampler carries three terms and the gate can falsify only one.** `tests/vllm/multimodal/test_ltx2_video.cpp`'s `InstrumentCeiling::Loop` times the process-wide mutex, `SampleNow()`'s `/proc/self/statm` read, and one flushed write standing for `EmitLocked`. A fresh review of [PR #1712](https://github.com/mudler/vllm.cpp/pull/1712) deleted the flushed write entirely -- the term the derivation argues corresponds to the live lane's per-boundary line -- and the case stayed GREEN 3 of 3 at `compile_status=0`, `assertions: 606 \| 606 passed \| 0 failed`, with the ceiling reading 3.24-9.19 ms against an unmutated range of 0.94-17.5 ms: nothing distinguishes the term's removal from run-to-run noise. That same measurement is what freed the write to be pointed at `/dev/null` instead of fd 2, where at 1 kHz it corrupted 27% of the `[render]` lines [#1413](https://github.com/mudler/vllm.cpp/issues/1413) exists for (158 occurrences in one capture, 116 beginning a line) -- so the finding paid for a repair while itself staying open. TWO further halves of the estimator are UNMEASURED rather than ungated. The SANITIZER lane: the third shape of this bound needed its own constant under ASan and TSan because a sanitizer instruments the scope boundary this quantity IS, and the sampler now inherits that automatically by compiling into the same binary -- but the sample count grows with the slowdown too, so the maximum is drawn from a deeper quantile, and nobody has run `sanitize-cpu` against the estimator (the review could not: a second full build tree does not fit at 93% disk). The FULL-SUITE regime: [#1576](https://github.com/mudler/vllm.cpp/issues/1576) measured this quantity at 3.223 ms inside the full 102-case `ctest` run against 18.8 us under a `-tc=` filter at comparable load, **171x**, and that swing is the whole reason the constant was declared unsafe -- yet both the row's 120-run validation and the review's 24-run one were taken almost entirely under `-tc=`. One full-binary run (containment case as #40 of 102) was indistinguishable from the filtered ones at ceiling 2.31 ms, bound 9.22 ms, worst slack 86 us, so the 171x did not reproduce at n=1. NOT FIXED IN FLOW: gating a term of the estimator is a second instrument over the first, and each unmeasured regime needs a build tree this box cannot hold beside the one it has. What would settle it: one `sanitize-cpu` build with the containment case run 40+ times, and one 40-run population through the FULL binary, both reported as distributions. Owned by [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `## Owed` | verification | +| [#1728](https://github.com/mudler/vllm.cpp/issues/1728) | `LTX25-DEVICE-RESIDENCY` | **HYPOTHESIS, filed unmeasured and labelled as such: on a ROCm runner (1c)'s boundary sampler would call `hipMemGetInfo` 1000 times a second under the process-wide phase mutex.** Raised by the fresh review of [PR #1712](https://github.com/mudler/vllm.cpp/pull/1712) on a CPU-only box with no GPU lease taken. `InstrumentCeiling` runs a thread calling `vllm::multimodal::phase::SampleNow()` once per millisecond for the whole of `ltx2 video: the three carrying phases contain their work and the load keeps its order` -- 3,764 to 27,975 draws per run, measured. `SampleNow()` takes the phase mutex and runs `SampleLocked()`, which invokes the installed `DeviceByteProbe`; `FixtureParams` sets `mp.device = 0` and `src/vllm/multimodal/ltx2_video.cpp` installs a probe calling `vt::Backend::DeviceMemoryInfo`. On the lanes the review could run this is harmless because `CudaBackend` does not override it and returns the -1 sentinel ([#1126](https://github.com/mudler/vllm.cpp/issues/1126)), but `src/vt/rocm/rocm_backend.hip:358` DOES override it. If the reading is right the consequence runs in two directions at once: the render slows because it contends for that mutex against a thread holding it across a driver round trip, and the sampler's own draws inflate, widening the very bound (1c) asserts against. What would settle it, cheaply: one lease on a ROCm device running that case and comparing the printed whole-case boundary median and maximum against this box's 18.4-22.9 us / 0.94-17.5 ms, plus the case wall. If it reproduces, the repair is a null device probe for the sampler's lifetime, or a device-counter cadence far below the boundary-timing cadence. NOT FIXED IN FLOW: the box that found it has no ROCm device and the session was CPU-only, so both reproduction and repair are unmeasured. Filed rather than left in a review transcript, because a 1 kHz driver call under a shared mutex is the kind of thing discovered as "the ROCm CI is slow" months later. Owned by [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `## Owed` | bug | diff --git a/.agents/specs/ltx25-device-residency.md b/.agents/specs/ltx25-device-residency.md index 71bd7a6ec..9a61921a3 100644 --- a/.agents/specs/ltx25-device-residency.md +++ b/.agents/specs/ltx25-device-residency.md @@ -667,6 +667,8 @@ its phase table lands, and W5 owes one when its wall is accepted. | [#1012](https://github.com/mudler/vllm.cpp/issues/1012) | O1 | owed | | [#1202](https://github.com/mudler/vllm.cpp/issues/1202) | deferred behind W1 | owed, with a number: 2.3% of one pass | | [#1439](https://github.com/mudler/vllm.cpp/issues/1439) | W0 (its own gate) | **REPAIRED but NOT CLOSED, by the `#1439 repaired` section below: the prologue is now named `load.open`, so the time is measured rather than tolerated, and the assertion is byte-for-byte unchanged. It stays OPEN because the repair was measured on a box at loadavg 24-31, where the render's wall is 7-10 s and BOTH arms sit near 99.96% — the flattering regime this issue itself identified. What is owed is one quiet-box run at fixture scale, in the 0.22-0.58 s regime where the floor actually bites. What follows is the record of why it was red.** **It was RED on `main` rather than on any branch.** `CHECK(leaves >= 0.95 * wall)` in `ltx2 video: a render through the ABI emits a phase table that SUMS to wall` is a RATIO, and the un-named residue is 4.80% to 6.32% of `wall` across twelve runs on one x86 box - so the 95% floor sits INSIDE the measurement's own range at the 64x64 / 9-frame fixture scale, and the case decides by coin flip, mostly red. With this lane's four files reverted so the binary is main at `89261c955`, six runs read 94.32%, 95.20%, 93.74%, 94.20%, 94.69%, 94.19%; the W0-live merge reads 93.82%, 93.68%, 94.34%, 94.62%, 94.39%, and 94.12% with `VLLM_RENDER_PROGRESS=0`, which exonerates the live emitter. Box contention is NOT the cause and main's one green disproves it: that run had `wall=0.579684s`, more than double the others, because the box was loaded - a SLOWER render passes. The tolerance was argued for the 21.004 B render, where the residue is a far smaller fraction of the wall. Naming the un-named time, or bounding `unaccounted_seconds` beside the ratio so the assertion says the same thing at both scales, is gate semantics and owes its own row | +| [#1727](https://github.com/mudler/vllm.cpp/issues/1727) | (1c), sixth shape | **owed, filed by the fresh review of PR #1712.** Two of the boundary sampler's three terms are unfalsifiable by the gate -- deleting the flushed write leaves the estimator inside its own scatter, green 3 of 3 -- and the estimator is unmeasured in two regimes that decide it: the sanitizer lanes, and the FULL 102-case binary rather than a `-tc=` filter, which is where #1576's 171x swing was measured | +| [#1728](https://github.com/mudler/vllm.cpp/issues/1728) | (1c), sixth shape | **owed, filed by the fresh review of PR #1712 as a HYPOTHESIS with no measurement behind it.** On a ROCm runner the 1 kHz boundary sampler would drive `hipMemGetInfo` through `SampleLocked`'s device probe, under the process-wide phase mutex, for the whole case. `CudaBackend` does not override `DeviceMemoryInfo` (#1126) so the CPU and CUDA lanes are unaffected; `src/vt/rocm/rocm_backend.hip:358` does. No GPU lease was taken | | [#1164](https://github.com/mudler/vllm.cpp/issues/1164) | W7 decision point | not owed here — owned by `ENG-CUDAGRAPH-DIFFUSION`; this row owns its unblock order | **[#1009](https://github.com/mudler/vllm.cpp/issues/1009) is not owed. It @@ -3142,3 +3144,199 @@ so a mutation that failed to build or failed to apply cannot read as a pass. also why the guard is RAII: a failing `REQUIRE` in this case would otherwise leave a thread writing to fd 2 while the later cases in the file `dup2` a capture file over it. + + +### The span-slack bound, FIFTH and SIXTH shapes: both MEASURED, both WITHDRAWN (#1572, #1576, #1536 stay open) + +**Nothing in this section lands.** (1c) on `main` is unchanged and still carries +the fourth shape's 30 ms constant. What follows is why two further shapes were +built, measured and withdrawn, so the next person does not build them again. + +| shape | what the denominator was | how it fails | measured | +|---|---|---|---| +| 4th (on `main`) | a flat 30 ms per leaf record | reds an unmutated tree under load, and cannot resolve `artifacts.frames` at all | [#1576](https://github.com/mudler/vllm.cpp/issues/1576), 171x on one binary | +| 5th | `4 x` the WORST boundary a 1 kHz sampler saw across the WHOLE case | one scheduling event on the sampler thread multiplies the bound by ~100 and silences a real 20 ms un-named phase | fresh review: defect reds 9/9 alone, 0/4 with one 200 ms sampler stall, case GREEN 2/4 | +| 6th | `4 x` the worst boundary inside the RECORD'S OWN window | too tight: the honest span slack's tail exceeds it, because the UN-instrumented remainder of a boundary is in the numerator and not in the denominator | **10 red in 45 consecutive runs** of the unmutated containment case, loadavg 21.8-61.5 | + +The fifth shape's derivation is right and a fresh review confirmed it +operation-by-operation against `render_phase_log.cpp`. **Its ESTIMATOR was +wrong, and the review broke it by execution rather than by argument.** + +#### The defect: an extreme-order statistic compared across sample sizes + +`ceiling_seconds()` was the maximum over every draw the sampler took across the +whole case -- **3,764 to 27,975 draws** per run -- while the numerator it bounds +is a maximum over **TWO**. Maxima do not compare across sample sizes. Over 24 +unmutated runs the reviewer measured the sampler's MEDIAN operation at +**18.4-22.9 us, stable to 1.24x across loadavg 25 to 82**, and its MAXIMUM at +**47x to 785x that median**. A single scheduling event on a thread that does +none of the render's work therefore multiplied the bound by about a hundred. + +The consequence is a silent green, staged and measured: + +| arm | what it stages | (1c) red | case verdict | +|---|---|---|---| +| `C20` | a 20 ms un-named phase in `decode.audio`'s head | 9 of 9 | red 9 of 9 | +| `C20D2` | the SAME defect plus ONE 200 ms descheduling of the sampler thread | **0 of 4** | **GREEN 2 of 4**, `assertions: 603 \| 603 passed \| 0 failed`, bound 0.800 s | + +A real un-named phase, no assertion fires, and the only thing that made it pass +is a scheduling event on a sleeping background thread. `V15D2` reproduces it on +the leaf where (1c) is the ONLY detector: `V15` reds (1c) 5 of 5 and `V15D2` +reds it 0 of 2. + +#### The repair is not a smaller multiplier + +Each draw is stamped with `PhaseLog::Elapsed()`, which is the very clock +`start_seconds` and `end_seconds` are on, and a leaf record's bound reads only +the draws that fall **inside that record**. The denominator is then drawn from +the same interval as the numerator. + +It is a **strict tightening**: a maximum over a subset is never larger than the +maximum over the whole, so no record's bound grew. Measured green on this box at +loadavg ~80, over both renders of the fixture: + +| leaf | record | bound, sixth shape | bound, fifth shape (whole-case) | flat constant it replaced | +|---|---|---|---|---| +| `artifacts.frames` | 4.33 ms | **71.9 us** | 6.1-20 ms | 30 ms -- **could never resolve this leaf at all** | +| `decode.audio` | 2.31 s | **0.601 ms** | 6.1 ms | 30 ms | +| `decode.video` | 0.224 s | **0.495 ms** | 6.1 ms | 30 ms | +| `denoise` | 7.92 s | **6.15 ms** | 6.1 ms | 30 ms | + +`artifacts.frames` is the leaf [#1470](https://github.com/mudler/vllm.cpp/issues/1470) +is about, and it is the first time any shape of this bound has been able to +resolve it. + +A preemption INSIDE a record still widens that record's bound, and that is +correct rather than a residual defect: if the box descheduled this thread by +200 ms inside this leaf then this leaf's own slack could honestly be 200 ms, and +(1c) is entitled to say it cannot resolve that. What it may no longer do is stay +silent about it -- see the next section. + +The same change retires two smaller defects the review found. The whole-case +ceiling is MONOTONE, so `CheckRenderPhases`'s third call re-checked render 1's +records against a bound **1.00x to 11.59x larger** than the first call's -- the +same records at two strictnesses in one run. And the bound and the number +printed beside it were two separate reads of a moving quantity, so the report +could disagree with the arithmetic it described. A per-record window fixes both +by construction. + +#### The skip path was a permanent report-only state, which `AGENTS.md` refuses + +A record is checked only when `record_seconds >= 8 x ceiling`, `span_checked` +and `span_unresolvable` were computed, printed and **asserted nowhere**, and the +lane that matters discards the printing: CI runs `ctest --output-on-failure`, so +on a green run nobody ever sees the line. The reviewer measured four +`CheckCarryingPhase` calls reporting `1 of 1`, `0 of 2`, `0 of 1`, `0 of 1` +while the case exited `Status: SUCCESS!` with a 20 ms un-named phase present. +`AGENTS.md` `## Gates` refuses that in as many words: *"Report exactly one result +for each applicable rule ... A permanent report-only state is not a result."* + +The assertion added is `longest_checked`: **(1c) must have resolved the LONGEST +record of each carrying leaf.** It names a record rather than a count or a share, +which is what keeps it free of a new constant. The longest record is the one that +can hide the most AND the one most likely to satisfy `record >= 8 x its own +window's worst boundary`, so the two orderings agree and demanding it is both the +strongest and the cheapest thing to demand. A count floor would be a constant; a +seconds share would be a constant; "the biggest one" is neither. + +A record too short to hold a single draw of a 1 kHz sampler is counted and +reported separately from a record the bound cannot resolve, because those are +different facts about the instrument. + +#### Two more findings, repaired without changing any bound + +* **The sampler's flushed write went to fd 2 at 1 kHz and corrupted the live + lane.** One capture held 158 `[render]` occurrences of which only **116 began + a line**, so `grep '^\[render\]'` silently lost 27% of them -- and #1413 is the + row that exists for that stream. The same review deleted the write entirely and + the estimator did not move (green 3 of 3, ceiling 3.24-9.19 ms against an + unmutated range of 0.94-17.5 ms), so the SINK is not what the term measures. + The write is kept and pointed at `/dev/null`. +* **`p999_seconds()` was dead**, called from nowhere, and would not have done what + its comment claimed: `Quantile(999, 1000)` clamps to `n - 1` for every + `n <= 1000`, so at the sample counts a short run produces it returned the + maximum. The accessor is deleted and the argument for rejecting a quantile is + kept. + +#### And the sixth shape was measured too, and it is WITHDRAWN + +The repair above was built, its whole mutation set was re-run against it, and it +detects better than anything before it: a **2 ms** un-named phase in +`decode.audio`'s head reds (1c) **3 of 3**, against 1 of 4 on the fifth shape and +0 of 4 at the 30 ms constant; a 15 ms one in `decode.video`'s head reds 3 of 3; +and the 20 ms defect with the 200 ms sampler stall that silenced the fifth shape +reds **3 of 3**. + +**And then it was validated the way this cluster's own rule demands, and it +failed.** 45 consecutive runs of the containment case, one pinned binary +(sha256 `242c7925...`), an unmutated tree, this 20-core box at loadavg +**21.78 to 61.52**, every run's counts recorded: + +``` +45 runs 35 green 10 RED (22%) +``` + +Two mechanisms, both honest: + +| assertion | reds | what happened | +|---|---:|---| +| `span_slack <= span_bound` | 5 | the honest head-and-tail exceeded `4 x` the record's own local worst boundary -- `decode.audio` 13.115 ms against 8.889 ms, `decode.video` 0.488 ms against 0.374 ms, `artifacts.frames` 70.6 us against 60.6 us | +| `longest_checked` | 9 | a leaf's longest record was not resolvable, most often `artifacts.frames` at about 1 ms with no draw of a 1 kHz sampler inside it | + +The first mechanism is the one that matters and it is the SAME one +[`ltx25-phase-residue.md`](ltx25-phase-residue.md) `## Design` 3 records for the +withdrawn residue bound: **the part of a boundary the instrument cannot measure +-- the `lock_guard` release, the `Close` return, the `Scope` destructor and +constructor, the call into `Open` up to its clock read -- dilates faster under +contention than the part it can.** It sits in the numerator and in no +denominator any sampler can build, so tightening the denominator onto the +record's own window moves the bound INTO the honest distribution's tail. Reads +against it are then decided by the scheduler, which is what all four previous +shapes were. + +The second mechanism was repaired mid-flight -- a record shorter than the +sampler's cadence now takes the worst of the draws bracketing it -- and the +first is not repairable by choosing a different denominator. + +**So the estimator is withdrawn and (1c) is left exactly as `main` carries it.** +Raising the multiplier would be repairing a red gate by widening it, which +`AGENTS.md` refuses and which [#1668](https://github.com/mudler/vllm.cpp/issues/1668) +names as the thing not to re-propose. Four shapes have now been tried and the +common factor is that all four compare a wall-clock quantity the scheduler owns +against a threshold the tree owns. + +#### The next traceable hypothesis, which this row does NOT take + +A defect is REPRODUCIBLE within a run and a scheduling event is not. The fixture +renders the same three carrying leaves TWICE, and `CheckRenderPhases` is called +for both passes. A bound that reds only when the SAME leaf exceeds it in BOTH +renders would keep every detection measured above -- the injected sleeps run on +every pass -- while a one-off preemption, which is what all ten reds above are, +hits one pass and not the other. In the five span-slack reds the failing leaf +exceeded the bound in exactly one render. + +That is a different assertion with a different failure mode and it owes its own +red-first evidence, its own mutation set, and its own 45-run validation. It is +recorded here rather than attempted, because attempting a fourth shape in the +same session that refuted the second and third is how this cluster got four +constants in the first place. + +#### What is still owed after this shape + +* **On this box the fifth shape did not buy back the resolution it claimed.** + The reviewer measured the bound looser than the 30 ms constant in **4 of 24 + runs**, and records resolvable at **212/408 (52.0%)** on the fifth shape + against **204/408 (50.0%)** at the constant -- 1.7x at the median, not the 25x + the section above claims. The sixth shape's per-record windows are what change + that; the fifth shape's "25x tighter on the quiet end" line is left standing + above as what was measured on ITS population and should be read against this + one. +* **The sanitizer lane is untested for this estimator.** The mechanism the third + shape's per-build constant existed for is inherited automatically, because the + sampler compiles into the same binary -- but the sample count also grows with + the slowdown. Nobody has run it. +* **The ROCm arm is a hypothesis, not a measurement.** `FixtureParams` sets + `mp.device = 0`, `ltx2_video.cpp` installs a `DeviceByteProbe`, and + `src/vt/rocm/rocm_backend.hip` implements `DeviceMemoryInfo` where + `CudaBackend` does not. On a ROCm runner this sampler would drive + `hipMemGetInfo` at 1 kHz under the phase mutex. No GPU lease was taken. diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 601046aa3..9443c00c2 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -30,13 +30,11 @@ #include #include #include -#include #include #include #include #include -#include #include #include @@ -3511,177 +3509,6 @@ std::vector LeafIntervals(const nlohmann::json& table, } // One carrying phase and the anchor that ties its name to its work. -// ─── WHAT ONE INSTRUMENT BOUNDARY COSTS, ON THE RUN THAT IS ASSERTING ─────── -// -// EVERY WALL-CLOCK TOLERANCE BELOW USED TO BE A COMPILE-TIME NUMBER, and each -// of the four this case has carried was afterwards found redding an UNMUTATED -// tree: the 0.95 coverage floor (#1439, #1494), 0.25 ms per leaf record (#1559, -// #1572), 3 ms under a sanitizer, and the 30 ms that replaced both (#1576). -// Each was calibrated against a sample of one box and each was wrong on the -// next one. -// -// The quantity all four bound is the wall a PHASE BOUNDARY costs, and that is -// not a property of this tree. It is a property of the machine, of the build -// configuration, and of what else held a core at that second: on ONE host, ONE -// build type and ONE binary it has measured 25.0 us and 10.032 ms with nothing -// mutated, a spread of 402x (#1559). #1576 measured 171x between two runs of one -// binary and said in its own text that a bigger constant "would only move the -// threshold at which the same ambiguity appears". That is what the 30 ms shape -// did: it widened the smallest swallow this case can see from about 0.5 ms to -// 60 ms and left a fixed budget with no allowance for load. -// -// SO THE NUMBER IS NOT CHOSEN HERE. IT IS MEASURED, ON THE RUN THAT ASSERTS. -// -// WHAT A BOUNDARY IS, READ OUT OF `render_phase_log.cpp` RATHER THAN GUESSED. -// `PhaseLog::Open` takes the process-wide mutex, stamps `o.start`, appends the -// open entry, calls `SampleLocked` -- one `/proc/self/statm` read folded into -// every live scope -- and, on the shipped default, writes and FLUSHES one line -// to stderr. `Close` takes the same mutex, calls `SampleLocked`, THEN stamps -// `r.end`, and flushes its own line. Both stamps sit inside that work, which is -// what puts the cost inside the leaf record and makes it the head and the tail -// that assertion (1c) below measures. -// -// One boundary is therefore one acquisition of that mutex, one sample, and one -// flushed write. This sampler times exactly those three, through the -// instrument's own public entry points, and keeps the WORST it saw. -// -// A MAXIMUM AND NOT A MEAN, AND THAT IS THE DESIGN RATHER THAN A DETAIL. -// The numerator of the bound below is itself a MAXIMUM over a small number of -// draws from this same distribution. A mean denominator under a maximum -// numerator is the shape `ltx25-phase-residue.md` `## Design` 3 measured at 4 -// reds in 45 runs -- worst ratio 4.115 against a bound of 1 -- and withdrew: -// under contention the un-instrumented remainder of a boundary dilates FASTER -// than the instrumented part, so that comparison has a heavy right tail rather -// than a shifted median, and a 20-run sample never sees it. Here the -// denominator is the MAXIMUM over tens of thousands of draws of the same -// operation taken across the same window, which is a tail quantity compared -// with a tail quantity. -// -// AND (2) BELOW IS DELIBERATELY NOT GIVEN THIS TERM. Adding `uncovered <= -// boundary allowance` to the coverage floor as a second arm was written, -// built and MEASURED here, and it is withdrawn: it turns mutation B -- -// `artifacts.frames.ppm` opened AFTER the `WriteFileBytes` loop instead of -// around it, which drops that leaf's coverage from 98% to 1.1% -- from a red -// into a green. The reason is that `artifacts.frames` is 0.2-7 ms on this -// fixture while one boundary on this host is 0.3-1.2 ms even pinned to two -// idle cores, so any allowance built from the boundary swamps the leaf. What -// the 0.50 floor there catches, and the load it reds under, are the same -// property of a leaf smaller than the instrument measuring it. #1470 owns -// that, and the repair is an anchor the writer's own seconds can be read -// against, not a threshold. -// -// AND IT IS SAMPLED ACROSS THE RENDER, NOT BESIDE IT. That is measured rather -// than assumed. A burst taken before and after each render covers about 40 ms of -// a case that runs for tens of seconds, and a contention spike inside a render -// is then absent from the denominator while being present in the numerator: with -// bursts alone the worst observed `slack / ceiling` was 3.96 over 161 leaf -// records. The sampler below runs FOR the render, which is what makes the -// denominator a statement about the window the numerator is drawn from. -// -// IT ALSO RETIRES THE PER-CONFIGURATION CONSTANT. A sanitizer instruments the -// scope boundary this quantity IS -- ASan checks a shadow byte on every access -// in it, TSan additionally keeps per-access happens-before state under a lock -- -// which is why the third shape needed 0.25 ms plain and 3 ms under a sanitizer, -// and why the nested `#if` that GCC's missing `__has_feature` made mandatory was -// in this file at all. This loop is built out of the same operations and is -// compiled into the same binary, so it inherits the same instrumentation. -class InstrumentCeiling { - public: - void Start() { - if (running_) return; - running_ = true; - stop_.store(false); - worker_ = std::thread([this]() { Loop(); }); - } - void Stop() { - if (!running_) return; - stop_.store(true); - worker_.join(); - running_ = false; - std::fputs("\n", stderr); - } - - // THE WORST boundary this run measured, and the two order statistics beside - // it that a reader of a red needs. Readable WHILE the sampler runs, which is - // what `mu_` is for: the assertions run in the middle of the case, and - // stopping the sampler to read it would make the denominator a statement - // about a window the numerator was not drawn from. - // - // A QUANTILE IS NOT ENOUGH AND THAT IS MEASURED, not assumed. The 99.9th - // percentile of the same population is 20x to 200x tighter and would be a far - // better instrument if it were safe. It is not: over 1440 leaf-record - // observations the worst `slack / p999` is **199.4**, against **0.645** for - // the maximum, and on the one run that redded the 30 ms constant p999 read - // 0.339 ms against a 67.55 ms boundary. The numerator is a maximum over two - // draws, so the denominator has to be a maximum too. - double ceiling_seconds() const { return Quantile(1, 1); } - double p999_seconds() const { return Quantile(999, 1000); } - double median_seconds() const { return Quantile(1, 2); } - size_t observations() const { - std::lock_guard lock(mu_); - return seconds_.size(); - } - - private: - double Quantile(size_t num, size_t den) const { - std::vector copy; - { - std::lock_guard lock(mu_); - copy = seconds_; - } - if (copy.empty()) return 0.0; - const size_t index = std::min(copy.size() - 1, (copy.size() * num) / den); - std::nth_element(copy.begin(), copy.begin() + static_cast(index), copy.end()); - return copy[index]; - } - - void Loop() { - while (!stop_.load()) { - const std::chrono::steady_clock::time_point before = std::chrono::steady_clock::now(); - // The three things `Open` and `Close` BOTH do. `SampleNow()` is the public - // entry point for the mutex plus the `/proc/self/statm` read; the flushed - // write is `EmitLocked`, which the live lane performs on every boundary on - // the shipped default. - vllm::multimodal::phase::SampleNow(); - std::fputs(".", stderr); - std::fflush(stderr); - const std::chrono::steady_clock::time_point after = std::chrono::steady_clock::now(); - const double elapsed = std::chrono::duration(after - before).count(); - { - std::lock_guard lock(mu_); - seconds_.push_back(elapsed); - } - // A CADENCE AND NOT A SPIN. One sample per millisecond draws tens of - // thousands of observations across a render and costs about 1% of one - // core; a spin would draw more and would change the contention it is - // measuring. - std::this_thread::sleep_for(std::chrono::milliseconds(1)); - } - } - - std::atomic stop_{false}; - std::thread worker_; - bool running_ = false; - mutable std::mutex mu_; - std::vector seconds_; -}; - -// Process-wide: the bounds read the boundary cost the WHOLE run measured. -InstrumentCeiling& RunCeiling() { - static InstrumentCeiling ceiling; - return ceiling; -} - -// RAII, and the reason is not tidiness. A failed `REQUIRE` inside the case -// throws, and a sampler left running would keep writing to fd 2 while the LATER -// cases in this file `dup2` a capture file over it -- so one failure here would -// become three unrelated ones there, in a file whose whole subject is -// attributing a red. -struct CeilingSamplerGuard { - CeilingSamplerGuard() { RunCeiling().Start(); } - ~CeilingSamplerGuard() { RunCeiling().Stop(); } -}; - struct Carrying { std::string leaf; // the name that claims the seconds std::vector parts; // the nested sub-scopes sitting on the work @@ -3880,11 +3707,8 @@ void CheckCarryingPhase(const nlohmann::json& table, const Carrying& c) { // limit of the instrument at this geometry, and `### Owed out of W0` is where // the leaf-by-leaf version lives. // - // THE FOURTH SHAPE'S RESOLUTION WAS 30 ms RATHER THAN 0.25 ms, which was a - // real loss and was the price of a bound that survives a two-core contended - // runner. The fifth shape below buys most of it back by measuring the number - // instead of choosing it: on this box the same bound ran 0.54 ms to 419 ms, - // median 9.7 ms, because it is now a property of the RUN. + // AND THE RESOLUTION IS NOW 30 ms RATHER THAN 0.25 ms, which is a real + // loss and is the price of a bound that survives a two-core contended runner. // The 0.25 ms constant was inside its own honest distribution: at head // 6b48edb2c, unmutated, it reddened 3 of 33 runs on a shared // 20-core box and 2 of 10 under an added eight-way spin load. @@ -3960,11 +3784,7 @@ void CheckCarryingPhase(const nlohmann::json& table, const Carrying& c) { // contended. The same probe on a 20-core box read 94 us / 569 us / 895 us in // the same order, so the ordering is the build and not the box.) // - // A NORMALISED BOUND WOULD BE BETTER AND THERE IS NO NORMALISER IN THE TABLE. - // That is what the five candidates below establish, and it was read as though - // it said there is none anywhere. The fifth shape MEASURES one, from outside - // the table, by running the instrument's own boundary operations beside the - // render -- see the note above `InstrumentCeiling`. The old 0.95 + // A NORMALISED BOUND WOULD BE BETTER AND THERE IS NO NORMALISER. The old 0.95 // coverage floor tolerated instrumentation ACCIDENTALLY, by being a ratio of two // quantities that inflate together: when TSan makes everything 8x slower both // `covered` and `leaf_seconds` grow and the ratio survives. The span slack is an @@ -3990,10 +3810,7 @@ void CheckCarryingPhase(const nlohmann::json& table, const Carrying& c) { // instead, the raw quantity is tight: 20.1-93.6 us plain, 128-569 us under ASan, // 52.1-895 us under TSan -- about 4.7x, 4.4x and 17.2x. So configuration is the // variable that actually explains this quantity, and a per-configuration - // constant is the most stable bound available FROM THE TABLE rather than a - // fallback. Every candidate in that column is a ratio to something the render - // produced; none of them is a measurement of the instrument, which is what the - // fifth shape adds and why it is not a sixth row of this table. + // constant is the most stable bound available rather than a fallback. // // WHY NOT A SINGLE FLAT 4 ms COVERING ALL THREE: it would be ~30x slack on the // plain lane, where the honest value is ~121 us and the assertion currently @@ -4031,9 +3848,7 @@ void CheckCarryingPhase(const nlohmann::json& table, const Carrying& c) { // bounded -- and it is strictly tighter than the sum on every leaf that has // more than one record. // - // AND THERE WAS ONE CONSTANT, NOT ONE PER BUILD CONFIGURATION -- until the - // fifth shape retired the constant itself. The paragraph is kept because its - // measurement is why a per-build number was not enough either. The third + // AND THERE IS ONE CONSTANT NOW, NOT ONE PER BUILD CONFIGURATION. The third // shape carried 0.25 ms plain and 3 ms under either sanitizer, on the measured // ground that a sanitizer instruments the scope boundary this quantity IS. That // ground is still true and it is no longer the biggest term. Within the PLAIN @@ -4051,29 +3866,7 @@ void CheckCarryingPhase(const nlohmann::json& table, const Carrying& c) { // `||` form compiled on the TSan leg only because `||` short-circuits before // reaching it, which is the kind of green that means nothing. Anyone // reintroducing a per-configuration constant needs the nested form again. - // AND THE BOUND IS FOUR OF THIS RUN'S OWN BOUNDARIES, NOT A CONSTANT. - // - // The head of a leaf record is `Open(leaf)`'s own sample and flushed line - // followed by `Open(sub)`'s acquisition of the same mutex; the tail is - // `Close(sub)`'s flushed line followed by `Close(leaf)`'s acquisition, sample - // and stamp. TWO boundary-sized operations, one at each end, both read out of - // `render_phase_log.cpp` in the note above `InstrumentCeiling` rather than - // assumed. `2 x ceiling` is the bound the structure gives; the four here - // carries one factor of two over it, and that factor is the whole of the - // judgement in this line. - // - // NOTHING ABOUT IT IS A BOX CONSTANT ANY MORE, which is what closes #1572 and - // #1576. On a quiet host the ceiling is tens of microseconds and this bound is - // far tighter than any number this case has carried; on a contended one it - // widens with the thing that made the quantity widen, and a record it can no - // longer resolve is REPORTED rather than decided. - const InstrumentCeiling& ceiling = RunCeiling(); - REQUIRE_MESSAGE(ceiling.observations() > 0, - "no `InstrumentCeiling` sampler ran before this check, so every bound below is " - "zero and this case would be asserting against an unmeasured number. The " - "`CeilingSamplerGuard` belongs in the case, around the renders"); - const double kBoundariesPerRecord = 4.0; - const double kSpanSlackPerRecord = kBoundariesPerRecord * ceiling.ceiling_seconds(); + const double kSpanSlackPerRecord = 0.03; // 30 ms, every build configuration // THE COVERAGE FLOOR (2) IS WHAT HOLDS A SINGLE-RECORD LEAF THE CAP BELOW // SWALLOWS -- and ONLY a single-record leaf, which is the scope this line // used to leave out. Where the cap binds, this bound degrades to "head plus @@ -4204,10 +3997,7 @@ void CheckCarryingPhase(const nlohmann::json& table, const Carrying& c) { "phase"); } MESSAGE(" " << c.leaf << " span slack: " << span_checked << " of " << leaves.size() - << " leaf record(s) checked at " << kSpanSlackPerRecord << "s (" - << kBoundariesPerRecord << " x this run's worst boundary of " - << ceiling.ceiling_seconds() << "s, median " << ceiling.median_seconds() - << "s over " << ceiling.observations() << " samples), " + << " leaf record(s) checked at " << kSpanSlackPerRecord << "s, " << span_unresolvable << " below this instrument's resolution; worst " << worst_span_slack << "s"); @@ -4748,12 +4538,6 @@ void CheckRenderPhases(const nlohmann::json& table, } // namespace TEST_CASE("ltx2 video: the three carrying phases contain their work and the load keeps its order") { - // THE INSTRUMENT MEASURES ITSELF FIRST, and it does it FOR the load and both - // renders rather than beside them. Every wall-clock bound below is a multiple - // of what this sampler observes; the note above `InstrumentCeiling` is the - // argument, and the `REQUIRE` inside `CheckCarryingPhase` is what stops the - // bounds from silently becoming zero if this line is ever deleted. - const CeilingSamplerGuard sampling; Workspace ws; // `max_phase = 0` is a LOAD extra and is what keeps this on one recipe phase: // the default `distilled_two_stage` kind would need the latent spatial From ee286b0f43d1951dc5b2508373f7b5a4eb9980d8 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 21:30:09 +0000 Subject: [PATCH 4/9] record(LTX25-DEVICE-RESIDENCY): #1572 is not discharged, and the residue spec said it was The row's own claim, written before the fifth shape was reviewed. Both the fifth and the sixth shape of that bound are now measured and withdrawn, so #1572 stays open and the cross-reference has to say so rather than pointing a reader at a discharge that did not happen. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-phase-residue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/specs/ltx25-phase-residue.md b/.agents/specs/ltx25-phase-residue.md index 1c5722406..0743f3944 100644 --- a/.agents/specs/ltx25-phase-residue.md +++ b/.agents/specs/ltx25-phase-residue.md @@ -234,7 +234,7 @@ once, at `tests/vllm/multimodal/test_ltx2_video.cpp:4325`, asking for it. | [#1569](https://github.com/mudler/vllm.cpp/issues/1569) | a gate on `WriteJson`'s clock ORDERING, **measured green under its own mutation**. Restoring the old order left the conservation case GREEN 10 of 10, at `wall 0.0608987s, unaccounted 0.000534223s, table charge 0.000301655s`, because the copy and sort of a three-record table are nanoseconds. Gating it needs a table with enough records for the sort to be measurable | | [#1570](https://github.com/mudler/vllm.cpp/issues/1570) | an upper bound on the instrument's own share of a leaf. `uncovered <= 2 * leaf_instrument` is stricter than the floor it replaces only while `leaf_instrument` stays small, and nothing bounds it. Moving the DiT `Tick` out of `Evaluate` would charge ~110 flushed writes to `denoise` and widen the gate while printing a small number | | [#1571](https://github.com/mudler/vllm.cpp/issues/1571) | a per-gap decomposition IN the emitted table. The 92% region above was found with a scratch script; a reader of `phase-log.json` still cannot see it without one, and the same investigation will be re-derived the next time the residue moves | -| [#1572](https://github.com/mudler/vllm.cpp/issues/1572) | assertion (1c)'s span slack reds intermittently on `main` — `decode.video` at `0.00256913` against a `0.00075` bound, 3.4x. Pre-existing from `6b48edb2c` and not this row's. **DISCHARGED** by `LTX25-DEVICE-RESIDENCY`'s fifth shape of that bound, which replaces the constant with `4 x` the boundary cost the run itself measured: `.agents/specs/ltx25-device-residency.md` `### The span-slack bound, fifth shape` | +| [#1572](https://github.com/mudler/vllm.cpp/issues/1572) | assertion (1c)'s span slack reds intermittently on `main` — `decode.video` at `0.00256913` against a `0.00075` bound, 3.4x. Pre-existing from `6b48edb2c` and not this row's. **STILL OPEN, and two further shapes of that bound are now MEASURED SHUT.** `LTX25-DEVICE-RESIDENCY` built a fifth shape (`4 x` the worst boundary a 1 kHz sampler saw across the whole case) and a sixth (`4 x` the worst inside the record's OWN window) and withdrew both: the fifth lets one 200 ms descheduling of the sampler thread turn a real 20 ms un-named phase from red 9 of 9 into a GREEN case, and the sixth reds an unmutated tree 10 times in 45 consecutive runs at loadavg 21.8-61.5. See `.agents/specs/ltx25-device-residency.md` `### The span-slack bound, FIFTH and SIXTH shapes` for both distributions and for the one hypothesis that has not been tried | | [#1619](https://github.com/mudler/vllm.cpp/issues/1619) | **the `merge=union` driver duplicates a row, MEASURED on this row's own merges.** Both sides appended before the same trailing anchor rather than at the true end, so the driver concatenated two regions that each carried `#1546` and the resolved index held it TWICE, byte-identical, at 538 lines where the correct union is 537. `git merge-tree` called that merge clean and `check-issue-index-append-only.py` passed it, because a duplicate is an ADDITION and that checker only collects removals. `check-agent-record.py` did NOT pass it -- a claim #1556's spec made and this row REFUTED by reproduction: regenerating the raw driver output and running that same tree's checker returns rc=1 with `issue #1546 listed twice`, and the refusal has existed since `8dd6508da` (2026-08-09), before the merge. So the blind gate is exactly one checker, not two, and the gap is narrower than #1556 recorded. The de-duplication half is CONDITIONAL, and the condition is what #1556's spec omitted: the checker reds a repair only when the DUPLICATE IS ALREADY IN THE BASE. Measured at three pairings -- `--base e2a9e035d` against the real canonical 537-line file rc=0, against a synthetic 537 rc=0, and `--base --head <537 de-dup>` rc=1. It diffs `merge-base..HEAD`, so when the base predates the duplicate the addition and the removal CANCEL and it passes. Since `origin/main` is preflight's base, and is the shape this branch used, the gate does NOT red someone who repairs driver output before committing it -- only someone repairing a corruption that already landed. The same range property is why relocating a base-reachable row DOES red it: moving row `#168` to the end gives rc=1 and a `removed:` line naming it. So "de-duplicating in place FAILS the checker", as #1556's spec put it, is false unqualified and true once the duplicate is base-reachable. #1556's spec added that the same driver dropped `#838` on a later re-merge, making this a recurring class; that is WITHDRAWN as unreproducible. Re-running `git merge-file --union` at every later merge where `#838` was on a side leaves it present in all of them, and `git log -S` finds it absent from no committed state -- mechanically a union driver cannot drop a line that is an addition on one side. If it ever went missing, that points at a wholesale take-ours resolution rather than at the driver | | [#1439](https://github.com/mudler/vllm.cpp/issues/1439) | **NOT closed by this row, and it must not be.** See `## Risks and decisions` D4 | | [#1470](https://github.com/mudler/vllm.cpp/issues/1470) | `test_ltx2_video` false-redded once on `main` under load and the failing case's identity was never captured. Untouched by THIS row, and **the identity and the rate are now measured** by `LTX25-DEVICE-RESIDENCY`: 1 red in 120 runs of the containment case at loadavg 40-155, on `artifacts.frames` render 2, where a 67.55 ms descheduling between `ppm_phase.Close()` and the leaf's destructor left 67.55 ms of a 74.87 ms leaf uncovered. It reds `covered >= 0.50 * leaf_seconds`. The same row also measured, and WITHDREW, the obvious repair: an instrument-relative second arm on that floor makes mutation `B-empty-ppm` — the writer's anchor opened after the write loop instead of around it, coverage 98% to 1.1% — pass. `artifacts.frames` is 0.2-7 ms on this fixture and one boundary on that host is 0.3-1.2 ms pinned to two idle cores, so no allowance built from the boundary is smaller than the leaf. The repair is an anchor, not a threshold | From ca40eec3ede9770619124b09dea5cec1e14da7a1 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 21:30:49 +0000 Subject: [PATCH 5/9] record(LTX25-DEVICE-RESIDENCY): the two filed issues are constraints on a re-proposal, not live defects Both were written while the sampler-based estimator was expected to land. It does not: both shapes of it are measured and withdrawn, and `(1c)` on `main` is unchanged. The two index rows now say so in their first sentence, so a reader does not go looking in `test_ltx2_video.cpp` for an `InstrumentCeiling` that is not there. What each one records is still owed by anyone who rebuilds that estimator. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/issue-index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index a107669ac..458c34a52 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -629,8 +629,8 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1691](https://github.com/mudler/vllm.cpp/issues/1691) | `DOCS-MODELS-HUMAN` | **`docs/QUICKSTART.md` said no container lane had ever been published, and three had.** The page carried `The container lanes have never been published, so no tag below resolves against the registry yet` and, on its `docker run` line, `The package carries only a stage tag today, which is a build artifact and not a lane`. Both were true when written and false on 2026-08-22: `ghcr.io/mudler/vllm.cpp` is public and carries `main-cpu`, `main-cuda` and `main-vulkan`, each a `linux/amd64` + `linux/arm64` manifest, published 07:30 UTC that day. `docs/guides/container-images.md` already documented the `:main-` tags, so the two pages disagreed. VERIFIED by execution rather than by reading the registry listing: `docker run --rm ghcr.io/mudler/vllm.cpp:main-cpu --version` answered `vllm.cpp 0.0.3 c-abi=23` at digest `sha256:7f88301ea282dad778748929e7aa6869d2418c8d295eef0e7900cca8310d06e5`, and the same image with a mounted `Qwen/Qwen3-0.6B` returned tokens through `/v1/completions` on host `mudler-ubuntu-box` (x86_64, Docker 29.1.2). The image also parses the `vllm_cpp` weight-residency document and echoes `mmap=on prefault=off expert_stream=on expert_stream_slots=4000`, which is what lets the Qwen3.8 2.4T page carry a container form of its recipe. FIXED IN FLOW: the false note is corrected, the `docker run` line names a tag that resolves, and the executed-row table gains its first real row. **This does NOT close [#1281](https://github.com/mudler/vllm.cpp/issues/1281)**: `:latest` still does not exist, `--model org/repo` is still blocked by [#1511](https://github.com/mudler/vllm.cpp/issues/1511) so the executed row mounts a local directory, and no GPU-lane row was run. The executed-row obligation stays owned by #1281 | bug | | [#1690](https://github.com/mudler/vllm.cpp/issues/1690) | `DOC-README-ABI-LANDING-SOURCE` | **`check-doc-checkpoint.py`'s `LANDING_SOURCE_FILES` omitted `include/vllm.h`, so a commit that bumps `VLLM_ABI_VERSION` could never repair the README claim it invalidated.** Found by the review of [PR #1655](https://github.com/mudler/vllm.cpp/pull/1655). The README `## Use it as a library (C API)` block quotes the ABI version out of the header; the header was in `USER_USAGE_FILES` (so an ABI change owes `docs/USAGE.md`) but not in `LANDING_SOURCE_FILES`, and the README rule refuses a claim change unaccompanied by a landing source. The claim could therefore be invalidated but not repaired by the same edit, which is how the README reached `VLLM_ABI_VERSION 21` against a header reading `23`, alongside a stale "46 exported functions" for a header declaring 47. That second half is stale by one rather than by six -- an earlier review figure of 51/52 swept in the `#define VLLM_API` visibility block and counted `vllm_*` identifiers that are typedefs and struct fields rather than exported functions -- so the case for deleting the count is that a live count of one file stored in another goes stale on any ABI addition, not that it is badly wrong. The set's own criterion already admitted it -- the checker's comment says every member is "something the README QUOTES" -- and the header was the only such source missing. FIXED IN FLOW: `include/vllm.h` is added to the set, red-before/green-after pinned by `test_the_c_abi_header_is_a_landing_source`, with `test_the_c_abi_header_permits_but_does_not_demand_readme` proving no new README obligation and the pre-existing no-class tests still green | bug | | [#1692](https://github.com/mudler/vllm.cpp/issues/1692) | `KERNEL-ACCEL-PROVIDER-DECLINE-EXACT` | **[#1584](https://github.com/mudler/vllm.cpp/issues/1584) is FIXED by this row, and its gate is CPU-only: both production call sites need a GPU and neither arm was executed.** The fix adds `GetOpFallbackUncounted`, sharing one body with `GetOpFallback` so resolution order, the reference-tier install, the drain and every throw stay identical and only the `fetch_add` is conditional -- in its existing position, BEFORE the "nothing below" check, because a decline that throws is still a decline. The two SHAPE-GATED providers that hoist the lookup into a function-local static switch to it: `cuda_attention_cross.cu` `BlockedFallback` and `metal_mlx_provider.mm` `MlxFallback`. **Candidate 1 of #1584 -- drop the count from `GetOpFallback` -- was REJECTED and measured rather than argued:** it edits five per-call sites (`vulkan_ops.cpp:950,1067,1488,1509`, `tenstorrent_ops.cpp:1341`), one already gated on an exact value at `test_vulkan_backend.cpp:2901` (`after.declines == before.declines + 1`), and its failure mode for a future caller is silent UNDER-counting, the Risk 4 the seam exists to expose. Net blast radius is ZERO backends, not the four #1584 estimated; exactly two providers change what they report, by one. #1555's `WarmDeclineOnce` workaround is REMOVED. **What this issue owns is the missing EXECUTION:** `test_ops_attention_cross` on CUDA (20 cases, ALL 20 SKIP on a CPU-only build -- 32 assertions, every one the skip guard, so that suite gives the change no coverage here), `test_metal_backend` on a `VLLM_CPP_MLX` build, and the `.agents/reachability.md` mutation on `BlockedFallback()` / `MlxFallback()`. The CUDA COMPILE is covered by the `-DVLLM_CPP_CUDA=ON` `vllm`-target job in `.github/workflows/ci.yml`; the Metal `.mm` file is compiled by NO job in this repository, because MLX needs `MLX_ROOT`. Red-before/green-after and the mutation table in [op-provider-decline-exact.md](specs/op-provider-decline-exact.md) | bug | -| [#1727](https://github.com/mudler/vllm.cpp/issues/1727) | `LTX25-DEVICE-RESIDENCY` | **(1c)'s boundary sampler carries three terms and the gate can falsify only one.** `tests/vllm/multimodal/test_ltx2_video.cpp`'s `InstrumentCeiling::Loop` times the process-wide mutex, `SampleNow()`'s `/proc/self/statm` read, and one flushed write standing for `EmitLocked`. A fresh review of [PR #1712](https://github.com/mudler/vllm.cpp/pull/1712) deleted the flushed write entirely -- the term the derivation argues corresponds to the live lane's per-boundary line -- and the case stayed GREEN 3 of 3 at `compile_status=0`, `assertions: 606 \| 606 passed \| 0 failed`, with the ceiling reading 3.24-9.19 ms against an unmutated range of 0.94-17.5 ms: nothing distinguishes the term's removal from run-to-run noise. That same measurement is what freed the write to be pointed at `/dev/null` instead of fd 2, where at 1 kHz it corrupted 27% of the `[render]` lines [#1413](https://github.com/mudler/vllm.cpp/issues/1413) exists for (158 occurrences in one capture, 116 beginning a line) -- so the finding paid for a repair while itself staying open. TWO further halves of the estimator are UNMEASURED rather than ungated. The SANITIZER lane: the third shape of this bound needed its own constant under ASan and TSan because a sanitizer instruments the scope boundary this quantity IS, and the sampler now inherits that automatically by compiling into the same binary -- but the sample count grows with the slowdown too, so the maximum is drawn from a deeper quantile, and nobody has run `sanitize-cpu` against the estimator (the review could not: a second full build tree does not fit at 93% disk). The FULL-SUITE regime: [#1576](https://github.com/mudler/vllm.cpp/issues/1576) measured this quantity at 3.223 ms inside the full 102-case `ctest` run against 18.8 us under a `-tc=` filter at comparable load, **171x**, and that swing is the whole reason the constant was declared unsafe -- yet both the row's 120-run validation and the review's 24-run one were taken almost entirely under `-tc=`. One full-binary run (containment case as #40 of 102) was indistinguishable from the filtered ones at ceiling 2.31 ms, bound 9.22 ms, worst slack 86 us, so the 171x did not reproduce at n=1. NOT FIXED IN FLOW: gating a term of the estimator is a second instrument over the first, and each unmeasured regime needs a build tree this box cannot hold beside the one it has. What would settle it: one `sanitize-cpu` build with the containment case run 40+ times, and one 40-run population through the FULL binary, both reported as distributions. Owned by [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `## Owed` | verification | -| [#1728](https://github.com/mudler/vllm.cpp/issues/1728) | `LTX25-DEVICE-RESIDENCY` | **HYPOTHESIS, filed unmeasured and labelled as such: on a ROCm runner (1c)'s boundary sampler would call `hipMemGetInfo` 1000 times a second under the process-wide phase mutex.** Raised by the fresh review of [PR #1712](https://github.com/mudler/vllm.cpp/pull/1712) on a CPU-only box with no GPU lease taken. `InstrumentCeiling` runs a thread calling `vllm::multimodal::phase::SampleNow()` once per millisecond for the whole of `ltx2 video: the three carrying phases contain their work and the load keeps its order` -- 3,764 to 27,975 draws per run, measured. `SampleNow()` takes the phase mutex and runs `SampleLocked()`, which invokes the installed `DeviceByteProbe`; `FixtureParams` sets `mp.device = 0` and `src/vllm/multimodal/ltx2_video.cpp` installs a probe calling `vt::Backend::DeviceMemoryInfo`. On the lanes the review could run this is harmless because `CudaBackend` does not override it and returns the -1 sentinel ([#1126](https://github.com/mudler/vllm.cpp/issues/1126)), but `src/vt/rocm/rocm_backend.hip:358` DOES override it. If the reading is right the consequence runs in two directions at once: the render slows because it contends for that mutex against a thread holding it across a driver round trip, and the sampler's own draws inflate, widening the very bound (1c) asserts against. What would settle it, cheaply: one lease on a ROCm device running that case and comparing the printed whole-case boundary median and maximum against this box's 18.4-22.9 us / 0.94-17.5 ms, plus the case wall. If it reproduces, the repair is a null device probe for the sampler's lifetime, or a device-counter cadence far below the boundary-timing cadence. NOT FIXED IN FLOW: the box that found it has no ROCm device and the session was CPU-only, so both reproduction and repair are unmeasured. Filed rather than left in a review transcript, because a 1 kHz driver call under a shared mutex is the kind of thing discovered as "the ROCm CI is slow" months later. Owned by [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `## Owed` | bug | +| [#1727](https://github.com/mudler/vllm.cpp/issues/1727) | `LTX25-DEVICE-RESIDENCY` | **CONSTRAINT ON ANY RE-PROPOSAL of a sampler-based span-slack denominator, which is NOT on `main`: the boundary sampler carries three terms and a gate can falsify only one.** The estimator this row built was measured and WITHDRAWN -- see [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `### The span-slack bound, FIFTH and SIXTH shapes` -- so what follows is what anyone rebuilding it already owes. `InstrumentCeiling::Loop` timed the process-wide mutex, `SampleNow()`'s `/proc/self/statm` read, and one flushed write standing for `EmitLocked`. A fresh review of [PR #1712](https://github.com/mudler/vllm.cpp/pull/1712) deleted the flushed write entirely -- the term the derivation argues corresponds to the live lane's per-boundary line -- and the case stayed GREEN 3 of 3 at `compile_status=0`, `assertions: 606 \| 606 passed \| 0 failed`, with the ceiling reading 3.24-9.19 ms against an unmutated range of 0.94-17.5 ms: nothing distinguishes the term's removal from run-to-run noise. That same measurement is what freed the write to be pointed at `/dev/null` instead of fd 2, where at 1 kHz it corrupted 27% of the `[render]` lines [#1413](https://github.com/mudler/vllm.cpp/issues/1413) exists for (158 occurrences in one capture, 116 beginning a line) -- so the finding paid for a repair while itself staying open. TWO further halves of the estimator are UNMEASURED rather than ungated. The SANITIZER lane: the third shape of this bound needed its own constant under ASan and TSan because a sanitizer instruments the scope boundary this quantity IS, and the sampler now inherits that automatically by compiling into the same binary -- but the sample count grows with the slowdown too, so the maximum is drawn from a deeper quantile, and nobody has run `sanitize-cpu` against the estimator (the review could not: a second full build tree does not fit at 93% disk). The FULL-SUITE regime: [#1576](https://github.com/mudler/vllm.cpp/issues/1576) measured this quantity at 3.223 ms inside the full 102-case `ctest` run against 18.8 us under a `-tc=` filter at comparable load, **171x**, and that swing is the whole reason the constant was declared unsafe -- yet both the row's 120-run validation and the review's 24-run one were taken almost entirely under `-tc=`. One full-binary run (containment case as #40 of 102) was indistinguishable from the filtered ones at ceiling 2.31 ms, bound 9.22 ms, worst slack 86 us, so the 171x did not reproduce at n=1. NOT FIXED IN FLOW: gating a term of the estimator is a second instrument over the first, and each unmeasured regime needs a build tree this box cannot hold beside the one it has. What would settle it: one `sanitize-cpu` build with the containment case run 40+ times, and one 40-run population through the FULL binary, both reported as distributions. Owned by [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `## Owed` | verification | +| [#1728](https://github.com/mudler/vllm.cpp/issues/1728) | `LTX25-DEVICE-RESIDENCY` | **HYPOTHESIS, filed unmeasured and labelled as such, and a CONSTRAINT ON ANY RE-PROPOSAL rather than a live defect, because the sampler it is about was measured and WITHDRAWN and is not on `main`: on a ROCm runner a 1 kHz boundary sampler inside `(1c)` would call `hipMemGetInfo` 1000 times a second under the process-wide phase mutex.** Raised by the fresh review of [PR #1712](https://github.com/mudler/vllm.cpp/pull/1712) on a CPU-only box with no GPU lease taken. `InstrumentCeiling` runs a thread calling `vllm::multimodal::phase::SampleNow()` once per millisecond for the whole of `ltx2 video: the three carrying phases contain their work and the load keeps its order` -- 3,764 to 27,975 draws per run, measured. `SampleNow()` takes the phase mutex and runs `SampleLocked()`, which invokes the installed `DeviceByteProbe`; `FixtureParams` sets `mp.device = 0` and `src/vllm/multimodal/ltx2_video.cpp` installs a probe calling `vt::Backend::DeviceMemoryInfo`. On the lanes the review could run this is harmless because `CudaBackend` does not override it and returns the -1 sentinel ([#1126](https://github.com/mudler/vllm.cpp/issues/1126)), but `src/vt/rocm/rocm_backend.hip:358` DOES override it. If the reading is right the consequence runs in two directions at once: the render slows because it contends for that mutex against a thread holding it across a driver round trip, and the sampler's own draws inflate, widening the very bound (1c) asserts against. What would settle it, cheaply: one lease on a ROCm device running that case and comparing the printed whole-case boundary median and maximum against this box's 18.4-22.9 us / 0.94-17.5 ms, plus the case wall. If it reproduces, the repair is a null device probe for the sampler's lifetime, or a device-counter cadence far below the boundary-timing cadence. NOT FIXED IN FLOW: the box that found it has no ROCm device and the session was CPU-only, so both reproduction and repair are unmeasured. Filed rather than left in a review transcript, because a 1 kHz driver call under a shared mutex is the kind of thing discovered as "the ROCm CI is slow" months later. Owned by [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `## Owed` | bug | | [#1649](https://github.com/mudler/vllm.cpp/issues/1649) | `GATE-WINDOWS-PORTABILITY-TARGET-SCOPE` | **`check-windows-portability.py` read `/w` anywhere in `CMakeLists.txt` as a project-wide negation of `/W4 /WX`, so the vendored BoringSSL targets' PRIVATE `/w` red `windows-msvc-cpu` and `windows-msvc-vulkan` on main and on every PR.** `check()` set `warnings = cmake` -- the whole file -- concatenated `cmake/CompilerWarnings.cmake` and did a flat token search, which cannot tell a global `add_compile_options(/w)` from a `target_compile_options( PRIVATE /w)`; only the first negates the policy. The refusal lands before any translation unit is read, so the job carries no `error C####`. SECOND red, which #1649 did not record: the same defect fails this checker's own suite -- `test_real_tree_msvc_warning_policy_reaches_the_cxx_compile` with `negation='/w'` on main at `8540a2755` (78 tests, 1 failure) -- so the tree asserted the contradiction in two places at once and one fix clears both. FIXED IN FLOW: `without_foreign_target_compile_options` blanks `target_compile_options` spans whose target PROVABLY names only targets this project never declares (`add_library`/`add_executable` first arguments, with `foreach` bindings resolved); anything unresolved stays in scope, and `cmake/CompilerWarnings.cmake` is kept whole because it applies the policy through an unresolvable function parameter. The widening is bounded by three guard properties proved discriminating by MUTATION -- forcing `_target_is_foreign` to `return True` reds exactly those three and nothing else (82 tests, 3 failures), tree restored byte-for-byte. 82 tests OK; the real tree now prints `Windows portability contract OK`. RESIDUAL, stated not hidden: a negation reaching a project target through a `set()` binding rather than a `foreach()` one is still not caught; no such construct exists in the tree today. Recorded under [`gate-windows-portability-target-scope.md`](specs/gate-windows-portability-target-scope.md) `## 4. Risks / decisions` | bug | | [#961](https://github.com/mudler/vllm.cpp/issues/961) | `BACKEND-CUDA-SM110` | `tests/tools/test_online_gate_startup.py:259` guards `shellcheck` absence with a check that cannot fire, so an absent instrument reads as a code verdict: `test_serve_low_tools` raises `FileNotFoundError: 'shellcheck'` instead of skipping. Filed by the sm_110 baseline lane ([#955](https://github.com/mudler/vllm.cpp/issues/955)), where the leased `thor:gpu0` worker carries no `shellcheck`, so the baseline names the failure as a known entry rather than an sm_110 fact. Indexed late: the issue was opened 2026-08-15 and its index row was lost with the unmerged repair of PR [#956](https://github.com/mudler/vllm.cpp/pull/956) | bug | | [#962](https://github.com/mudler/vllm.cpp/issues/962) | `BACKEND-CUDA-SM110` | NVFP4 marlin disagrees with itself on sm_110: `test_ops_moe_grouped:1262` fails `CHECK(bitdiff == 0)` on `NVFP4 block8-vs-block16 M=8 K=4096 N=4096 bitdiff=15/32768`, against a kernel configure reports as `marlin-nvfp4: ENABLED for [110]`. A live kernel defect on an ENABLED feature, not an absent one, and the only substantive standing sm_110 finding in the [#955](https://github.com/mudler/vllm.cpp/issues/955) baseline. Indexed late, same cause as the row above | bug | From 73c4248b9d6920bae83c9803282adbf12eec419d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 22 Aug 2026 21:52:38 +0000 Subject: [PATCH 6/9] record(LTX25-DEVICE-RESIDENCY): reconcile the pre-withdrawal prose with the withdrawal A fresh review of the record found the file asserting both things at once. The fifth-shape section was written while that shape was expected to land, and nothing in it was touched when it was withdrawn, so a reader who greps #1572 landed on a table saying **closed here**. Three contradictions, all repaired: * the fifth-shape section's own table said #1576 and #1572 were closed and #1536's thesis was refuted. All three are OPEN, and the section now carries a block quote at its head saying the shape does not land and pointing at the withdrawal. It is annotated rather than deleted, because its measurements are what the withdrawal rests on; * `## W0-live`'s coverage note said the 60 ms floor **is now** a per-run number the case prints. It is not: `test_ltx2_video.cpp` still reads `kSpanSlackPerRecord = 0.03` and prints a flat `2 x` it; * the same section's "#1536's thesis is refuted rather than repaired" said in prose what the pull request body says the opposite of. The population that contradicts #1536 stands; the word "refuted" was a consequence of a shape that did not land. Three smaller ones, each of which would otherwise ride into an unrepairable squash message: `rocm_backend.hip:358` is a comment inside `UnifiedMemory()` and the `DeviceMemoryInfo` override is at `:373`; #1668 was cited for a sentence it does not contain, and `AGENTS.md` is cited instead; and the sixth shape's headline 10 reds in 45 mixes a mechanism that was repaired while the population was still running, so 5 in 45 is stated as the defensible figure with the other five reported rather than dropped. The next hypothesis now names its own cost: requiring a leaf to exceed the bound in BOTH renders silences any defect present in only one of them, and whoever builds it owes a mutation of that shape. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-device-residency.md | 62 +++++++++++++++++++------ 1 file changed, 48 insertions(+), 14 deletions(-) diff --git a/.agents/specs/ltx25-device-residency.md b/.agents/specs/ltx25-device-residency.md index 9a61921a3..1b07d2c61 100644 --- a/.agents/specs/ltx25-device-residency.md +++ b/.agents/specs/ltx25-device-residency.md @@ -54,7 +54,7 @@ does not override `vt::Backend::DeviceMemoryInfo` at all** — [#1126](https://github.com/mudler/vllm.cpp/issues/1126), and `grep -rn 'DeviceMemoryInfo' src/vt include/vt` returns the base declaration (`include/vt/backend.h:94`) and exactly one override, ROCm's -(`src/vt/rocm/rocm_backend.hip:358`). A CUDA render on `dgx:gpu0` would print +(`src/vt/rocm/rocm_backend.hip:373`). A CUDA render on `dgx:gpu0` would print `-1` in that column too. W0 did not wire it, and the reason is written down where the seam is: @@ -650,7 +650,7 @@ its phase table lands, and W5 owes one when its wall is accepted. | [#1010](https://github.com/mudler/vllm.cpp/issues/1010) | W0 | **closed, for a run that FINISHES.** The render writes a phase table on the shipped default and the ABI names it through `vllm_video_last_phase_log` (v23). Read the row below it before quoting that as "the render is instrumented" | | [#1413](https://github.com/mudler/vllm.cpp/issues/1413) | W0-live | **closed by `## W0-live` below.** W0's table is written by the success path only, and nothing at all is emitted while a render runs, so an aborted render and a working one both report nothing and the ~162 s DiT forward of [#1375](https://github.com/mudler/vllm.cpp/issues/1375) has no in-process counter | | **a table on a run that does NOT finish** | [#1413](https://github.com/mudler/vllm.cpp/issues/1413), stage W0-live | **the LIVE half is closed by this change; a PARTIAL TABLE on abort is still owed and has no issue.** `WritePhaseLog` has exactly two call sites (`ltx2_video.cpp:2259` audio-only, `:4677` video) and both sit immediately before a successful `return`, three lines after `im.trace.completed = true`. Nineteen `VT_CHECK` sites throw out of `Ltx2VideoEngine::Generate` above them and that body contains no `try` and no `catch`; `vllm_video_generate`'s own two catches set an error and return, and `engine->last_phase_log` is assigned on the success path only. **So a render that is killed, aborted by a lease governor, refused by a guard, or still running leaves no `phase-log.json` at all** — not a truncated one, not an empty one, nothing. The mutation is the demonstration: deleting the video call site removes the file entirely and the W0 gate goes red on `REQUIRE(probe.good())`. That matters here more than anywhere, because the runs this campaign has are the ones that died: [#1375](https://github.com/mudler/vllm.cpp/issues/1375) is `child exit=-15` at 0 frames, and [`ltx25-decode-speed.md`](ltx25-decode-speed.md)'s two rungs are `EXIT=137` and `EXIT=1` at 0 frames. A reader who takes "the render writes a phase table" at face value will expect a 2.5 h render that is killed at 2.4 h to leave a table naming where it was. It leaves none. #1413 CLOSED the live half, in `## W0-live` below — a line per phase boundary and per DiT forward, so a killed run is legible from its stderr; a signal handler that flushed a PARTIAL table on abort is a separate change with its own re-entrancy argument and is not owed by either | -| the phase table's DEVICE column | W1, and it needs [#1126](https://github.com/mudler/vllm.cpp/issues/1126) first | **owed, and a LEASE WILL NOT CLOSE IT.** The column is defined as the driver's live in-use bytes, read per phase through `vt::Backend::DeviceMemoryInfo`. It reports the `-1` no-probe sentinel in W0's artifact, and there are TWO reasons stacked, only one of which is a scheduling problem. **(1)** The render W0 could take was the CPU arm, where the sentinel is correct — `dgx:gpu0` was busy with two queued jobs and `orin:gpu0`, the one free device, holds no LTX-2.5 checkpoints. **(2)** The one that matters: **`CudaBackend` does not override `DeviceMemoryInfo` at all**, which is [#1126](https://github.com/mudler/vllm.cpp/issues/1126), and `grep -rn 'DeviceMemoryInfo' src/vt include/vt` returns exactly the base declaration at `include/vt/backend.h:94` and one override, `src/vt/rocm/rocm_backend.hip:358`. So a CUDA render on `dgx:gpu0` would print `-1` in every row of this column too, and it would print it for a reason no lease can fix. **What the column cannot report today, stated as three things:** how many device bytes the DiT staging leaves resident; whether the denoise grows device residency across steps; and whether the ~59 GiB #1014 asks about is device-class at all. **Why W0 did not just wire it:** `include/vllm/platforms/interface.h:68-72` records that CUDA's absence from that seam is load-bearing — `Gemma4MoE`'s device-expert LRU is the seam's only consumer and is DEAD on CUDA. Narrowed, because the broad form of this sentence is wrong: the bf16 arm `EnsureGemma4Fp8ExpertOnDevice` (`gemma4_moe.cpp:548`) is dead for a SECOND, independent reason — it refuses at `:571` on `vt::HasMatmulBTAlphaBeta`, implemented only by ROCm ([#1205](https://github.com/mudler/vllm.cpp/issues/1205)) — so the override alone would not wake it. The arm the override WOULD wake is the FP8-native one, `EnsureGemma4Fp8NativeOnDevice` (`:611-628`), which reaches `MakeRoom` behind no device gate but the probe, with a 2048 MiB fill-only budget by default (`:416-432`). One named arm, not every CUDA model — and still a behaviour change with its own measurement. That is #1126's change to make, not an instrument's. **What W1 does instead, until #1126 lands:** sample `nvidia-smi --query-compute-apps=used_memory` per phase beside the table — the fallback this spec's W6 row already names, and the one instrument GB10 answers, since `--query-gpu=memory.used` returns `[N/A]` there. On GB10 the peak HOST column is not a poor substitute either: the pool is unified, so host resident bytes and device bytes are the same 119 GiB arena, and that column does report | +| the phase table's DEVICE column | W1, and it needs [#1126](https://github.com/mudler/vllm.cpp/issues/1126) first | **owed, and a LEASE WILL NOT CLOSE IT.** The column is defined as the driver's live in-use bytes, read per phase through `vt::Backend::DeviceMemoryInfo`. It reports the `-1` no-probe sentinel in W0's artifact, and there are TWO reasons stacked, only one of which is a scheduling problem. **(1)** The render W0 could take was the CPU arm, where the sentinel is correct — `dgx:gpu0` was busy with two queued jobs and `orin:gpu0`, the one free device, holds no LTX-2.5 checkpoints. **(2)** The one that matters: **`CudaBackend` does not override `DeviceMemoryInfo` at all**, which is [#1126](https://github.com/mudler/vllm.cpp/issues/1126), and `grep -rn 'DeviceMemoryInfo' src/vt include/vt` returns exactly the base declaration at `include/vt/backend.h:94` and one override, `src/vt/rocm/rocm_backend.hip:373`. So a CUDA render on `dgx:gpu0` would print `-1` in every row of this column too, and it would print it for a reason no lease can fix. **What the column cannot report today, stated as three things:** how many device bytes the DiT staging leaves resident; whether the denoise grows device residency across steps; and whether the ~59 GiB #1014 asks about is device-class at all. **Why W0 did not just wire it:** `include/vllm/platforms/interface.h:68-72` records that CUDA's absence from that seam is load-bearing — `Gemma4MoE`'s device-expert LRU is the seam's only consumer and is DEAD on CUDA. Narrowed, because the broad form of this sentence is wrong: the bf16 arm `EnsureGemma4Fp8ExpertOnDevice` (`gemma4_moe.cpp:548`) is dead for a SECOND, independent reason — it refuses at `:571` on `vt::HasMatmulBTAlphaBeta`, implemented only by ROCm ([#1205](https://github.com/mudler/vllm.cpp/issues/1205)) — so the override alone would not wake it. The arm the override WOULD wake is the FP8-native one, `EnsureGemma4Fp8NativeOnDevice` (`:611-628`), which reaches `MakeRoom` behind no device gate but the probe, with a 2048 MiB fill-only budget by default (`:416-432`). One named arm, not every CUDA model — and still a behaviour change with its own measurement. That is #1126's change to make, not an instrument's. **What W1 does instead, until #1126 lands:** sample `nvidia-smi --query-compute-apps=used_memory` per phase beside the table — the fallback this spec's W6 row already names, and the one instrument GB10 answers, since `--query-gpu=memory.used` returns `[N/A]` there. On GB10 the peak HOST column is not a poor substitute either: the pool is unified, so host resident bytes and device bytes are the same 119 GiB arena, and that column does report | | [#1040](https://github.com/mudler/vllm.cpp/issues/1040) | W0 (contract) + W1 (closes) | contract half **met** — the table is a file beside the frames rather than a console line, and it is retrievable from this repo at `benchmarks/demo/`. The closing half is W1's | | [#1024](https://github.com/mudler/vllm.cpp/issues/1024) | W1 | owed; its `utilization.gpu` positive control is still unrecorded in this tree | | [#1016](https://github.com/mudler/vllm.cpp/issues/1016) | W2a | owed | @@ -667,8 +667,8 @@ its phase table lands, and W5 owes one when its wall is accepted. | [#1012](https://github.com/mudler/vllm.cpp/issues/1012) | O1 | owed | | [#1202](https://github.com/mudler/vllm.cpp/issues/1202) | deferred behind W1 | owed, with a number: 2.3% of one pass | | [#1439](https://github.com/mudler/vllm.cpp/issues/1439) | W0 (its own gate) | **REPAIRED but NOT CLOSED, by the `#1439 repaired` section below: the prologue is now named `load.open`, so the time is measured rather than tolerated, and the assertion is byte-for-byte unchanged. It stays OPEN because the repair was measured on a box at loadavg 24-31, where the render's wall is 7-10 s and BOTH arms sit near 99.96% — the flattering regime this issue itself identified. What is owed is one quiet-box run at fixture scale, in the 0.22-0.58 s regime where the floor actually bites. What follows is the record of why it was red.** **It was RED on `main` rather than on any branch.** `CHECK(leaves >= 0.95 * wall)` in `ltx2 video: a render through the ABI emits a phase table that SUMS to wall` is a RATIO, and the un-named residue is 4.80% to 6.32% of `wall` across twelve runs on one x86 box - so the 95% floor sits INSIDE the measurement's own range at the 64x64 / 9-frame fixture scale, and the case decides by coin flip, mostly red. With this lane's four files reverted so the binary is main at `89261c955`, six runs read 94.32%, 95.20%, 93.74%, 94.20%, 94.69%, 94.19%; the W0-live merge reads 93.82%, 93.68%, 94.34%, 94.62%, 94.39%, and 94.12% with `VLLM_RENDER_PROGRESS=0`, which exonerates the live emitter. Box contention is NOT the cause and main's one green disproves it: that run had `wall=0.579684s`, more than double the others, because the box was loaded - a SLOWER render passes. The tolerance was argued for the 21.004 B render, where the residue is a far smaller fraction of the wall. Naming the un-named time, or bounding `unaccounted_seconds` beside the ratio so the assertion says the same thing at both scales, is gate semantics and owes its own row | -| [#1727](https://github.com/mudler/vllm.cpp/issues/1727) | (1c), sixth shape | **owed, filed by the fresh review of PR #1712.** Two of the boundary sampler's three terms are unfalsifiable by the gate -- deleting the flushed write leaves the estimator inside its own scatter, green 3 of 3 -- and the estimator is unmeasured in two regimes that decide it: the sanitizer lanes, and the FULL 102-case binary rather than a `-tc=` filter, which is where #1576's 171x swing was measured | -| [#1728](https://github.com/mudler/vllm.cpp/issues/1728) | (1c), sixth shape | **owed, filed by the fresh review of PR #1712 as a HYPOTHESIS with no measurement behind it.** On a ROCm runner the 1 kHz boundary sampler would drive `hipMemGetInfo` through `SampleLocked`'s device probe, under the process-wide phase mutex, for the whole case. `CudaBackend` does not override `DeviceMemoryInfo` (#1126) so the CPU and CUDA lanes are unaffected; `src/vt/rocm/rocm_backend.hip:358` does. No GPU lease was taken | +| [#1727](https://github.com/mudler/vllm.cpp/issues/1727) | (1c), sixth shape | **owed, filed by the fresh review of PR #1712, and a CONSTRAINT ON A RE-PROPOSAL rather than a live defect: the sampler it is about is NOT in this tree.** Two of the boundary sampler's three terms are unfalsifiable by the gate -- deleting the flushed write leaves the estimator inside its own scatter, green 3 of 3 -- and the estimator is unmeasured in two regimes that decide it: the sanitizer lanes, and the FULL 102-case binary rather than a `-tc=` filter, which is where #1576's 171x swing was measured | +| [#1728](https://github.com/mudler/vllm.cpp/issues/1728) | (1c), sixth shape | **owed, filed by the fresh review of PR #1712 as a HYPOTHESIS with no measurement behind it, and about a sampler that is NOT in this tree.** On a ROCm runner the 1 kHz boundary sampler would drive `hipMemGetInfo` through `SampleLocked`'s device probe, under the process-wide phase mutex, for the whole case. `CudaBackend` does not override `DeviceMemoryInfo` (#1126) so the CPU and CUDA lanes are unaffected; `src/vt/rocm/rocm_backend.hip:373` does. No GPU lease was taken | | [#1164](https://github.com/mudler/vllm.cpp/issues/1164) | W7 decision point | not owed here — owned by `ENG-CUDAGRAPH-DIFFUSION`; this row owns its unblock order | **[#1009](https://github.com/mudler/vllm.cpp/issues/1009) is not owed. It @@ -1466,10 +1466,12 @@ which is where this spec's `### Decisions taken here` already said they would be the capped bound is a 50% share and the honest head-plus-tail on those records measured 4.6-72.3% of the record itself ([#1559](https://github.com/mudler/vllm.cpp/issues/1559)). - **That floor was a flat 60 ms and is now a per-run number the case prints** - (`### The span-slack bound, fifth shape`). Over 120 runs on this box it ran - 2.4 ms to 838 ms, median 26.6 ms, so the ranges in the table below -- which - were measured against the 60 ms floor -- now say which records are resolvable + **That floor is a flat 60 ms, and the shape that would have made it a per-run + number the case prints was WITHDRAWN** -- see `### The span-slack bound, FIFTH + and SIXTH shapes`. Measured while that shape existed, over 120 runs on this + box, it ran 2.4 ms to 838 ms, median 26.6 ms; on this tree the ranges in the + table below are measured against the flat 60 ms floor and say which records + are resolvable on a QUIET run rather than on every run. A record the run cannot resolve is counted in the case's own `N below this instrument's resolution` line. @@ -2962,14 +2964,22 @@ written when a document had a different shape is how a stale obligation survives ### The span-slack bound, fifth shape: the instrument measures its own resolution (#1572, #1576) +> **THIS SHAPE DOES NOT LAND. Read `### The span-slack bound, FIFTH and SIXTH +> shapes` below before anything here.** A fresh review broke this estimator by +> execution, a sixth shape built to repair it reddened an unmutated tree, and +> both are withdrawn: (1c) on `main` still carries the fourth shape's 30 ms +> constant. Everything measured in this section was measured; nothing concluded +> in it about an issue's state survived. The section is kept whole rather than +> edited, because its measurements are what the withdrawal rests on. + Four issues describe one behaviour -- `test_ltx2_video` decides by box load -- and every repair so far has replaced one constant with a larger constant. | Issue | What it recorded | State after this change | |---|---|---| -| [#1576](https://github.com/mudler/vllm.cpp/issues/1576) | (1c) is a fixed wall-clock budget with an allowance for INSTRUMENTATION and none for LOAD, measured swinging 171x on one binary | **closed here** | -| [#1572](https://github.com/mudler/vllm.cpp/issues/1572) | (1c) redding intermittently on `main`, `decode.video` at 0.00256913 against 0.00075 | **closed here** | -| [#1536](https://github.com/mudler/vllm.cpp/issues/1536) | the phase-coverage floors missed by 0.18-0.8% on an idle box, while the suites that ARE load-dependent passed | **thesis refuted by measurement**, see below | +| [#1576](https://github.com/mudler/vllm.cpp/issues/1576) | (1c) is a fixed wall-clock budget with an allowance for INSTRUMENTATION and none for LOAD, measured swinging 171x on one binary | **STILL OPEN** -- this shape was withdrawn | +| [#1572](https://github.com/mudler/vllm.cpp/issues/1572) | (1c) redding intermittently on `main`, `decode.video` at 0.00256913 against 0.00075 | **STILL OPEN** -- this shape was withdrawn | +| [#1536](https://github.com/mudler/vllm.cpp/issues/1536) | the phase-coverage floors missed by 0.18-0.8% on an idle box, while the suites that ARE load-dependent passed | **STILL OPEN.** Its thesis is contradicted by the 120-run population below, which was taken on an unmutated tree and stands; but "refuted" was written as a consequence of this shape landing, and it did not land | | [#1470](https://github.com/mudler/vllm.cpp/issues/1470) | one case of 96 false-redded under load, four greens after, case identity never captured | **identity and rate measured, STAYS OPEN**, see below | The history of the number they are about is the argument for not choosing @@ -3131,7 +3141,9 @@ so a mutation that failed to build or failed to apply cannot read as a pass. * **It does not close #1470.** The coverage floors still decide by load on `artifacts.frames`, at a measured rate of 1 run in 120 on this box, and the section above is the measurement that says why a bound cannot fix it. -* **#1536's thesis is refuted rather than repaired.** That issue's claim is that +* **#1536 STAYS OPEN. Its thesis is contradicted by the population below, and + "refuted rather than repaired" was written on the assumption that this shape + landed. It did not.** That issue's claim is that the suite is "persistently red on an idle box, not load-flaky". At `73ada0df8` it is the opposite: 119 of 120 green, and the single red was at loadavg 155.65. Its 0.18-0.8% misses were against the 0.95 floor that @@ -3276,6 +3288,16 @@ failed.** 45 consecutive runs of the containment case, one pinned binary 45 runs 35 green 10 RED (22%) ``` +**THE HEADLINE 22% IS NOT THE RED RATE OF THE SHAPE'S FINAL FORM, and the honest +figure is the 5.** Two mechanisms produced those ten, and the second one was +identified and repaired while the population was still running -- so its nine +reds were measured on a binary that predates its own repair, and that repair was +never re-validated. **11%, five reds in 45, is the defensible number**, against +a measured 1 red in 120 (0.83%) for what is on `main`. It is 13x, it is decided +by the same mechanism at any multiplier, and it is what the withdrawal rests on. +The nine are reported rather than dropped because a population that reports only +its surviving mechanism is an argument and not a measurement. + Two mechanisms, both honest: | assertion | reds | what happened | @@ -3300,8 +3322,12 @@ first is not repairable by choosing a different denominator. **So the estimator is withdrawn and (1c) is left exactly as `main` carries it.** Raising the multiplier would be repairing a red gate by widening it, which -`AGENTS.md` refuses and which [#1668](https://github.com/mudler/vllm.cpp/issues/1668) -names as the thing not to re-propose. Four shapes have now been tried and the +`AGENTS.md` `## Changing the rules or a checker` refuses in as many words. +[#1668](https://github.com/mudler/vllm.cpp/issues/1668) does not say that and is +not cited for it -- what it says, quoting this file's own comment, is that +"NAMING THE UN-NAMED TIME WOULD SETTLE IT PROPERLY", i.e. that the repair is an +ANCHOR and not a threshold, which points the same way by a different route. Four +shapes have now been tried and the common factor is that all four compare a wall-clock quantity the scheduler owns against a threshold the tree owns. @@ -3315,6 +3341,14 @@ every pass -- while a one-off preemption, which is what all ten reds above are, hits one pass and not the other. In the five span-slack reds the failing leaf exceeded the bound in exactly one render. +**AND ITS COST IS NAMED BEFORE ANYBODY BUILDS IT.** Requiring both renders +silences any real defect present in only ONE of them -- a first-render lazy +allocation, a cold-cache path, anything the second pass warms. On this fixture +the three carrying leaves run the same code on both passes, which is why the +staged sleeps red 3 of 3, but that is a property of the fixture and not of the +assertion. Whoever builds it owes a mutation that is present in render 1 alone, +and has to say what happens to it. + That is a different assertion with a different failure mode and it owes its own red-first evidence, its own mutation set, and its own 45-run validation. It is recorded here rather than attempted, because attempting a fourth shape in the From da3e14cb6bdb5df9ca964f3468d6dd2b0cd5907f Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 23 Aug 2026 01:00:19 +0000 Subject: [PATCH 7/9] record(LTX25-DEVICE-RESIDENCY): a heading that outlived its shape, and a code anchor the index cannot repair later A fresh review of #1712 found six places where this records-only branch still reads as if the withdrawn fifth shape had landed, or points at a line that has moved. Two of them are why this commit exists rather than a follow-up. The `### Owed out of W0` bullet on (1c) was headed `8 x THIS RUN'S BOUNDARY CEILING` while its own paragraph seven lines below says the floor is a flat 60 ms and that the per-run shape was WITHDRAWN. `main` carries `const double kSpanSlackPerRecord = 0.03` and prints `2.0 * kSpanSlackPerRecord` (`tests/vllm/multimodal/test_ltx2_video.cpp:3869`, `:3974`), so the heading asserted a bound that was measured and never landed. It is restored to the merge base's `60 ms` wording. A heading is what a skimming reader takes away, and this one contradicted the sentence under it. The `#1728` index row cited `src/vt/rocm/rocm_backend.hip:358`, which is a comment; the `DeviceMemoryInfo` override the row is ABOUT is at `:373` on `main` (`:356` is `UnifiedMemory`). `73c4248b9` already corrected the same anchor twice in the spec and missed the index. The index is append-only, so this cannot be repaired after the merge without a union-duplicating edit -- it has to be right now or it is wrong forever. The remaining four are readability, not correctness of record: - The corrected "5 in 45, not 10" headline had not reached the shape-summary table or the `#1572` row in `ltx25-phase-residue.md`. Both raw counts are kept; both now carry the defensible figure and point at the note. - The 45-run population's arithmetic did not close: 5 + 9 = 14 against 10 red RUNS. The run-by-run counts are not reproduced in this file and no cell says whether it counts runs or assertion instances, so the split is stated as an UNRECORDED quantity rather than invented. Only the 10 reds and the 5-in-45 figure are load-bearing, and neither needs the split. - `#### Two more findings, repaired without changing any bound` read in the present tense about a `/dev/null` redirect and an accessor deletion that are not in the tree, which risks a reader believing the `[render]` corruption #1413 names is fixed. `p999_seconds` and `InstrumentCeiling` do not exist on `main`; the heading and both bullets now say so. - Rows `#1727` and `#1728` were inserted before `#1649` with ten rows after them. #1619 records that appending before a trailing anchor is exactly what makes the union driver duplicate a row. They are moved to the true end. Neither row is base-reachable, so this nets to a pure addition and `check-issue-index-append-only.py --base origin/main` is OK. No bound is re-proposed here. The whole point of this branch is that the bound was measured and withdrawn. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 4 +-- .agents/specs/ltx25-device-residency.md | 33 +++++++++++++++++++------ .agents/specs/ltx25-phase-residue.md | 2 +- 3 files changed, 28 insertions(+), 11 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 9d43a09dc..9069982b4 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -628,8 +628,6 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1691](https://github.com/mudler/vllm.cpp/issues/1691) | `DOCS-MODELS-HUMAN` | **`docs/QUICKSTART.md` said no container lane had ever been published, and three had.** The page carried `The container lanes have never been published, so no tag below resolves against the registry yet` and, on its `docker run` line, `The package carries only a stage tag today, which is a build artifact and not a lane`. Both were true when written and false on 2026-08-22: `ghcr.io/mudler/vllm.cpp` is public and carries `main-cpu`, `main-cuda` and `main-vulkan`, each a `linux/amd64` + `linux/arm64` manifest, published 07:30 UTC that day. `docs/guides/container-images.md` already documented the `:main-` tags, so the two pages disagreed. VERIFIED by execution rather than by reading the registry listing: `docker run --rm ghcr.io/mudler/vllm.cpp:main-cpu --version` answered `vllm.cpp 0.0.3 c-abi=23` at digest `sha256:7f88301ea282dad778748929e7aa6869d2418c8d295eef0e7900cca8310d06e5`, and the same image with a mounted `Qwen/Qwen3-0.6B` returned tokens through `/v1/completions` on host `mudler-ubuntu-box` (x86_64, Docker 29.1.2). The image also parses the `vllm_cpp` weight-residency document and echoes `mmap=on prefault=off expert_stream=on expert_stream_slots=4000`, which is what lets the Qwen3.8 2.4T page carry a container form of its recipe. FIXED IN FLOW: the false note is corrected, the `docker run` line names a tag that resolves, and the executed-row table gains its first real row. **This does NOT close [#1281](https://github.com/mudler/vllm.cpp/issues/1281)**: `:latest` still does not exist, `--model org/repo` is still blocked by [#1511](https://github.com/mudler/vllm.cpp/issues/1511) so the executed row mounts a local directory, and no GPU-lane row was run. The executed-row obligation stays owned by #1281 | bug | | [#1690](https://github.com/mudler/vllm.cpp/issues/1690) | `DOC-README-ABI-LANDING-SOURCE` | **`check-doc-checkpoint.py`'s `LANDING_SOURCE_FILES` omitted `include/vllm.h`, so a commit that bumps `VLLM_ABI_VERSION` could never repair the README claim it invalidated.** Found by the review of [PR #1655](https://github.com/mudler/vllm.cpp/pull/1655). The README `## Use it as a library (C API)` block quotes the ABI version out of the header; the header was in `USER_USAGE_FILES` (so an ABI change owes `docs/USAGE.md`) but not in `LANDING_SOURCE_FILES`, and the README rule refuses a claim change unaccompanied by a landing source. The claim could therefore be invalidated but not repaired by the same edit, which is how the README reached `VLLM_ABI_VERSION 21` against a header reading `23`, alongside a stale "46 exported functions" for a header declaring 47. That second half is stale by one rather than by six -- an earlier review figure of 51/52 swept in the `#define VLLM_API` visibility block and counted `vllm_*` identifiers that are typedefs and struct fields rather than exported functions -- so the case for deleting the count is that a live count of one file stored in another goes stale on any ABI addition, not that it is badly wrong. The set's own criterion already admitted it -- the checker's comment says every member is "something the README QUOTES" -- and the header was the only such source missing. FIXED IN FLOW: `include/vllm.h` is added to the set, red-before/green-after pinned by `test_the_c_abi_header_is_a_landing_source`, with `test_the_c_abi_header_permits_but_does_not_demand_readme` proving no new README obligation and the pre-existing no-class tests still green | bug | | [#1692](https://github.com/mudler/vllm.cpp/issues/1692) | `KERNEL-ACCEL-PROVIDER-DECLINE-EXACT` | **[#1584](https://github.com/mudler/vllm.cpp/issues/1584) is FIXED by this row, and its gate is CPU-only: both production call sites need a GPU and neither arm was executed.** The fix adds `GetOpFallbackUncounted`, sharing one body with `GetOpFallback` so resolution order, the reference-tier install, the drain and every throw stay identical and only the `fetch_add` is conditional -- in its existing position, BEFORE the "nothing below" check, because a decline that throws is still a decline. The two SHAPE-GATED providers that hoist the lookup into a function-local static switch to it: `cuda_attention_cross.cu` `BlockedFallback` and `metal_mlx_provider.mm` `MlxFallback`. **Candidate 1 of #1584 -- drop the count from `GetOpFallback` -- was REJECTED and measured rather than argued:** it edits five per-call sites (`vulkan_ops.cpp:950,1067,1488,1509`, `tenstorrent_ops.cpp:1341`), one already gated on an exact value at `test_vulkan_backend.cpp:2901` (`after.declines == before.declines + 1`), and its failure mode for a future caller is silent UNDER-counting, the Risk 4 the seam exists to expose. Net blast radius is ZERO backends, not the four #1584 estimated; exactly two providers change what they report, by one. #1555's `WarmDeclineOnce` workaround is REMOVED. **What this issue owns is the missing EXECUTION:** `test_ops_attention_cross` on CUDA (20 cases, ALL 20 SKIP on a CPU-only build -- 32 assertions, every one the skip guard, so that suite gives the change no coverage here), `test_metal_backend` on a `VLLM_CPP_MLX` build, and the `.agents/reachability.md` mutation on `BlockedFallback()` / `MlxFallback()`. The CUDA COMPILE is covered by the `-DVLLM_CPP_CUDA=ON` `vllm`-target job in `.github/workflows/ci.yml`; the Metal `.mm` file is compiled by NO job in this repository, because MLX needs `MLX_ROOT`. Red-before/green-after and the mutation table in [op-provider-decline-exact.md](specs/op-provider-decline-exact.md) | bug | -| [#1727](https://github.com/mudler/vllm.cpp/issues/1727) | `LTX25-DEVICE-RESIDENCY` | **CONSTRAINT ON ANY RE-PROPOSAL of a sampler-based span-slack denominator, which is NOT on `main`: the boundary sampler carries three terms and a gate can falsify only one.** The estimator this row built was measured and WITHDRAWN -- see [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `### The span-slack bound, FIFTH and SIXTH shapes` -- so what follows is what anyone rebuilding it already owes. `InstrumentCeiling::Loop` timed the process-wide mutex, `SampleNow()`'s `/proc/self/statm` read, and one flushed write standing for `EmitLocked`. A fresh review of [PR #1712](https://github.com/mudler/vllm.cpp/pull/1712) deleted the flushed write entirely -- the term the derivation argues corresponds to the live lane's per-boundary line -- and the case stayed GREEN 3 of 3 at `compile_status=0`, `assertions: 606 \| 606 passed \| 0 failed`, with the ceiling reading 3.24-9.19 ms against an unmutated range of 0.94-17.5 ms: nothing distinguishes the term's removal from run-to-run noise. That same measurement is what freed the write to be pointed at `/dev/null` instead of fd 2, where at 1 kHz it corrupted 27% of the `[render]` lines [#1413](https://github.com/mudler/vllm.cpp/issues/1413) exists for (158 occurrences in one capture, 116 beginning a line) -- so the finding paid for a repair while itself staying open. TWO further halves of the estimator are UNMEASURED rather than ungated. The SANITIZER lane: the third shape of this bound needed its own constant under ASan and TSan because a sanitizer instruments the scope boundary this quantity IS, and the sampler now inherits that automatically by compiling into the same binary -- but the sample count grows with the slowdown too, so the maximum is drawn from a deeper quantile, and nobody has run `sanitize-cpu` against the estimator (the review could not: a second full build tree does not fit at 93% disk). The FULL-SUITE regime: [#1576](https://github.com/mudler/vllm.cpp/issues/1576) measured this quantity at 3.223 ms inside the full 102-case `ctest` run against 18.8 us under a `-tc=` filter at comparable load, **171x**, and that swing is the whole reason the constant was declared unsafe -- yet both the row's 120-run validation and the review's 24-run one were taken almost entirely under `-tc=`. One full-binary run (containment case as #40 of 102) was indistinguishable from the filtered ones at ceiling 2.31 ms, bound 9.22 ms, worst slack 86 us, so the 171x did not reproduce at n=1. NOT FIXED IN FLOW: gating a term of the estimator is a second instrument over the first, and each unmeasured regime needs a build tree this box cannot hold beside the one it has. What would settle it: one `sanitize-cpu` build with the containment case run 40+ times, and one 40-run population through the FULL binary, both reported as distributions. Owned by [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `## Owed` | verification | -| [#1728](https://github.com/mudler/vllm.cpp/issues/1728) | `LTX25-DEVICE-RESIDENCY` | **HYPOTHESIS, filed unmeasured and labelled as such, and a CONSTRAINT ON ANY RE-PROPOSAL rather than a live defect, because the sampler it is about was measured and WITHDRAWN and is not on `main`: on a ROCm runner a 1 kHz boundary sampler inside `(1c)` would call `hipMemGetInfo` 1000 times a second under the process-wide phase mutex.** Raised by the fresh review of [PR #1712](https://github.com/mudler/vllm.cpp/pull/1712) on a CPU-only box with no GPU lease taken. `InstrumentCeiling` runs a thread calling `vllm::multimodal::phase::SampleNow()` once per millisecond for the whole of `ltx2 video: the three carrying phases contain their work and the load keeps its order` -- 3,764 to 27,975 draws per run, measured. `SampleNow()` takes the phase mutex and runs `SampleLocked()`, which invokes the installed `DeviceByteProbe`; `FixtureParams` sets `mp.device = 0` and `src/vllm/multimodal/ltx2_video.cpp` installs a probe calling `vt::Backend::DeviceMemoryInfo`. On the lanes the review could run this is harmless because `CudaBackend` does not override it and returns the -1 sentinel ([#1126](https://github.com/mudler/vllm.cpp/issues/1126)), but `src/vt/rocm/rocm_backend.hip:358` DOES override it. If the reading is right the consequence runs in two directions at once: the render slows because it contends for that mutex against a thread holding it across a driver round trip, and the sampler's own draws inflate, widening the very bound (1c) asserts against. What would settle it, cheaply: one lease on a ROCm device running that case and comparing the printed whole-case boundary median and maximum against this box's 18.4-22.9 us / 0.94-17.5 ms, plus the case wall. If it reproduces, the repair is a null device probe for the sampler's lifetime, or a device-counter cadence far below the boundary-timing cadence. NOT FIXED IN FLOW: the box that found it has no ROCm device and the session was CPU-only, so both reproduction and repair are unmeasured. Filed rather than left in a review transcript, because a 1 kHz driver call under a shared mutex is the kind of thing discovered as "the ROCm CI is slow" months later. Owned by [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `## Owed` | bug | | [#1649](https://github.com/mudler/vllm.cpp/issues/1649) | `GATE-WINDOWS-PORTABILITY-TARGET-SCOPE` | **`check-windows-portability.py` read `/w` anywhere in `CMakeLists.txt` as a project-wide negation of `/W4 /WX`, so the vendored BoringSSL targets' PRIVATE `/w` red `windows-msvc-cpu` and `windows-msvc-vulkan` on main and on every PR.** `check()` set `warnings = cmake` -- the whole file -- concatenated `cmake/CompilerWarnings.cmake` and did a flat token search, which cannot tell a global `add_compile_options(/w)` from a `target_compile_options( PRIVATE /w)`; only the first negates the policy. The refusal lands before any translation unit is read, so the job carries no `error C####`. SECOND red, which #1649 did not record: the same defect fails this checker's own suite -- `test_real_tree_msvc_warning_policy_reaches_the_cxx_compile` with `negation='/w'` on main at `8540a2755` (78 tests, 1 failure) -- so the tree asserted the contradiction in two places at once and one fix clears both. FIXED IN FLOW: `without_foreign_target_compile_options` blanks `target_compile_options` spans whose target PROVABLY names only targets this project never declares (`add_library`/`add_executable` first arguments, with `foreach` bindings resolved); anything unresolved stays in scope, and `cmake/CompilerWarnings.cmake` is kept whole because it applies the policy through an unresolvable function parameter. The widening is bounded by three guard properties proved discriminating by MUTATION -- forcing `_target_is_foreign` to `return True` reds exactly those three and nothing else (82 tests, 3 failures), tree restored byte-for-byte. 82 tests OK; the real tree now prints `Windows portability contract OK`. RESIDUAL, stated not hidden: a negation reaching a project target through a `set()` binding rather than a `foreach()` one is still not caught; no such construct exists in the tree today. Recorded under [`gate-windows-portability-target-scope.md`](specs/gate-windows-portability-target-scope.md) `## 4. Risks / decisions` | bug | | [#961](https://github.com/mudler/vllm.cpp/issues/961) | `BACKEND-CUDA-SM110` | `tests/tools/test_online_gate_startup.py:259` guards `shellcheck` absence with a check that cannot fire, so an absent instrument reads as a code verdict: `test_serve_low_tools` raises `FileNotFoundError: 'shellcheck'` instead of skipping. Filed by the sm_110 baseline lane ([#955](https://github.com/mudler/vllm.cpp/issues/955)), where the leased `thor:gpu0` worker carries no `shellcheck`, so the baseline names the failure as a known entry rather than an sm_110 fact. Indexed late: the issue was opened 2026-08-15 and its index row was lost with the unmerged repair of PR [#956](https://github.com/mudler/vllm.cpp/pull/956) | bug | | [#962](https://github.com/mudler/vllm.cpp/issues/962) | `BACKEND-CUDA-SM110` | NVFP4 marlin disagrees with itself on sm_110: `test_ops_moe_grouped:1262` fails `CHECK(bitdiff == 0)` on `NVFP4 block8-vs-block16 M=8 K=4096 N=4096 bitdiff=15/32768`, against a kernel configure reports as `marlin-nvfp4: ENABLED for [110]`. A live kernel defect on an ENABLED feature, not an absent one, and the only substantive standing sm_110 finding in the [#955](https://github.com/mudler/vllm.cpp/issues/955) baseline. Indexed late, same cause as the row above | bug | @@ -638,3 +636,5 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1725](https://github.com/mudler/vllm.cpp/issues/1725) | `BACKEND-CUDA-SM110` | `kMatmulFp8Cutlass` and `kMatmulFp8BlockScaled` fall through to the portable CPU tier and SEGFAULT on sm_110 (`test_ops_fp8_cutlass:191`, `test_ops_matmul_fp8_block_cuda:345`, measured at `0764ded2b`). The sm_110 baseline attributed them to [#960](https://github.com/mudler/vllm.cpp/issues/960), which was CLOSED COMPLETED three days earlier by `d607fec4c` -- that fix covered `QuantFp8Static` only, while these two ops are registered from TUs `CMakeLists.txt:1790-1791` builds solely for `VT_CUTLASS_FP8_ARCHS`. `cuda_matmul_fp8_block_cutlass.cu:56-58` asserts they refuse by name instead, which the measurement contradicts; `cffe59b02` has since rewritten that dispatch, and no CI lane can see either way because `cutlass-fp8` is ENABLED on the GB10 gate host | bug | | [#1731](https://github.com/mudler/vllm.cpp/issues/1731) | `ENG-RECORD-CONFLICT-SURFACES` | `check-agent-record` is RED on `main`: `#1649` is listed twice in `.agents/issue-index.md`, at `:592` (added by `a7bb3130b`, the lane that FILED it) and `:632` (added by `2f2a70925`, the lane that FIXED it). Neither is wrong and neither could see the other -- `merge=union` combines two appends silently, so a duplicate is the ordinary outcome when filing and fixing happen on different branches, and the checker only notices once both have landed on main, where every later PR inherits the red. Not repaired in flow: the index preamble forbids editing or deleting a row, and the two bodies carry different facts (`:632` has a second red and the mutation evidence, `:592` has the attribution to `a50c57d69`), so choosing which survives is a judgement | bug | | [#1733](https://github.com/mudler/vllm.cpp/issues/1733) | `ENG-RECORD-CONFLICT-SURFACES` | CLOSED as a duplicate of [#1731](https://github.com/mudler/vllm.cpp/issues/1731), filed seventeen minutes after it against the same red, and recorded here rather than left unindexed because its one MEASURED claim is FALSE and an append-only row cannot be corrected in place later. Both issues report that `.agents/issue-index.md` lists [#1649](https://github.com/mudler/vllm.cpp/issues/1649) twice on `main` at `038ff61e5`, at `:592` from `a7bb3130b` (the lane that FILED it) and at `:632` from `2f2a70925` (the lane that FIXED it). #1731 is the earlier filing, already carries an index row, and is the record. #1733 adds one thing #1731 does not, and it is wrong: "the repair is measured and it is NOT blocked", on the evidence that removing one of the two rows in a worktree and running both checkers gives `agent record OK: ENGINE=170 MODEL=377 ...` and `OK: issue index append-only`. The second half is an artefact of the instrument. `scripts/check-issue-index-append-only.py:50-51` diffs `merge-base(origin/main, HEAD)..HEAD`, which reads COMMITS, so an UNCOMMITTED deletion is invisible to it. Measured on `row/FIX-ISSUE-INDEX-1649-DUP` at base `038ff61e5`: deleting `:592` in the WORKING TREE alone returns `OK: issue index append-only` at rc 0 with `git diff --numstat 038ff61e5..HEAD -- .agents/issue-index.md` EMPTY, and committing the byte-identical deletion turns the same checker rc 1 with a `removed:` line naming the row. #1733's own quoted `agent record OK: ENGINE=170 MODEL=377` is the tell, because that is the working-tree reading and the committed tree cannot produce it while the duplicate stands. So the duplicate IS base-reachable, the two checkers ARE in genuine contradiction on this tree, and the repair is the argued exception #1731's row anticipated rather than the free edit #1733 reported | bug | +| [#1727](https://github.com/mudler/vllm.cpp/issues/1727) | `LTX25-DEVICE-RESIDENCY` | **CONSTRAINT ON ANY RE-PROPOSAL of a sampler-based span-slack denominator, which is NOT on `main`: the boundary sampler carries three terms and a gate can falsify only one.** The estimator this row built was measured and WITHDRAWN -- see [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `### The span-slack bound, FIFTH and SIXTH shapes` -- so what follows is what anyone rebuilding it already owes. `InstrumentCeiling::Loop` timed the process-wide mutex, `SampleNow()`'s `/proc/self/statm` read, and one flushed write standing for `EmitLocked`. A fresh review of [PR #1712](https://github.com/mudler/vllm.cpp/pull/1712) deleted the flushed write entirely -- the term the derivation argues corresponds to the live lane's per-boundary line -- and the case stayed GREEN 3 of 3 at `compile_status=0`, `assertions: 606 \| 606 passed \| 0 failed`, with the ceiling reading 3.24-9.19 ms against an unmutated range of 0.94-17.5 ms: nothing distinguishes the term's removal from run-to-run noise. That same measurement is what freed the write to be pointed at `/dev/null` instead of fd 2, where at 1 kHz it corrupted 27% of the `[render]` lines [#1413](https://github.com/mudler/vllm.cpp/issues/1413) exists for (158 occurrences in one capture, 116 beginning a line) -- so the finding paid for a repair while itself staying open. TWO further halves of the estimator are UNMEASURED rather than ungated. The SANITIZER lane: the third shape of this bound needed its own constant under ASan and TSan because a sanitizer instruments the scope boundary this quantity IS, and the sampler now inherits that automatically by compiling into the same binary -- but the sample count grows with the slowdown too, so the maximum is drawn from a deeper quantile, and nobody has run `sanitize-cpu` against the estimator (the review could not: a second full build tree does not fit at 93% disk). The FULL-SUITE regime: [#1576](https://github.com/mudler/vllm.cpp/issues/1576) measured this quantity at 3.223 ms inside the full 102-case `ctest` run against 18.8 us under a `-tc=` filter at comparable load, **171x**, and that swing is the whole reason the constant was declared unsafe -- yet both the row's 120-run validation and the review's 24-run one were taken almost entirely under `-tc=`. One full-binary run (containment case as #40 of 102) was indistinguishable from the filtered ones at ceiling 2.31 ms, bound 9.22 ms, worst slack 86 us, so the 171x did not reproduce at n=1. NOT FIXED IN FLOW: gating a term of the estimator is a second instrument over the first, and each unmeasured regime needs a build tree this box cannot hold beside the one it has. What would settle it: one `sanitize-cpu` build with the containment case run 40+ times, and one 40-run population through the FULL binary, both reported as distributions. Owned by [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `## Owed` | verification | +| [#1728](https://github.com/mudler/vllm.cpp/issues/1728) | `LTX25-DEVICE-RESIDENCY` | **HYPOTHESIS, filed unmeasured and labelled as such, and a CONSTRAINT ON ANY RE-PROPOSAL rather than a live defect, because the sampler it is about was measured and WITHDRAWN and is not on `main`: on a ROCm runner a 1 kHz boundary sampler inside `(1c)` would call `hipMemGetInfo` 1000 times a second under the process-wide phase mutex.** Raised by the fresh review of [PR #1712](https://github.com/mudler/vllm.cpp/pull/1712) on a CPU-only box with no GPU lease taken. `InstrumentCeiling` runs a thread calling `vllm::multimodal::phase::SampleNow()` once per millisecond for the whole of `ltx2 video: the three carrying phases contain their work and the load keeps its order` -- 3,764 to 27,975 draws per run, measured. `SampleNow()` takes the phase mutex and runs `SampleLocked()`, which invokes the installed `DeviceByteProbe`; `FixtureParams` sets `mp.device = 0` and `src/vllm/multimodal/ltx2_video.cpp` installs a probe calling `vt::Backend::DeviceMemoryInfo`. On the lanes the review could run this is harmless because `CudaBackend` does not override it and returns the -1 sentinel ([#1126](https://github.com/mudler/vllm.cpp/issues/1126)), but `src/vt/rocm/rocm_backend.hip:373` DOES override it. If the reading is right the consequence runs in two directions at once: the render slows because it contends for that mutex against a thread holding it across a driver round trip, and the sampler's own draws inflate, widening the very bound (1c) asserts against. What would settle it, cheaply: one lease on a ROCm device running that case and comparing the printed whole-case boundary median and maximum against this box's 18.4-22.9 us / 0.94-17.5 ms, plus the case wall. If it reproduces, the repair is a null device probe for the sampler's lifetime, or a device-counter cadence far below the boundary-timing cadence. NOT FIXED IN FLOW: the box that found it has no ROCm device and the session was CPU-only, so both reproduction and repair are unmeasured. Filed rather than left in a review transcript, because a 1 kHz driver call under a shared mutex is the kind of thing discovered as "the ROCm CI is slow" months later. Owned by [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `## Owed` | bug | diff --git a/.agents/specs/ltx25-device-residency.md b/.agents/specs/ltx25-device-residency.md index 1b07d2c61..02352a7fd 100644 --- a/.agents/specs/ltx25-device-residency.md +++ b/.agents/specs/ltx25-device-residency.md @@ -1459,9 +1459,9 @@ which is where this spec's `### Decisions taken here` already said they would be The `load` and `generate` records are SPANS and are never summed, so they are not in this list; `sum_rule` in every emitted table says which records add up. -* **(1c) DOES NOT HOLD A LEAF RECORD SHORTER THAN `8 x THIS RUN'S BOUNDARY - CEILING`**, and this entry names which ones so no later stage inherits the - silence. The span-slack bound is per leaf RECORD and its `min` cap binds below +* **(1c) DOES NOT HOLD A LEAF RECORD SHORTER THAN 60 ms**, and this + entry names which ones so no later stage inherits the silence. The span-slack + bound is per leaf RECORD and its `min` cap binds below `2 * kSpanSlackPerRecord`; a record there is reported and not checked, because the capped bound is a 50% share and the honest head-plus-tail on those records measured 4.6-72.3% of the record itself ([#1559](https://github.com/mudler/vllm.cpp/issues/1559)). @@ -3168,7 +3168,7 @@ built, measured and withdrawn, so the next person does not build them again. |---|---|---|---| | 4th (on `main`) | a flat 30 ms per leaf record | reds an unmutated tree under load, and cannot resolve `artifacts.frames` at all | [#1576](https://github.com/mudler/vllm.cpp/issues/1576), 171x on one binary | | 5th | `4 x` the WORST boundary a 1 kHz sampler saw across the WHOLE case | one scheduling event on the sampler thread multiplies the bound by ~100 and silences a real 20 ms un-named phase | fresh review: defect reds 9/9 alone, 0/4 with one 200 ms sampler stall, case GREEN 2/4 | -| 6th | `4 x` the worst boundary inside the RECORD'S OWN window | too tight: the honest span slack's tail exceeds it, because the UN-instrumented remainder of a boundary is in the numerator and not in the denominator | **10 red in 45 consecutive runs** of the unmutated containment case, loadavg 21.8-61.5 | +| 6th | `4 x` the worst boundary inside the RECORD'S OWN window | too tight: the honest span slack's tail exceeds it, because the UN-instrumented remainder of a boundary is in the numerator and not in the denominator | **10 red in 45 consecutive runs** of the unmutated containment case, loadavg 21.8-61.5 -- of which **5 in 45 is the defensible figure**, because a second mechanism was repaired while the population was still running and its reds were measured on a binary that predates its own repair; see the note under the population below | The fifth shape's derivation is right and a fresh review confirmed it operation-by-operation against `render_phase_log.cpp`. **Its ESTIMATOR was @@ -3255,7 +3255,7 @@ A record too short to hold a single draw of a 1 kHz sampler is counted and reported separately from a record the bound cannot resolve, because those are different facts about the instrument. -#### Two more findings, repaired without changing any bound +#### Two more findings, repaired in the withdrawn shape, NONE of which is in the tree * **The sampler's flushed write went to fd 2 at 1 kHz and corrupted the live lane.** One capture held 158 `[render]` occurrences of which only **116 began @@ -3263,12 +3263,17 @@ different facts about the instrument. row that exists for that stream. The same review deleted the write entirely and the estimator did not move (green 3 of 3, ceiling 3.24-9.19 ms against an unmutated range of 0.94-17.5 ms), so the SINK is not what the term measures. - The write is kept and pointed at `/dev/null`. + In the withdrawn shape the write was kept and pointed at `/dev/null`. **That + repair went out with the shape and is NOT in the tree**, so the `[render]` + stream-corruption [#1413](https://github.com/mudler/vllm.cpp/issues/1413) names + is not fixed by anything here; what stands is the measurement that the sink is + not the term. * **`p999_seconds()` was dead**, called from nowhere, and would not have done what its comment claimed: `Quantile(999, 1000)` clamps to `n - 1` for every `n <= 1000`, so at the sample counts a short run produces it returned the - maximum. The accessor is deleted and the argument for rejecting a quantile is - kept. + maximum. In the withdrawn shape the accessor was deleted; **that deletion is + NOT in the tree either**, because `p999_seconds()` only ever existed on the + withdrawn estimator. What is kept is the argument for rejecting a quantile. #### And the sixth shape was measured too, and it is WITHDRAWN @@ -3305,6 +3310,18 @@ Two mechanisms, both honest: | `span_slack <= span_bound` | 5 | the honest head-and-tail exceeded `4 x` the record's own local worst boundary -- `decode.audio` 13.115 ms against 8.889 ms, `decode.video` 0.488 ms against 0.374 ms, `artifacts.frames` 70.6 us against 60.6 us | | `longest_checked` | 9 | a leaf's longest record was not resolvable, most often `artifacts.frames` at about 1 ms with no draw of a 1 kHz sampler inside it | +**Those two counts do not sum to the ten red RUNS, and this record cannot close +the gap.** 5 + 9 = 14 against 10, so the two columns overlap; the run-by-run +counts behind `45 runs / 35 green / 10 RED` were recorded during the population +but are not reproduced in this file, and nothing here says whether a cell counts +red RUNS or red ASSERTION INSTANCES. Read as runs the overlap is forced to be at +least four, because 14 events cannot be spread over 10 runs without four +doubling up; read as instances one run can contribute two reds on two leaves and +the overlap is not bounded at all. **The split is therefore an unrecorded +quantity and no number here should be read as one.** What is load-bearing, and +what the withdrawal rests on, is the 10 red runs in 45 and the 5-in-45 +defensible figure -- neither of which needs the split. + The first mechanism is the one that matters and it is the SAME one [`ltx25-phase-residue.md`](ltx25-phase-residue.md) `## Design` 3 records for the withdrawn residue bound: **the part of a boundary the instrument cannot measure diff --git a/.agents/specs/ltx25-phase-residue.md b/.agents/specs/ltx25-phase-residue.md index 0743f3944..38aae0b73 100644 --- a/.agents/specs/ltx25-phase-residue.md +++ b/.agents/specs/ltx25-phase-residue.md @@ -234,7 +234,7 @@ once, at `tests/vllm/multimodal/test_ltx2_video.cpp:4325`, asking for it. | [#1569](https://github.com/mudler/vllm.cpp/issues/1569) | a gate on `WriteJson`'s clock ORDERING, **measured green under its own mutation**. Restoring the old order left the conservation case GREEN 10 of 10, at `wall 0.0608987s, unaccounted 0.000534223s, table charge 0.000301655s`, because the copy and sort of a three-record table are nanoseconds. Gating it needs a table with enough records for the sort to be measurable | | [#1570](https://github.com/mudler/vllm.cpp/issues/1570) | an upper bound on the instrument's own share of a leaf. `uncovered <= 2 * leaf_instrument` is stricter than the floor it replaces only while `leaf_instrument` stays small, and nothing bounds it. Moving the DiT `Tick` out of `Evaluate` would charge ~110 flushed writes to `denoise` and widen the gate while printing a small number | | [#1571](https://github.com/mudler/vllm.cpp/issues/1571) | a per-gap decomposition IN the emitted table. The 92% region above was found with a scratch script; a reader of `phase-log.json` still cannot see it without one, and the same investigation will be re-derived the next time the residue moves | -| [#1572](https://github.com/mudler/vllm.cpp/issues/1572) | assertion (1c)'s span slack reds intermittently on `main` — `decode.video` at `0.00256913` against a `0.00075` bound, 3.4x. Pre-existing from `6b48edb2c` and not this row's. **STILL OPEN, and two further shapes of that bound are now MEASURED SHUT.** `LTX25-DEVICE-RESIDENCY` built a fifth shape (`4 x` the worst boundary a 1 kHz sampler saw across the whole case) and a sixth (`4 x` the worst inside the record's OWN window) and withdrew both: the fifth lets one 200 ms descheduling of the sampler thread turn a real 20 ms un-named phase from red 9 of 9 into a GREEN case, and the sixth reds an unmutated tree 10 times in 45 consecutive runs at loadavg 21.8-61.5. See `.agents/specs/ltx25-device-residency.md` `### The span-slack bound, FIFTH and SIXTH shapes` for both distributions and for the one hypothesis that has not been tried | +| [#1572](https://github.com/mudler/vllm.cpp/issues/1572) | assertion (1c)'s span slack reds intermittently on `main` — `decode.video` at `0.00256913` against a `0.00075` bound, 3.4x. Pre-existing from `6b48edb2c` and not this row's. **STILL OPEN, and two further shapes of that bound are now MEASURED SHUT.** `LTX25-DEVICE-RESIDENCY` built a fifth shape (`4 x` the worst boundary a 1 kHz sampler saw across the whole case) and a sixth (`4 x` the worst inside the record's OWN window) and withdrew both: the fifth lets one 200 ms descheduling of the sampler thread turn a real 20 ms un-named phase from red 9 of 9 into a GREEN case, and the sixth reds an unmutated tree 10 times in 45 consecutive runs at loadavg 21.8-61.5 -- of which **5 in 45 is the defensible figure**, because a second mechanism was identified and repaired while that population was still running, so its reds were measured on a binary that predates its own repair. See `.agents/specs/ltx25-device-residency.md` `### The span-slack bound, FIFTH and SIXTH shapes` for both distributions and for the one hypothesis that has not been tried | | [#1619](https://github.com/mudler/vllm.cpp/issues/1619) | **the `merge=union` driver duplicates a row, MEASURED on this row's own merges.** Both sides appended before the same trailing anchor rather than at the true end, so the driver concatenated two regions that each carried `#1546` and the resolved index held it TWICE, byte-identical, at 538 lines where the correct union is 537. `git merge-tree` called that merge clean and `check-issue-index-append-only.py` passed it, because a duplicate is an ADDITION and that checker only collects removals. `check-agent-record.py` did NOT pass it -- a claim #1556's spec made and this row REFUTED by reproduction: regenerating the raw driver output and running that same tree's checker returns rc=1 with `issue #1546 listed twice`, and the refusal has existed since `8dd6508da` (2026-08-09), before the merge. So the blind gate is exactly one checker, not two, and the gap is narrower than #1556 recorded. The de-duplication half is CONDITIONAL, and the condition is what #1556's spec omitted: the checker reds a repair only when the DUPLICATE IS ALREADY IN THE BASE. Measured at three pairings -- `--base e2a9e035d` against the real canonical 537-line file rc=0, against a synthetic 537 rc=0, and `--base --head <537 de-dup>` rc=1. It diffs `merge-base..HEAD`, so when the base predates the duplicate the addition and the removal CANCEL and it passes. Since `origin/main` is preflight's base, and is the shape this branch used, the gate does NOT red someone who repairs driver output before committing it -- only someone repairing a corruption that already landed. The same range property is why relocating a base-reachable row DOES red it: moving row `#168` to the end gives rc=1 and a `removed:` line naming it. So "de-duplicating in place FAILS the checker", as #1556's spec put it, is false unqualified and true once the duplicate is base-reachable. #1556's spec added that the same driver dropped `#838` on a later re-merge, making this a recurring class; that is WITHDRAWN as unreproducible. Re-running `git merge-file --union` at every later merge where `#838` was on a side leaves it present in all of them, and `git log -S` finds it absent from no committed state -- mechanically a union driver cannot drop a line that is an addition on one side. If it ever went missing, that points at a wholesale take-ours resolution rather than at the driver | | [#1439](https://github.com/mudler/vllm.cpp/issues/1439) | **NOT closed by this row, and it must not be.** See `## Risks and decisions` D4 | | [#1470](https://github.com/mudler/vllm.cpp/issues/1470) | `test_ltx2_video` false-redded once on `main` under load and the failing case's identity was never captured. Untouched by THIS row, and **the identity and the rate are now measured** by `LTX25-DEVICE-RESIDENCY`: 1 red in 120 runs of the containment case at loadavg 40-155, on `artifacts.frames` render 2, where a 67.55 ms descheduling between `ppm_phase.Close()` and the leaf's destructor left 67.55 ms of a 74.87 ms leaf uncovered. It reds `covered >= 0.50 * leaf_seconds`. The same row also measured, and WITHDREW, the obvious repair: an instrument-relative second arm on that floor makes mutation `B-empty-ppm` — the writer's anchor opened after the write loop instead of around it, coverage 98% to 1.1% — pass. `artifacts.frames` is 0.2-7 ms on this fixture and one boundary on that host is 0.3-1.2 ms pinned to two idle cores, so no allowance built from the boundary is smaller than the leaf. The repair is an anchor, not a threshold | From ac17ff604553034235173654e3abf9a1c0d33a95 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 23 Aug 2026 01:39:59 +0000 Subject: [PATCH 8/9] record(LTX25-DEVICE-RESIDENCY): two survivors of the class the last commit closed, one of them on the append-only surface A second fresh review found that the previous repair fixed the two bullets it was pointed at and missed two siblings of the same class: prose that reads as if the withdrawn shape's code were in the tree. The `#1727` index row still ended with the `/dev/null` redirect as an accomplished repair -- "so the finding paid for a repair while itself staying open". No redirect exists anywhere under `src/vllm/multimodal/` or `tests/vllm/multimodal/`, on `origin/main` or here. So the branch asserted both halves at once: the spec at `:3266` already said the repair went out with the shape, while the index banked it against #1413, which is live and open. The row now says the redirect is not in the tree and that #1413 is unfixed. This is first in this commit because the index is append-only and `#1727` is not reachable from `origin/main` -- it is free to edit now and impossible to repair cleanly after the merge, the same reason the `:373` anchor could not wait. The `#### The skip path was a permanent report-only state` section had the same mixed tense, and worse: its opening sentence stated the WITHDRAWN skip condition, `record_seconds >= 8 x ceiling`, as if it were the tree's. `main` skips on `span_bound < kSpanSlackPerRecord` (`test_ltx2_video.cpp:3971`), the flat 60 ms. `longest_checked` does not exist in the tree at all. What makes this one worth care rather than a tense fix is that the paragraph's underlying observation IS true of `main`: `span_checked` and `span_unresolvable` are incremented at `:3972` and `:3983`, printed once at `:3999-4001`, and asserted nowhere. A reader who discounted the paragraph as withdrawn-shape prose would lose a live report-only state that `AGENTS.md` `## Gates` refuses. So the observation is kept and attributed to `main`, the assertion is put in the past and named as absent, and the section now says plainly that the report-only state is STILL OPEN. The argument for that shape of assertion is what survives; the assertion is not here. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 2 +- .agents/specs/ltx25-device-residency.md | 42 ++++++++++++++++--------- 2 files changed, 28 insertions(+), 16 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 9069982b4..eba4c771f 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -636,5 +636,5 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1725](https://github.com/mudler/vllm.cpp/issues/1725) | `BACKEND-CUDA-SM110` | `kMatmulFp8Cutlass` and `kMatmulFp8BlockScaled` fall through to the portable CPU tier and SEGFAULT on sm_110 (`test_ops_fp8_cutlass:191`, `test_ops_matmul_fp8_block_cuda:345`, measured at `0764ded2b`). The sm_110 baseline attributed them to [#960](https://github.com/mudler/vllm.cpp/issues/960), which was CLOSED COMPLETED three days earlier by `d607fec4c` -- that fix covered `QuantFp8Static` only, while these two ops are registered from TUs `CMakeLists.txt:1790-1791` builds solely for `VT_CUTLASS_FP8_ARCHS`. `cuda_matmul_fp8_block_cutlass.cu:56-58` asserts they refuse by name instead, which the measurement contradicts; `cffe59b02` has since rewritten that dispatch, and no CI lane can see either way because `cutlass-fp8` is ENABLED on the GB10 gate host | bug | | [#1731](https://github.com/mudler/vllm.cpp/issues/1731) | `ENG-RECORD-CONFLICT-SURFACES` | `check-agent-record` is RED on `main`: `#1649` is listed twice in `.agents/issue-index.md`, at `:592` (added by `a7bb3130b`, the lane that FILED it) and `:632` (added by `2f2a70925`, the lane that FIXED it). Neither is wrong and neither could see the other -- `merge=union` combines two appends silently, so a duplicate is the ordinary outcome when filing and fixing happen on different branches, and the checker only notices once both have landed on main, where every later PR inherits the red. Not repaired in flow: the index preamble forbids editing or deleting a row, and the two bodies carry different facts (`:632` has a second red and the mutation evidence, `:592` has the attribution to `a50c57d69`), so choosing which survives is a judgement | bug | | [#1733](https://github.com/mudler/vllm.cpp/issues/1733) | `ENG-RECORD-CONFLICT-SURFACES` | CLOSED as a duplicate of [#1731](https://github.com/mudler/vllm.cpp/issues/1731), filed seventeen minutes after it against the same red, and recorded here rather than left unindexed because its one MEASURED claim is FALSE and an append-only row cannot be corrected in place later. Both issues report that `.agents/issue-index.md` lists [#1649](https://github.com/mudler/vllm.cpp/issues/1649) twice on `main` at `038ff61e5`, at `:592` from `a7bb3130b` (the lane that FILED it) and at `:632` from `2f2a70925` (the lane that FIXED it). #1731 is the earlier filing, already carries an index row, and is the record. #1733 adds one thing #1731 does not, and it is wrong: "the repair is measured and it is NOT blocked", on the evidence that removing one of the two rows in a worktree and running both checkers gives `agent record OK: ENGINE=170 MODEL=377 ...` and `OK: issue index append-only`. The second half is an artefact of the instrument. `scripts/check-issue-index-append-only.py:50-51` diffs `merge-base(origin/main, HEAD)..HEAD`, which reads COMMITS, so an UNCOMMITTED deletion is invisible to it. Measured on `row/FIX-ISSUE-INDEX-1649-DUP` at base `038ff61e5`: deleting `:592` in the WORKING TREE alone returns `OK: issue index append-only` at rc 0 with `git diff --numstat 038ff61e5..HEAD -- .agents/issue-index.md` EMPTY, and committing the byte-identical deletion turns the same checker rc 1 with a `removed:` line naming the row. #1733's own quoted `agent record OK: ENGINE=170 MODEL=377` is the tell, because that is the working-tree reading and the committed tree cannot produce it while the duplicate stands. So the duplicate IS base-reachable, the two checkers ARE in genuine contradiction on this tree, and the repair is the argued exception #1731's row anticipated rather than the free edit #1733 reported | bug | -| [#1727](https://github.com/mudler/vllm.cpp/issues/1727) | `LTX25-DEVICE-RESIDENCY` | **CONSTRAINT ON ANY RE-PROPOSAL of a sampler-based span-slack denominator, which is NOT on `main`: the boundary sampler carries three terms and a gate can falsify only one.** The estimator this row built was measured and WITHDRAWN -- see [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `### The span-slack bound, FIFTH and SIXTH shapes` -- so what follows is what anyone rebuilding it already owes. `InstrumentCeiling::Loop` timed the process-wide mutex, `SampleNow()`'s `/proc/self/statm` read, and one flushed write standing for `EmitLocked`. A fresh review of [PR #1712](https://github.com/mudler/vllm.cpp/pull/1712) deleted the flushed write entirely -- the term the derivation argues corresponds to the live lane's per-boundary line -- and the case stayed GREEN 3 of 3 at `compile_status=0`, `assertions: 606 \| 606 passed \| 0 failed`, with the ceiling reading 3.24-9.19 ms against an unmutated range of 0.94-17.5 ms: nothing distinguishes the term's removal from run-to-run noise. That same measurement is what freed the write to be pointed at `/dev/null` instead of fd 2, where at 1 kHz it corrupted 27% of the `[render]` lines [#1413](https://github.com/mudler/vllm.cpp/issues/1413) exists for (158 occurrences in one capture, 116 beginning a line) -- so the finding paid for a repair while itself staying open. TWO further halves of the estimator are UNMEASURED rather than ungated. The SANITIZER lane: the third shape of this bound needed its own constant under ASan and TSan because a sanitizer instruments the scope boundary this quantity IS, and the sampler now inherits that automatically by compiling into the same binary -- but the sample count grows with the slowdown too, so the maximum is drawn from a deeper quantile, and nobody has run `sanitize-cpu` against the estimator (the review could not: a second full build tree does not fit at 93% disk). The FULL-SUITE regime: [#1576](https://github.com/mudler/vllm.cpp/issues/1576) measured this quantity at 3.223 ms inside the full 102-case `ctest` run against 18.8 us under a `-tc=` filter at comparable load, **171x**, and that swing is the whole reason the constant was declared unsafe -- yet both the row's 120-run validation and the review's 24-run one were taken almost entirely under `-tc=`. One full-binary run (containment case as #40 of 102) was indistinguishable from the filtered ones at ceiling 2.31 ms, bound 9.22 ms, worst slack 86 us, so the 171x did not reproduce at n=1. NOT FIXED IN FLOW: gating a term of the estimator is a second instrument over the first, and each unmeasured regime needs a build tree this box cannot hold beside the one it has. What would settle it: one `sanitize-cpu` build with the containment case run 40+ times, and one 40-run population through the FULL binary, both reported as distributions. Owned by [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `## Owed` | verification | +| [#1727](https://github.com/mudler/vllm.cpp/issues/1727) | `LTX25-DEVICE-RESIDENCY` | **CONSTRAINT ON ANY RE-PROPOSAL of a sampler-based span-slack denominator, which is NOT on `main`: the boundary sampler carries three terms and a gate can falsify only one.** The estimator this row built was measured and WITHDRAWN -- see [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `### The span-slack bound, FIFTH and SIXTH shapes` -- so what follows is what anyone rebuilding it already owes. `InstrumentCeiling::Loop` timed the process-wide mutex, `SampleNow()`'s `/proc/self/statm` read, and one flushed write standing for `EmitLocked`. A fresh review of [PR #1712](https://github.com/mudler/vllm.cpp/pull/1712) deleted the flushed write entirely -- the term the derivation argues corresponds to the live lane's per-boundary line -- and the case stayed GREEN 3 of 3 at `compile_status=0`, `assertions: 606 \| 606 passed \| 0 failed`, with the ceiling reading 3.24-9.19 ms against an unmutated range of 0.94-17.5 ms: nothing distinguishes the term's removal from run-to-run noise. That same measurement is what freed the write to be pointed at `/dev/null` instead of fd 2, where at 1 kHz it corrupted 27% of the `[render]` lines [#1413](https://github.com/mudler/vllm.cpp/issues/1413) exists for (158 occurrences in one capture, 116 beginning a line) -- **but that redirect went out with the withdrawn shape and is NOT in the tree**, on `origin/main` or on this branch, so #1413's stream corruption is UNFIXED and nothing here may be banked against it. What stands is the measurement, not a repair: the sink is not what the term measures. TWO further halves of the estimator are UNMEASURED rather than ungated. The SANITIZER lane: the third shape of this bound needed its own constant under ASan and TSan because a sanitizer instruments the scope boundary this quantity IS, and the sampler now inherits that automatically by compiling into the same binary -- but the sample count grows with the slowdown too, so the maximum is drawn from a deeper quantile, and nobody has run `sanitize-cpu` against the estimator (the review could not: a second full build tree does not fit at 93% disk). The FULL-SUITE regime: [#1576](https://github.com/mudler/vllm.cpp/issues/1576) measured this quantity at 3.223 ms inside the full 102-case `ctest` run against 18.8 us under a `-tc=` filter at comparable load, **171x**, and that swing is the whole reason the constant was declared unsafe -- yet both the row's 120-run validation and the review's 24-run one were taken almost entirely under `-tc=`. One full-binary run (containment case as #40 of 102) was indistinguishable from the filtered ones at ceiling 2.31 ms, bound 9.22 ms, worst slack 86 us, so the 171x did not reproduce at n=1. NOT FIXED IN FLOW: gating a term of the estimator is a second instrument over the first, and each unmeasured regime needs a build tree this box cannot hold beside the one it has. What would settle it: one `sanitize-cpu` build with the containment case run 40+ times, and one 40-run population through the FULL binary, both reported as distributions. Owned by [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `## Owed` | verification | | [#1728](https://github.com/mudler/vllm.cpp/issues/1728) | `LTX25-DEVICE-RESIDENCY` | **HYPOTHESIS, filed unmeasured and labelled as such, and a CONSTRAINT ON ANY RE-PROPOSAL rather than a live defect, because the sampler it is about was measured and WITHDRAWN and is not on `main`: on a ROCm runner a 1 kHz boundary sampler inside `(1c)` would call `hipMemGetInfo` 1000 times a second under the process-wide phase mutex.** Raised by the fresh review of [PR #1712](https://github.com/mudler/vllm.cpp/pull/1712) on a CPU-only box with no GPU lease taken. `InstrumentCeiling` runs a thread calling `vllm::multimodal::phase::SampleNow()` once per millisecond for the whole of `ltx2 video: the three carrying phases contain their work and the load keeps its order` -- 3,764 to 27,975 draws per run, measured. `SampleNow()` takes the phase mutex and runs `SampleLocked()`, which invokes the installed `DeviceByteProbe`; `FixtureParams` sets `mp.device = 0` and `src/vllm/multimodal/ltx2_video.cpp` installs a probe calling `vt::Backend::DeviceMemoryInfo`. On the lanes the review could run this is harmless because `CudaBackend` does not override it and returns the -1 sentinel ([#1126](https://github.com/mudler/vllm.cpp/issues/1126)), but `src/vt/rocm/rocm_backend.hip:373` DOES override it. If the reading is right the consequence runs in two directions at once: the render slows because it contends for that mutex against a thread holding it across a driver round trip, and the sampler's own draws inflate, widening the very bound (1c) asserts against. What would settle it, cheaply: one lease on a ROCm device running that case and comparing the printed whole-case boundary median and maximum against this box's 18.4-22.9 us / 0.94-17.5 ms, plus the case wall. If it reproduces, the repair is a null device probe for the sampler's lifetime, or a device-counter cadence far below the boundary-timing cadence. NOT FIXED IN FLOW: the box that found it has no ROCm device and the session was CPU-only, so both reproduction and repair are unmeasured. Filed rather than left in a review transcript, because a 1 kHz driver call under a shared mutex is the kind of thing discovered as "the ROCm CI is slow" months later. Owned by [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) `## Owed` | bug | diff --git a/.agents/specs/ltx25-device-residency.md b/.agents/specs/ltx25-device-residency.md index 02352a7fd..d8e014178 100644 --- a/.agents/specs/ltx25-device-residency.md +++ b/.agents/specs/ltx25-device-residency.md @@ -3232,28 +3232,40 @@ printed beside it were two separate reads of a moving quantity, so the report could disagree with the arithmetic it described. A per-record window fixes both by construction. -#### The skip path was a permanent report-only state, which `AGENTS.md` refuses +#### The skip path was a permanent report-only state, which `AGENTS.md` refuses -- and the assertion that answered it is NOT in the tree -A record is checked only when `record_seconds >= 8 x ceiling`, `span_checked` -and `span_unresolvable` were computed, printed and **asserted nowhere**, and the -lane that matters discards the printing: CI runs `ctest --output-on-failure`, so +The observation holds against `main` and is not about the withdrawn shape: +`span_checked` and `span_unresolvable` are computed (`test_ltx2_video.cpp:3972`, +`:3983`), printed once in a `MESSAGE` (`:3999-4001`) and **asserted nowhere**, +and the lane that matters discards the printing: CI runs `ctest --output-on-failure`, so on a green run nobody ever sees the line. The reviewer measured four `CheckCarryingPhase` calls reporting `1 of 1`, `0 of 2`, `0 of 1`, `0 of 1` while the case exited `Status: SUCCESS!` with a 20 ms un-named phase present. `AGENTS.md` `## Gates` refuses that in as many words: *"Report exactly one result for each applicable rule ... A permanent report-only state is not a result."* -The assertion added is `longest_checked`: **(1c) must have resolved the LONGEST -record of each carrying leaf.** It names a record rather than a count or a share, -which is what keeps it free of a new constant. The longest record is the one that -can hide the most AND the one most likely to satisfy `record >= 8 x its own -window's worst boundary`, so the two orderings agree and demanding it is both the -strongest and the cheapest thing to demand. A count floor would be a constant; a -seconds share would be a constant; "the biggest one" is neither. - -A record too short to hold a single draw of a 1 kHz sampler is counted and -reported separately from a record the bound cannot resolve, because those are -different facts about the instrument. +The skip condition that goes with it is `main`'s, not the withdrawn shape's: +a record is checked only when `record_seconds >= 2 * kSpanSlackPerRecord`, the +flat 60 ms, which is what `span_bound < kSpanSlackPerRecord` tests at +`test_ltx2_video.cpp:3971`. The `8 x ceiling` form below belongs to the +withdrawn per-record window and never landed. + +**The assertion the withdrawn shape added was `longest_checked`** -- (1c) must +have resolved the LONGEST record of each carrying leaf -- **and it is NOT in the +tree**, because it went out with the shape it was built on. It named a record +rather than a count or a share, which is what kept it free of a new constant: +the longest record is the one that can hide the most AND the one most likely to +satisfy `record >= 8 x its own window's worst boundary`, so the two orderings +agreed and demanding it was both the strongest and the cheapest thing to demand. +A count floor would be a constant; a seconds share would be a constant; "the +biggest one" is neither. **So the report-only state above is still open on +`main`**, and the argument for this shape of assertion is what survives, not the +assertion. + +In that shape a record too short to hold a single draw of a 1 kHz sampler was +counted and reported separately from a record the bound could not resolve, +because those are different facts about the instrument. That separation is not +in the tree either; `main` counts `span_unresolvable` alone. #### Two more findings, repaired in the withdrawn shape, NONE of which is in the tree From 5521cc5b429f8e8d97e3eda2fc5a92bbac2b591d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 23 Aug 2026 01:49:22 +0000 Subject: [PATCH 9/9] record(LTX25-DEVICE-RESIDENCY): the two counter anchors were bound to each other's names A round-3 fresh review passed the substance and found the name-to-anchor mapping reversed. The sentence read "`span_checked` and `span_unresolvable` are computed (`test_ltx2_video.cpp:3972`, `:3983`)". Both anchors are real increment sites, but on `origin/main` `:3972` is `++span_unresolvable` and `:3983` is `++span_checked`, so a reader pairing them in order got them backwards. Neither anchor was wrong and no claim was false, which is exactly what makes it worth one line: a wrong-but-plausible anchor is the failure this row already spent a full review round on, and a positional pairing is the shape that goes stale silently. Each anchor now sits beside its own name, so the order carries no meaning and cannot be misread. Also gave the withdrawn quantity one name. It was "the `8 x ceiling` form" eight lines above "`record >= 8 x its own window's worst boundary`", which are the same thing said twice; the first now names the record's own window too, so nobody has to work out that the two mentions are one quantity. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-device-residency.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/.agents/specs/ltx25-device-residency.md b/.agents/specs/ltx25-device-residency.md index d8e014178..bdc753539 100644 --- a/.agents/specs/ltx25-device-residency.md +++ b/.agents/specs/ltx25-device-residency.md @@ -3235,10 +3235,10 @@ by construction. #### The skip path was a permanent report-only state, which `AGENTS.md` refuses -- and the assertion that answered it is NOT in the tree The observation holds against `main` and is not about the withdrawn shape: -`span_checked` and `span_unresolvable` are computed (`test_ltx2_video.cpp:3972`, -`:3983`), printed once in a `MESSAGE` (`:3999-4001`) and **asserted nowhere**, -and the lane that matters discards the printing: CI runs `ctest --output-on-failure`, so -on a green run nobody ever sees the line. The reviewer measured four +`span_unresolvable` (`test_ltx2_video.cpp:3972`) and `span_checked` (`:3983`) +are computed, printed once in a `MESSAGE` (`:3999-4001`) and **asserted +nowhere**, and the lane that matters discards the printing: CI runs +`ctest --output-on-failure`, so on a green run nobody ever sees the line. The reviewer measured four `CheckCarryingPhase` calls reporting `1 of 1`, `0 of 2`, `0 of 1`, `0 of 1` while the case exited `Status: SUCCESS!` with a 20 ms un-named phase present. `AGENTS.md` `## Gates` refuses that in as many words: *"Report exactly one result @@ -3247,8 +3247,9 @@ for each applicable rule ... A permanent report-only state is not a result."* The skip condition that goes with it is `main`'s, not the withdrawn shape's: a record is checked only when `record_seconds >= 2 * kSpanSlackPerRecord`, the flat 60 ms, which is what `span_bound < kSpanSlackPerRecord` tests at -`test_ltx2_video.cpp:3971`. The `8 x ceiling` form below belongs to the -withdrawn per-record window and never landed. +`test_ltx2_video.cpp:3971`. The `8 x` form below -- a record against its OWN +window's worst boundary -- belongs to the withdrawn per-record window and never +landed. **The assertion the withdrawn shape added was `longest_checked`** -- (1c) must have resolved the LONGEST record of each carrying leaf -- **and it is NOT in the