An LTX-2.5 render leaks DevicePool blocks from the Gemma-4 rope cache, and the lane that would report it keeps getting cancelled

[localai-bot]

Found while closing a sanitizer gap on main: PR #1036 merged with no completed x86 build-test and no sanitizer verdict because two ci runs fired on the same SHA and the second cancelled the first, then skipped its own build jobs. I ran the lane locally afterwards. The leak it found is not #1036's — see the control below.

The finding

test_ltx2_video under CI's own VLLM_CPP_SANITIZE=address,undefined lane, at main d1b0ea3a8:

[doctest] test cases: 57 | 57 passed | 0 failed | 0 skipped
EXIT=1
==714711==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 65536 byte(s) in 1 object(s) allocated from:
    #0  aligned_alloc
    #1  AllocAligned64            src/vt/cpu/cpu_backend.cpp:20
    #2  Alloc                     src/vt/cpu/cpu_backend.cpp:42
    #3  vllm::DevicePool::Get     include/vllm/model_executor/models/device_pool.h
    #4  vllm::dense_attn::DBuf::DBuf
    #5  BuildProportionalRopeCache  src/vllm/model_executor/models/gemma4.cpp:192
    #6  ForwardBody                 src/vllm/model_executor/models/gemma4.cpp:438
    #7  Gemma4Model::ForwardHiddenStates
    #8  vllm::Ltx2EncodePromptToConditioning
    #9  Ltx2VideoEngine::Generate
   #10  RenderPrompt              tests/vllm/multimodal/test_ltx2_video.cpp:3493

50 direct leaks, 91456 bytes total. The path is the text encoder inside an ordinary LTX-2.5 render, not a test fixture.

Note the shape: the doctest summary reads 57 passed | 0 failed and the process exits 1. A reader grepping assertions: or failed sees a clean run. Only the exit code carries the verdict — the same trap this repo has now hit for thrown cases, non-building mutations, and cancelled CI.

Pre-existing, proven rather than argued

Same lane, same configuration, on the parent commit 6621576ac:

PARENT test_ltx2_video EXIT=1
leak reports on PARENT: 1
Direct leak of 65536 byte(s) in 1 object(s)   <- identical
Direct leak of 3200  byte(s) in 2 object(s)   <- identical
Direct leak of 2560  byte(s) in 1 object(s)   <- identical

Identical leaks on both sides. #1036 touched ltx2_video_vae.cpp and test_ltx2_vae.cpp only — never gemma4.cpp or device_pool.h. It is exonerated by measurement.

Why nobody has seen it

CI does run ASAN_OPTIONS: detect_leaks=1 (.github/workflows/ci.yml:1045), so this is a lane that would report it. Of the last 12 main runs: 6 success, 4 cancelled, 1 failure. And on #1036 specifically the lane never ran at all — cancelled at 31% Building with zero tests executed, while gh pr checks rendered that as a failure.

So the detector works and its output is being lost. That is the same mechanism that let #904's use-after-free sit on main for five hours today: the lane was dismissed as known-red rather than read.

What is actually owed

Someone who knows the pool's intent should decide whether this is a leak or a deliberate never-freed cache. DevicePool is a free-list allocator; a pool that retains blocks for reuse and never returns them at exit is a legitimate design that LeakSanitizer will always flag. If that is the intent, the fix is a suppression with a stated reason — not silence, and not detect_leaks=0, which would blind the lane to real leaks.

If it is not intended, then an LTX-2.5 render leaks 91456 bytes per invocation on the text-encoder path.

Either way the current state is the worst one: a real detector firing into a lane whose output is routinely discarded.

Related: #904 (the use-after-free the same lane caught), #949 (a defect class visible only to continue-on-error lanes), #584.

FOLLOWING_AGENTS_PROTOCOL

[localai-bot]

Settled: intentional by design, and a suppression is the wrong fix. Analysis so whoever takes this does not re-derive it.

Why LSan calls it a direct leak rather than "still reachable"

~DevicePool() (include/vllm/model_executor/models/device_pool.h:198-212) only prints optional VT_POOL_STATS output. It never touches free_. So at process exit the std::unordered_map<size_t, std::vector<void*>> free_ (:271) is destroyed, the block pointers it held are discarded with it, and the blocks become unreachable — which is precisely a direct leak, not "still reachable".

That is the documented intent, stated in the file header:

Blocks are never returned to the driver (leak at process exit, like the cublasLt workspace); the pool is bounded by the forward's peak concurrent scratch.

So the three reported blocks (65536, 3200, 2560) are pooled scratch that a DBuf correctly returned via Put, sitting in the free list at exit. Nothing is lost during the run, nothing grows unboundedly, and the rope-cache stack in the report is just whichever allocation happened to be the last owner of that size class. It is not a bug in the Gemma-4 rope cache, and it is not #1036's — the control on parent 6621576ac produced byte-identical leak sizes.

Why a suppression is still the wrong resolution

A suppression makes the finding invisible, and this is the lane that caught a real heap-use-after-free on main (#904, fixed by #936). Teaching the address,undefined lane to stay quiet about DevicePool buys silence in exactly the place that has already earned its keep.

Drain() already exists and already does the right thing (:183-196): it walks free_, calls b.Free(p) on each block, and clears the map. Its own SAFETY note argues the operation is sound:

free_ only ever holds blocks a DBuf already returned, so nothing live is touched. […] because a pool now holds ONE device's blocks, b.Free is guaranteed to be the allocator that made them.

So the candidate fix is calling Drain(*backend_) from ~DevicePool().

The one hazard to check before doing that is static destruction order: the destructor would call into vt::Backend, and if the backend registry were destroyed first this becomes a use-after-free in the teardown path — the very thing the lane exists to catch, introduced by the fix for it. I believe it is safe, because a DevicePool is constructed from a vt::Backend& and therefore always completes construction after it, and function-local statics are destroyed in reverse order of construction completion. But that is an argument, not a measurement, and it must be verified rather than assumed — the same lane, under thread, is what should confirm it.

What a row owes

• Drain from ~DevicePool(), or a reasoned decision that the ordering hazard makes a suppression the lesser evil — with the ordering question measured, not argued.
• A red-first gate: test_ltx2_video under VLLM_CPP_SANITIZE=address,undefined must exit 1 before and 0 after. Note the trap this issue was born from — that suite prints 57 passed | 0 failed with EXIT=1, so the exit code is the only honest signal.
• Confirmation that VT_POOL_BYPASS=1 still behaves (the header says Drain is a no-op there, since Put frees straight through).
• A check of whether AuxPool needs the same treatment.

This needs its own row, spec and fresh review — it changes a shared seam's teardown semantics. Not an in-flow fix.

Related: #904, #936, #516 (pool-device-key.md).