(feat): add 2-tier pool safety — compile-time escape detection + runtime validation by mgyoo86 · Pull Request #25 · ProjectTorreyPines/AdaptiveArrayPools.jl

mgyoo86 · 2026-03-10T00:26:55Z

Summary

Add a two-layer safety system: compile-time AST analysis catches escapes before any code runs, and runtime POOL_SAFETY_LV levels provide progressive protection from guard invalidation to full borrow tracking.

Key changes

1. Compile-time escape detection (`STATIC_POOL_CHECKS`)

The @with_pool macro analyzes the body AST at macro-expansion time and throws PoolEscapeError if a pool-backed variable escapes:

# Compile-time error — zero runtime cost
@with_pool pool begin
    v = acquire!(pool, Float64, 10)
    v  # ← PoolEscapeError: pool variable `v` would escape scope
end

# Safe: return a scalar, not the pool-backed array
@with_pool pool begin
    v = acquire!(pool, Float64, 10)
    v .= 1.0
    sum(v)  # ← OK: returns Float64
end

Detects: bare returns, return v, tuple/array containers (v, w), aliases, destructuring, and explicit return in all branches (if/else, loops, ternary).

Gated by STATIC_POOL_CHECKS (compile-time const via LocalPreferences.toml: pool_checks). When false, all safety code — both compile-time and runtime — is elided.

2. POOL_SAFETY_LV — runtime safety levels

POOL_SAFETY_LV[] = 0  # off
POOL_SAFETY_LV[] = 1  # guard: invalidate released slots on rewind (default)
POOL_SAFETY_LV[] = 2  # full: guard + escape detection + NaN/sentinel poisoning
POOL_SAFETY_LV[] = 3  # debug: full + borrow registry (callsite tracking)

LV 1 — Guard invalidation: On rewind, released slots are structurally invalidated (resize! + setfield!) so stale references immediately error instead of silently reading corrupted data. Backported to legacy (Julia 1.10) path.

LV 2 — Runtime escape detection: _validate_pool_return recursively inspects return values (Tuple, NamedTuple, Pair, Dict, Set, Vector) for pool-backed arrays via pointer overlap. Throws PoolRuntimeEscapeError with type/size info. NaN/sentinel poisoning fills released memory.

LV 3 — Borrow registry: Tracks which acquire! call produced each pooled array. On escape, the error message includes exact source locations:

PoolEscapeError: Returning pool-backed SubArray{Float64,1} (size: (10,))
  acquired at: src/model.jl:42
      acquire!(pool, Float64, 10)
  returned at: src/model.jl:48
      return (result, v)

3. Tag dispatch for safety helpers

Replaced ~50 repetitive @static if STATIC_POOL_CHECKS blocks with Julia-idiomatic multiple dispatch on singleton tags:

struct _CheckOn end    # safety checks enabled
struct _CheckOff end   # safety checks disabled → all helpers become no-ops

const _POOL_CHECK_TAG = STATIC_POOL_CHECKS ? _CheckOn() : _CheckOff()

@inline _set_pending_callsite!(::_CheckOn, pool, msg) = ...   # active
@inline _set_pending_callsite!(::_CheckOff, ::AbstractArrayPool, ::String) = nothing

@with_pool and @maybe_with_pool function definition forms were missing POOL_DEBUG escape detection. Block forms had it, but the 4 function code-generation paths (_generate_function_pool_code and _generate_function_pool_code_with_backend, force_enable=true/false) did not capture _result or call _validate_pool_return. Also fix latent bug in _check_pointer_overlap where iterating BitTypedPool.vectors called pointer(::BitVector) which is undefined. Added `v isa Array || continue` guard to skip BitVector entries.

Split src/utils.jl into two focused files: - src/debug.jl: POOL_DEBUG, _validate_pool_return, _check_pointer_overlap - src/utils.jl: pool_stats, Base.show (statistics and display only) debug.jl is included after bitarray.jl, eliminating the forward reference to _check_bitchunks_overlap that existed when this code lived in utils.jl. Corresponding test split: - test/test_debug.jl: 17 testsets for escape detection - test/test_utils.jl: 8 testsets for statistics and display

…wind Add 2-tier safety toggle: STATIC_POOL_CHECKS (compile-time via @load_preference) gates POOL_SAFETY (runtime Ref, levels 0/1/2). Level 1 (guard, default): on rewind/reset, released slots are structurally invalidated — backing vectors resize!'d to 0 and cached Array/BitArray wrappers get setfield! size zeroed. Stale references throw BoundsError instead of silently returning corrupted data. Level 2 extends Level 1 with escape detection (POOL_DEBUG backward compat preserved: both POOL_SAFETY>=2 and POOL_DEBUG[] trigger _validate_pool_return). Zero-alloc round-trip preserved: resize!(v,0) keeps capacity, so re-acquire restores length within allocation.

…tency

Add structural invalidation on rewind for the legacy (Julia 1.10) code path: - TypedPool: resize! backing vectors to 0 + reset view_lengths to force SubArray cache misses (Array setfield! unavailable on 1.10) - BitTypedPool: resize! BitVectors to 0 + zero nd_ptrs to force N-way cache misses + setfield! BitArray len/dims to 0 - Version-gate unsafe_acquire! Array wrapper tests to Julia 1.11+ - Disable POOL_SAFETY_LV during zero-allocation tests (invalidation forces cache misses → new allocations on legacy)

Poison released backing vectors with detectable sentinel values (NaN for floats, typemax for integers) before structural invalidation. Shared _poison_released_vectors! in debug.jl, called from per-path _invalidate_released_slots! when POOL_SAFETY_LV[] >= 2.

AST analysis at macro expansion time detects pool-backed variables in return expressions. Definite escapes (bare `v`, `return v`) throw compile-time errors; possible escapes (containers) emit warnings. Zero runtime cost — runs only during compilation. Runtime guard tests updated to use identity() bypass so they properly exercise _validate_pool_return at LV2.

…case tests Add 79 new test assertions (11→90 total) covering all macro forms, escape scenarios, false-positive prevention, and error message verification: - Block/function forms for @with_pool, @maybe_with_pool, backend (:cpu) - Tricky-but-safe true negative edge cases (pipe, comprehension, ternary, broadcast, copy, similar, let/do blocks, Dict, string interpolation) - NamedTuple key-name-vs-value distinction, reassignment chain tracking - Nested scope detection, error/warning message content verification

…ring support - Promote tuple/array/NamedTuple container escapes from warning to error (zero false-positive risk after systematic analysis of all reassignment patterns) - Add tuple destructuring support to _extract_acquired_vars and _remove_flat_reassigned! — handles (a, v) = expr and v, x = x, v swap - Remove _is_definite_escape (no longer needed — all escapes are errors) - Unify error message: lists all escaped variables with collect() guidance, singular/plural aware

Replace flat error() string with custom PoolEscapeError exception type and Base.showerror using printstyled for terminal-aware colored output: - Bold red variable names, gray annotations and file path - Multi-line layout with per-variable listing - Two-tier suggestions: trace definitions + identity() escape hatch - Tests use type matching (@test_throws PoolEscapeError) and field inspection

…ernary) Previously only checked the last expression of the block body, missing explicit `return` inside if/else branches, loops, and try/catch. Add _collect_all_return_values with two-source scanning: - _collect_explicit_returns!: walks AST for all `return expr` (skips nested fns) - _collect_implicit_return_values!: recurses into if/elseif/else branch tails

…essions Add EscapePoint tracking and _render_return_expr for rich diagnostics: - Each problematic return shown with source line and expression - Escaped variable names highlighted in red within the expression context - NamedTuple keys stay white, only values get colored (v = v vs v = v) - Line numbers tracked via LineNumberNode walk in _collect_*_returns!

…nction

… and identity transparency - Track simple aliases (d = z), container wrapping (d = (z,), d = [z]), and identity() calls as acquired variables - Recurse into nested containers in _find_direct_exposure for deep escape detection - Add @skip_check_vars macro to suppress false positives for listed variables - Make identity() transparent in escape analysis (no longer serves as escape hatch) - Suppress stacktrace via 3-arg Base.showerror for PoolEscapeError - Color PoolEscapeError header red and improve Fix/False-positive messages

…ontainers Runtime LV2 escape detection now inspects inside common container types, catching pool-backed arrays hidden in return values like (sum(v), v) or (data=v, n=10). Also fix identity() transparency tests to use @skip_check_vars for compile-time suppression.

Add _validate_pool_return dispatches for AbstractDict and AbstractSet, and recurse into Array elements when eltype may contain arrays. Includes isconcretetype guard for pointer overlap and _eltype_may_contain_arrays guard to skip leaf-type element iteration.

Replace generic "returned an Array backed by pool memory" with descriptive messages using summary(val) showing exact type, dimensions, and pool type. Pass original_val through overlap checks so SubArray/ReshapedArray cases report the user-visible value, not the internal parent vector. Add LV=3 hint for future acquire-site tracking.

…atted escape messages Introduce PoolRuntimeEscapeError exception type with colored showerror matching compile-time PoolEscapeError style. Shows escaped array summary, pool type, fix suggestions, and LV=3 hint. Suppress stacktrace for cleaner output.

…_LV=3) At LV=3, escape detection errors now show WHERE the problematic acquire! originated (file:line), not just WHAT escaped. Uses a two-field design: _pending_callsite (set by macro before each acquire) + _borrow_log (IdDict keyed by vector identity, lazily initialized). Macro AST pass (_inject_pending_callsite) walks block statements tracking LineNumberNodes to inject callsite assignments. Public API functions use isempty() guard to only set generic fallback labels for direct calls. Zero overhead at LV<3 (compile-time gated + branch-predicted away).

…tection Transform `return expr` in macro-generated code to validate the return value before exiting, preventing pool-backed arrays from bypassing _validate_pool_return via explicit return. Stops recursion at :function and :-> boundaries to respect nested function semantics.

…acking Show the complete source statement (e.g. `v = acquire!(pool, Float64, 10)`) instead of just the acquire call. Add return-site tracking so escape errors display both where the array was acquired and where it escapes via `return`.

…r/alias Enhance compile-time PoolEscapeError messages to distinguish between: - pool-acquired view (acquire!, zeros!, ones!, similar!, reshape!) - pool-acquired array (unsafe_acquire!, acquire_array!) - pool-acquired BitArray (trues!, falses!) - container wrapping pool variables (e.g., a = [v, 1]) - alias of pool variable (e.g., d = v) Fix suggestions now target the actual pool variables (not containers), and note when pool vars should be copied before wrapping.

Return PoolEscapeError from _check_compile_time_escape instead of throwing directly, so the macro returns :(throw(err)) — placing the throw at the user's code location. This keeps only Base-internal frames in the backtrace, which Julia's show_backtrace filters out.

Remove @skip_check_vars macro (and its export) since the escape detector no longer supports suppression annotations. Tests now use _test_leak(x) = x as an opaque function to bypass compile-time escape analysis while still exercising runtime LV2 detection.

Extract and display where each escaping variable was assigned (acquire!, alias, container) with source file and line locations. Improves error readability with descriptive header, declaration section, and location markers on escape return points.

Introduce _CheckOn/_CheckOff singleton tags and _POOL_CHECK_TAG const for idiomatic Julia multiple dispatch instead of @static if blocks. Replace ~50 repetitive 3-line @static if STATIC_POOL_CHECKS blocks across acquire.jl, bitarray.jl, and convenience.jl with single-line _set_pending_callsite! and _maybe_record_borrow! helper calls. Net result: -33 lines (99 insertions, 132 deletions).

Copilot

Pull request overview

This PR introduces a two-tier pool safety system: compile-time AST escape detection (STATIC_POOL_CHECKS / PoolEscapeError) that catches pool variable escapes at macro expansion with zero runtime cost, and a new runtime safety level system (POOL_SAFETY_LV 0–3) that provides progressive protections including structural invalidation, NaN/sentinel poisoning of released memory, and a borrow registry for acquire call-site tracking in error messages. The POOL_DEBUG flag is retained as a legacy backward-compatible alias for escape detection.

Changes:

New POOL_SAFETY_LV runtime enum with structural invalidation (LV1), poisoning (LV2), and borrow registry (LV3); replaces/supersedes POOL_DEBUG
Compile-time escape detection via _check_compile_time_escape AST analysis at macro expansion time, emitting PoolEscapeError
New test files (test_safety.jl, test_debug.jl, test_compile_escape.jl, test_borrow_registry.jl) plus migration of old debug tests from test_utils.jl

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
`src/types.jl`	Adds `STATIC_POOL_CHECKS`, `POOL_SAFETY_LV`, borrow registry fields, and tag dispatch helpers
`src/debug.jl`	New file: `POOL_DEBUG`, `_validate_pool_return`, `PoolRuntimeEscapeError`, poisoning, and borrow registry
`src/state.jl`	Structural invalidation (`_invalidate_released_slots!`) on rewind and reset
`src/macros.jl`	Compile-time escape analysis, `PoolEscapeError`, callsite injection, return stmt transformation
`src/acquire.jl`	Borrow tracking hooks (`_set_pending_callsite!`, `_maybe_record_borrow!`) on each acquire
`src/convenience.jl`	Borrow tracking hooks on all convenience acquire functions
`src/bitarray.jl`	Updated escape error reporting with `PoolRuntimeEscapeError`
`src/utils.jl`	Removed old `_validate_pool_return` implementation (moved to `debug.jl`)
`src/AdaptiveArrayPools.jl`	Exports new public API; includes `debug.jl`
`src/legacy/types.jl`	Adds borrow registry fields to legacy `AdaptiveArrayPool`
`src/legacy/state.jl`	Structural invalidation on legacy path; handles no `setfield!(:size)` on 1.10
`src/legacy/bitarray.jl`	Updated escape error reporting
`test/runtests.jl`	Adds new test files to both branches
`test/test_safety.jl`	New: invalidation level tests
`test/test_debug.jl`	New: runtime escape validation tests (moved + extended from test_utils.jl)
`test/test_compile_escape.jl`	New: comprehensive compile-time escape detection tests
`test/test_borrow_registry.jl`	New: LV=3 borrow tracking tests
`test/test_utils.jl`	Removed old debug tests
`test/test_state.jl`	Temporarily disables invalidation in capacity-preserving tests
`test/test_basic.jl`	Temporarily disables invalidation in backing-vector size tests
`test/test_allocation.jl`	Disables invalidation for zero-alloc reuse tests
`test/test_backend_macro_expansion.jl`	Updates assertion to reflect new `_validate_pool_return` in generated code

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/legacy/state.jl

test/test_safety.jl

src/types.jl

src/macros.jl

src/debug.jl

test/test_borrow_registry.jl

test/runtests.jl

src/debug.jl

src/macros.jl

src/state.jl

- Change `return PoolEscapeError(...)` back to `throw(PoolEscapeError(...))` in `_check_compile_time_escape` (regression from 0a474ef broke all @test_throws) - Add _CheckOn/_CheckOff tag dispatch infrastructure to legacy/types.jl so shared convenience.jl can call _set_pending_callsite! on Julia < 1.11 - Fix test string: "escape the" → "escapes the" matching updated showerror

…anup - Clear _pending_callsite in _record_borrow_from_pending! after consuming, so subsequent acquire! calls get fresh callsite (was stale after first) - Add _pending_return_site clearing to reset! (modern + legacy paths) - Add full borrow registry cleanup to legacy reset! (was missing entirely) - Fix POOL_SAFETY_LV comment: mention level 3 (borrow registry)

codecov · 2026-03-10T01:48:07Z

Codecov Report

❌ Patch coverage is 96.81093% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.41%. Comparing base (e296691) to head (4002151).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
src/legacy/state.jl	87.75%	6 Missing ⚠️
src/legacy/types.jl	45.45%	6 Missing ⚠️
src/debug.jl	97.64%	4 Missing ⚠️
src/macros.jl	99.22%	4 Missing ⚠️
src/state.jl	92.30%	4 Missing ⚠️
src/types.jl	60.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #25      +/-   ##
==========================================
+ Coverage   95.92%   96.41%   +0.48%     
==========================================
  Files          13       14       +1     
  Lines        1768     2591     +823     
==========================================
+ Hits         1696     2498     +802     
- Misses         72       93      +21

Files with missing lines	Coverage Δ
src/AdaptiveArrayPools.jl	`100.00% <ø> (ø)`
src/acquire.jl	`90.95% <100.00%> (+1.54%)`	⬆️
src/bitarray.jl	`94.44% <100.00%> (+0.61%)`	⬆️
src/convenience.jl	`98.34% <100.00%> (+0.21%)`	⬆️
src/legacy/bitarray.jl	`94.28% <100.00%> (-5.72%)`	⬇️
src/utils.jl	`95.18% <ø> (+0.78%)`	⬆️
src/debug.jl	`97.64% <97.64%> (ø)`
src/macros.jl	`98.25% <99.22%> (+3.51%)`	⬆️
src/state.jl	`97.02% <92.30%> (-1.14%)`	⬇️
src/types.jl	`88.46% <60.00%> (-6.78%)`	⬇️
... and 2 more

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

mgyoo86 added 27 commits March 7, 2026 13:51

refactor: rename POOL_SAFETY to POOL_SAFETY_LV for clarity and consis…

fabdc97

…tency

fix: enhance error message formatting in PoolEscapeError showerror fu…

237e662

…nction

Runic formatting

9f578eb

mgyoo86 requested a review from Copilot March 10, 2026 00:27

Copilot started reviewing on behalf of mgyoo86 March 10, 2026 00:27 View session

Copilot AI reviewed Mar 10, 2026

View reviewed changes

mgyoo86 added 3 commits March 9, 2026 18:30

Runic formatting

1984c3a

mgyoo86 added 2 commits March 9, 2026 19:19

increase test coverage

78e23b7

Runic formatting

4002151

mgyoo86 merged commit 33b1f65 into master Mar 10, 2026
11 checks passed

mgyoo86 deleted the feat/debug_mode branch March 10, 2026 02:33

mgyoo86 referenced this pull request Mar 11, 2026

chore: bump version to 0.2.3 in Project.toml

a3da1a3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(feat): add 2-tier pool safety — compile-time escape detection + runtime validation#25

(feat): add 2-tier pool safety — compile-time escape detection + runtime validation#25
mgyoo86 merged 32 commits intomasterfrom
feat/debug_mode

mgyoo86 commented Mar 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Mar 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mgyoo86 commented Mar 10, 2026

Summary

Key changes

1. Compile-time escape detection (STATIC_POOL_CHECKS)

2. POOL_SAFETY_LV — runtime safety levels

3. Tag dispatch for safety helpers

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

1. Compile-time escape detection (`STATIC_POOL_CHECKS`)

codecov bot commented Mar 10, 2026 •

edited

Loading