Skip to content

fix(mem): route ordinary malloc through mimalloc on Linux (#1360) - #1477

Merged
DeusData merged 1 commit into
mainfrom
fix/linux-allocator-override
Aug 7, 2026
Merged

fix(mem): route ordinary malloc through mimalloc on Linux (#1360)#1477
DeusData merged 1 commit into
mainfrom
fix/linux-allocator-override

Conversation

@DeusData

@DeusData DeusData commented Aug 6, 2026

Copy link
Copy Markdown
Owner

The claim that was never true

The comment above MIMALLOC_OVERRIDE_DEFINE said Unix "relies on static-link-order override". It doesn't work that way. mimalloc emits strong malloc/free definitions only when MI_MALLOC_OVERRIDE is set — the define its source gates alloc-override.c on — and the build set that for MinGW only. MI_OVERRIDE, which the build does set everywhere, is this project's own prod/test marker that mimalloc never reads.

With the override body compiled out there were no strong symbols for link order to prefer, so ordinary malloc went to libc.

Measured, not inferred

The shipped v0.9.1-rc.1 artifacts report 0/6 allocator-owned size classes on linux-arm64 glibc and musl-static. Same A/B reproduced on ubuntu-arm64 for this PR:

build startup audit
main warn mem.allocator.not_owned owned_classes=0/6
this PR info mem.allocator.owned classes=all

So every purge/reclaim option cbm_mem_init sets was inert on Linux, applying only to the bound sqlite/tree-sitter populations. Same class of defect as #581, where committed memory ratcheted for months because nothing asserted the wiring on a real artifact.

macOS stays off, deliberately and permanently

Enabling the override there compiles alloc-override.c's forwarding definitions; under the two-level namespace this binary's free becomes mi_free while system libraries keep allocating from the system allocator, so the first pointer crossing that boundary aborts with mi_free: invalid pointer. ELF's flat namespace has no such split — which is why Linux can have this and macOS cannot.

Three parts

  1. Makefile.cbm switches MI_MALLOC_OVERRIDE and a new CBM_MEM_GLOBAL_OVERRIDE together in one place, for MinGW and Linux. The latter tells our own sources what to expect and goes into CFLAGS_PROD only — MIMALLOC_CFLAGS_TEST builds mimalloc with -DMI_OVERRIDE=0 and no override define, so a test binary has no global override by construction and must never be told to expect one. Deriving the expectation from the platform instead would make every Linux test run warn about a correctly configured build.

  2. mem.c splits the startup audit by what the build actually asked for. Where no override was requested, ordinary malloc reaching libc is the design, so it reports the measured ownership at INFO and names what IS bound. A warning that fires on every run of a correct build is not a tripwire, it's noise — it trains readers to ignore the one line that catches Memory leak: process grows to 50+ GB virtual memory over hours/days, crashes Windows #581. The genuine warning still fires wherever an override was requested and did not take.

  3. smoke-test.sh Phase 1b pins the SHIPPED artifact's wiring per platform and fails in both directions: Windows/Linux must own all classes, macOS must not. This cannot live in a unit test — a from-source test build never has the override — which is precisely why the defect survived so long.

Verification

ubuntu-arm64: prod owns all classes; test build reports bound-populations-only at INFO (0/6) and the mem suite stays green; smoke Phase 1b passes — and fails with the expected message when the same tree is built with the override forced off (revert-check on the guard itself).

macOS: prod reports bound_populations_only at INFO, smoke Phase 1b passes, mem suite 51/51. make lint-ci clean.

Worth a maintainer's eye

This changes allocator behaviour in shipped Linux binaries — that is the point, but it means Linux RSS/throughput characteristics now reflect the tuning that was previously inert. Prior Linux memory baselines were taken against an allocator we believed was configured and wasn't, so they aren't directly comparable.

The comment above MIMALLOC_OVERRIDE_DEFINE claimed Unix "relies on
static-link-order override". That was never true. mimalloc emits strong
malloc/free definitions only when MI_MALLOC_OVERRIDE is set — the define its
source gates alloc-override.c on — and the build set it for MinGW only.
MI_OVERRIDE, which the build does set everywhere, is this project's own
prod/test marker that mimalloc never reads. With the override body compiled
out there were no strong symbols for link order to prefer, so ordinary malloc
went to libc.

Measured, not inferred. The shipped v0.9.1-rc.1 artifacts report 0/6
allocator-owned size classes on linux-arm64 glibc AND musl-static, and the
same A/B on ubuntu-arm64 here:

  main            warn  mem.allocator.not_owned  owned_classes=0/6
  with this fix   info  mem.allocator.owned      classes=all

Every purge/reclaim option cbm_mem_init sets was therefore inert on Linux,
applying only to the bound sqlite/tree-sitter populations — the same class of
defect as #581, where committed memory ratcheted for months because nothing
asserted the wiring on a real artifact.

macOS stays off deliberately and permanently: enabling the override there
compiles alloc-override.c's forwarding definitions, and under the two-level
namespace this binary's free becomes mi_free while system libraries keep
allocating from the system allocator, so the first pointer crossing that
boundary aborts with "mi_free: invalid pointer". ELF's flat namespace has no
such split, which is why Linux can have this and macOS cannot.

Three parts:

* Makefile.cbm switches MI_MALLOC_OVERRIDE and a new CBM_MEM_GLOBAL_OVERRIDE
  together in one place, for MinGW and Linux. The latter tells our own sources
  what to expect and goes into CFLAGS_PROD only, because MIMALLOC_CFLAGS_TEST
  builds mimalloc with -DMI_OVERRIDE=0 and no override define: a test binary
  has no global override BY CONSTRUCTION and must never be told to expect one.
  Deriving that expectation from the platform instead would make every Linux
  test run warn about a correctly configured build.

* mem.c splits the startup audit by what the build actually asked for. Where
  no override was requested, ordinary malloc reaching libc is the design, so
  it reports the measured ownership at INFO and names what IS bound. A warning
  that fires on every run of a correct build is not a tripwire, it is noise —
  it trains readers to ignore the one line that catches #581. The genuine
  warning still fires wherever an override was requested and did not take.

* smoke-test.sh Phase 1b pins the SHIPPED artifact's wiring per platform and
  fails in BOTH directions: Windows/Linux must own all classes, macOS must
  not. This cannot live in a unit test — a from-source test build never has
  the override — which is exactly why the defect survived so long.

Verified on ubuntu-arm64: prod owns all classes; the test build reports
bound-populations-only at INFO (0/6) and the mem suite stays green; smoke
Phase 1b passes, and fails with the expected message when the same tree is
built with the override forced off. On macOS: prod reports
bound_populations_only at INFO, smoke Phase 1b passes, mem suite 51/51.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData
DeusData enabled auto-merge August 6, 2026 16:42
@DeusData
DeusData merged commit 4f7db18 into main Aug 7, 2026
58 of 63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant