fix(mem): route ordinary malloc through mimalloc on Linux (#1360) - #1477
Merged
Conversation
The comment above MIMALLOC_OVERRIDE_DEFINE claimed Unix "relies on static-link-order override". That was never true. mimalloc emits strong malloc/free definitions only when MI_MALLOC_OVERRIDE is set — the define its source gates alloc-override.c on — and the build set it for MinGW only. MI_OVERRIDE, which the build does set everywhere, is this project's own prod/test marker that mimalloc never reads. With the override body compiled out there were no strong symbols for link order to prefer, so ordinary malloc went to libc. Measured, not inferred. The shipped v0.9.1-rc.1 artifacts report 0/6 allocator-owned size classes on linux-arm64 glibc AND musl-static, and the same A/B on ubuntu-arm64 here: main warn mem.allocator.not_owned owned_classes=0/6 with this fix info mem.allocator.owned classes=all Every purge/reclaim option cbm_mem_init sets was therefore inert on Linux, applying only to the bound sqlite/tree-sitter populations — the same class of defect as #581, where committed memory ratcheted for months because nothing asserted the wiring on a real artifact. macOS stays off deliberately and permanently: enabling the override there compiles alloc-override.c's forwarding definitions, and under the two-level namespace this binary's free becomes mi_free while system libraries keep allocating from the system allocator, so the first pointer crossing that boundary aborts with "mi_free: invalid pointer". ELF's flat namespace has no such split, which is why Linux can have this and macOS cannot. Three parts: * Makefile.cbm switches MI_MALLOC_OVERRIDE and a new CBM_MEM_GLOBAL_OVERRIDE together in one place, for MinGW and Linux. The latter tells our own sources what to expect and goes into CFLAGS_PROD only, because MIMALLOC_CFLAGS_TEST builds mimalloc with -DMI_OVERRIDE=0 and no override define: a test binary has no global override BY CONSTRUCTION and must never be told to expect one. Deriving that expectation from the platform instead would make every Linux test run warn about a correctly configured build. * mem.c splits the startup audit by what the build actually asked for. Where no override was requested, ordinary malloc reaching libc is the design, so it reports the measured ownership at INFO and names what IS bound. A warning that fires on every run of a correct build is not a tripwire, it is noise — it trains readers to ignore the one line that catches #581. The genuine warning still fires wherever an override was requested and did not take. * smoke-test.sh Phase 1b pins the SHIPPED artifact's wiring per platform and fails in BOTH directions: Windows/Linux must own all classes, macOS must not. This cannot live in a unit test — a from-source test build never has the override — which is exactly why the defect survived so long. Verified on ubuntu-arm64: prod owns all classes; the test build reports bound-populations-only at INFO (0/6) and the mem suite stays green; smoke Phase 1b passes, and fails with the expected message when the same tree is built with the override forced off. On macOS: prod reports bound_populations_only at INFO, smoke Phase 1b passes, mem suite 51/51. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData
enabled auto-merge
August 6, 2026 16:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The claim that was never true
The comment above
MIMALLOC_OVERRIDE_DEFINEsaid Unix "relies on static-link-order override". It doesn't work that way. mimalloc emits strongmalloc/freedefinitions only whenMI_MALLOC_OVERRIDEis set — the define its source gatesalloc-override.con — and the build set that for MinGW only.MI_OVERRIDE, which the build does set everywhere, is this project's own prod/test marker that mimalloc never reads.With the override body compiled out there were no strong symbols for link order to prefer, so ordinary
mallocwent to libc.Measured, not inferred
The shipped v0.9.1-rc.1 artifacts report 0/6 allocator-owned size classes on linux-arm64 glibc and musl-static. Same A/B reproduced on ubuntu-arm64 for this PR:
warn mem.allocator.not_owned owned_classes=0/6info mem.allocator.owned classes=allSo every purge/reclaim option
cbm_mem_initsets was inert on Linux, applying only to the bound sqlite/tree-sitter populations. Same class of defect as #581, where committed memory ratcheted for months because nothing asserted the wiring on a real artifact.macOS stays off, deliberately and permanently
Enabling the override there compiles
alloc-override.c's forwarding definitions; under the two-level namespace this binary'sfreebecomesmi_freewhile system libraries keep allocating from the system allocator, so the first pointer crossing that boundary aborts withmi_free: invalid pointer. ELF's flat namespace has no such split — which is why Linux can have this and macOS cannot.Three parts
Makefile.cbmswitchesMI_MALLOC_OVERRIDEand a newCBM_MEM_GLOBAL_OVERRIDEtogether in one place, for MinGW and Linux. The latter tells our own sources what to expect and goes intoCFLAGS_PRODonly —MIMALLOC_CFLAGS_TESTbuilds mimalloc with-DMI_OVERRIDE=0and no override define, so a test binary has no global override by construction and must never be told to expect one. Deriving the expectation from the platform instead would make every Linux test run warn about a correctly configured build.mem.csplits the startup audit by what the build actually asked for. Where no override was requested, ordinary malloc reaching libc is the design, so it reports the measured ownership at INFO and names what IS bound. A warning that fires on every run of a correct build is not a tripwire, it's noise — it trains readers to ignore the one line that catches Memory leak: process grows to 50+ GB virtual memory over hours/days, crashes Windows #581. The genuine warning still fires wherever an override was requested and did not take.smoke-test.shPhase 1b pins the SHIPPED artifact's wiring per platform and fails in both directions: Windows/Linux must own all classes, macOS must not. This cannot live in a unit test — a from-source test build never has the override — which is precisely why the defect survived so long.Verification
ubuntu-arm64: prod owns all classes; test build reports bound-populations-only at INFO (0/6) and the
memsuite stays green; smoke Phase 1b passes — and fails with the expected message when the same tree is built with the override forced off (revert-check on the guard itself).macOS: prod reports
bound_populations_onlyat INFO, smoke Phase 1b passes,memsuite 51/51.make lint-ciclean.Worth a maintainer's eye
This changes allocator behaviour in shipped Linux binaries — that is the point, but it means Linux RSS/throughput characteristics now reflect the tuning that was previously inert. Prior Linux memory baselines were taken against an allocator we believed was configured and wasn't, so they aren't directly comparable.