metal: pread pool dispatch stats and IO-tier pinning for streaming experts#570
Open
jasontitus wants to merge 2 commits into
Open
metal: pread pool dispatch stats and IO-tier pinning for streaming experts#570jasontitus wants to merge 2 commits into
jasontitus wants to merge 2 commits into
Conversation
One extra line next to the streaming expert timing report (same DS4_METAL_STREAMING_EXPERT_TIMING_SUMMARY gate): dispatches, tasks, bytes, average dispatch wall time, effective queue depth (qd_avg = sum of per-task pread ms / pool wall ms) and delivered bandwidth. Makes streaming IO regressions measurable without dtrace.
Reader threads inherit the parent QoS: launched via taskpolicy -b or a background launchd job they end up on the throttled IO tier and every expert miss pays for it. Pin them to user-initiated QoS and IOPOL_IMPORTANT; DS4_METAL_DISABLE_STREAMING_EXPERT_PREAD_QOS opts out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent pieces from the streaming-IO investigation in #533,
refiled without the withdrawn default change (see that thread for the
cross-machine data).
metal: report pread pool dispatch stats in the timing summary —
one extra line under the existing
DS4_METAL_STREAMING_EXPERT_TIMING_SUMMARYgate: dispatches, tasks,bytes, average dispatch wall time, effective queue depth (sum of
per-task pread ms / pool wall ms) and delivered bandwidth. Sample from
a ctx-2048 streaming prefill (Flash IQ2XXS imatrix, 33 GiB cache
budget,
--simulate-used-memory 64):This line is what surfaced both halves of the metal: stop expert-miss readahead racing the pread pool (+13% GLM streaming decode) #533 result (the
glm5.2-line eval-thread stall and the main-side prefetch win); it
makes streaming IO regressions visible without dtrace.
metal: keep expert pread threads on the important IO tier — reader
threads inherit the parent QoS, so under
taskpolicy -bor backgroundlaunchd jobs every expert miss pays the throttled IO tier. Pin them to
user-initiated QoS +
IOPOL_IMPORTANT;DS4_METAL_DISABLE_STREAMING_EXPERT_PREAD_QOSopts out. Foregroundruns are unaffected (measured neutral on this machine and
independently on an M5 Pro in the metal: stop expert-miss readahead racing the pread pool (+13% GLM streaming decode) #533 thread).
Machine: M1 Ultra Mac Studio 128 GB / 8 TB SSD, macOS 15, Metal backend
(one machine only).
Tests:
make clean && make(zero warnings);ds4_test --serverandds4_test --metal-kernelsOK;--logprob-vectorspre-existing ERR(long_code_audit) with the IQ2XXS imatrix quant, identical on unpatched
main.