Skip to content

metal: pread pool dispatch stats and IO-tier pinning for streaming experts#570

Open
jasontitus wants to merge 2 commits into
antirez:mainfrom
jasontitus:metal-streaming-io-stats
Open

metal: pread pool dispatch stats and IO-tier pinning for streaming experts#570
jasontitus wants to merge 2 commits into
antirez:mainfrom
jasontitus:metal-streaming-io-stats

Conversation

@jasontitus

Copy link
Copy Markdown

Two independent pieces from the streaming-IO investigation in #533,
refiled without the withdrawn default change (see that thread for the
cross-machine data).

  1. metal: report pread pool dispatch stats in the timing summary
    one extra line under the existing
    DS4_METAL_STREAMING_EXPERT_TIMING_SUMMARY gate: dispatches, tasks,
    bytes, average dispatch wall time, effective queue depth (sum of
    per-task pread ms / pool wall ms) and delivered bandwidth. Sample from
    a ctx-2048 streaming prefill (Flash IQ2XXS imatrix, 33 GiB cache
    budget, --simulate-used-memory 64):

    ds4:   streaming pread pool dispatches=2031 tasks=27279 tasks_avg=13.4 workers_avg=8.5 bytes=59.94 GiB wall_avg=1.286 ms qd_avg=5.82 pool_gbps=24.64 task_gbps=4.23
    

    This line is what surfaced both halves of the metal: stop expert-miss readahead racing the pread pool (+13% GLM streaming decode) #533 result (the
    glm5.2-line eval-thread stall and the main-side prefetch win); it
    makes streaming IO regressions visible without dtrace.

  2. metal: keep expert pread threads on the important IO tier — reader
    threads inherit the parent QoS, so under taskpolicy -b or background
    launchd jobs every expert miss pays the throttled IO tier. Pin them to
    user-initiated QoS + IOPOL_IMPORTANT;
    DS4_METAL_DISABLE_STREAMING_EXPERT_PREAD_QOS opts out. Foreground
    runs are unaffected (measured neutral on this machine and
    independently on an M5 Pro in the metal: stop expert-miss readahead racing the pread pool (+13% GLM streaming decode) #533 thread).

Machine: M1 Ultra Mac Studio 128 GB / 8 TB SSD, macOS 15, Metal backend
(one machine only).

Tests: make clean && make (zero warnings); ds4_test --server and
ds4_test --metal-kernels OK; --logprob-vectors pre-existing ERR
(long_code_audit) with the IQ2XXS imatrix quant, identical on unpatched
main.

One extra line next to the streaming expert timing report (same
DS4_METAL_STREAMING_EXPERT_TIMING_SUMMARY gate): dispatches, tasks,
bytes, average dispatch wall time, effective queue depth
(qd_avg = sum of per-task pread ms / pool wall ms) and delivered
bandwidth. Makes streaming IO regressions measurable without dtrace.
Reader threads inherit the parent QoS: launched via taskpolicy -b or a
background launchd job they end up on the throttled IO tier and every
expert miss pays for it. Pin them to user-initiated QoS and
IOPOL_IMPORTANT; DS4_METAL_DISABLE_STREAMING_EXPERT_PREAD_QOS opts
out.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant