Add lightweight tool-use verifier evaluations / 添加轻量级工具调用验证评估 - #2634

Merged

adibarra merged 115 commits into

mainfrom

experiment/tool-use-eval-full

Sep 1, 2026

Commits on Aug 10, 2026

feat: add stock kimi tool-use eval
adibarra
committed
e268794
refactor: simplify and expose tool-use eval
adibarra
committed
b7d0d7e
chore: merge latest main for release
adibarra
committed
840537b
refactor: isolate and clarify verifier integration
adibarra
committed
da29994
chore: merge latest main before release
adibarra
committed
be0ca08
fix: distinguish successful failure artifact writes
adibarra
committed
b0fd8cc
fix: preserve agentic eval decoding mode
adibarra
committed
1684f55
chore: merge latest main before release
adibarra
committed
0223399
fix: complete eval-only workflow runs
adibarra
committed
5885859
revert: keep eval-only collection scoped
adibarra
committed
83d7f57
chore: merge latest main before review
adibarra
committed
ee80bce
fix: correct verifier failure metadata and links
adibarra
committed
07dc9d9
chore: preserve existing collector test formatting
adibarra
committed
6ec09bb
fix: preserve intended verifier sample count
adibarra
committed
2b6e5e8

Commits on Aug 11, 2026

chore: merge latest main before final review
adibarra
committed
5e3bc30
fix: preserve configurable eval dispatch behavior
adibarra
committed
6793c9e
Merge remote-tracking branch 'origin/main' into feat/tool-use-eval-smoke
adibarra
committed
4ed2975
fix: harden verifier review paths
adibarra
committed
294e39d
fix: scope eval state and links
adibarra
committed
ce37f46
ci: exclude faulty b300 node from slurm
adibarra
committed
405dc0e
ci: apply B300 node exclusions to allocations
adibarra
committed
847982f

Commits on Aug 12, 2026

feat: enable multinode kimi verifier
adibarra
committed
15b0857
fix: harden Kimi eval runtime failures
adibarra
committed
13c5a45
test: capture Kimi response diagnostics
adibarra
committed
134906e
Revert "test: capture Kimi response diagnostics"
adibarra
committed
23851fb
test: capture deterministic Kimi diagnostics
adibarra
committed
9a661a6
fix: enable Kimi structural tool constraints
adibarra
committed
53b3ca8
chore: format Kimi recipe regression
adibarra
committed
6d08f1e
fix: retry transient Kimi verifier downloads
adibarra
committed
f1fb29d
fix: stabilize and clean Kimi verifier
adibarra
committed
0e679c5
Merge remote-tracking branch 'origin/main' into feat/tool-use-eval-smoke
adibarra
committed
34986b9

Commits on Aug 14, 2026

Merge remote-tracking branch 'origin/main' into feat/tool-use-eval-smoke
adibarra
committed
37e7363
fix: preserve tool eval dispatch contracts
adibarra
committed
8952cdc
fix: enable structured Qwen tool calls
adibarra
committed
ba5b363
fix: preserve single-node eval topology metadata
adibarra
committed
a4f69b9
test: cover isolated verifier runtime bootstrap
adibarra
committed
333a032
fix: configure DSV4 tool call parsers
adibarra
committed
26c0154
fix: configure sglang tool call parsers
adibarra
committed
a2ad629
fix: harden tool eval dispatch and artifacts
adibarra
committed
ffe9054
fix: preserve selected SRT eval concurrency
adibarra
committed
209fbf4
fix: preserve dsv4 thinking request mode
adibarra
committed
38a867e
chore: merge main into verifier branch
adibarra
committed
6d19146
feat: add minimax m3 provider smoke
adibarra
committed
ad9512b
fix: preserve minimax m3 generation budget
adibarra
committed
301dcdf
fix: retry truncated minimax response bodies
adibarra
committed
165135e

Commits on Aug 15, 2026

fix: narrow minimax smoke to schema case
adibarra
committed
ef08f9b
fix: remove obsolete gb200 recipe field
adibarra
committed
f6cb909
fix: enable frontend tool call parsing
adibarra
committed
439d884
fix: parse GB200 vendor tool calls
adibarra
committed
4fed421
fix: force guided Kimi tool calls
adibarra
committed
6f62744
fix: guide all GB200 Kimi recipes
adibarra
committed
fbb8fe2
fix: parse GB200 Kimi tool calls
adibarra
committed
92c8c4f
chore: merge current main
adibarra
committed
49fd66f
fix: harden vendor artifact cleanup
adibarra
committed
7edb59b
fix: harden transient endpoint registration retries
adibarra
committed
ade0abd
fix: allow trusted Kimi frontend code
adibarra
committed
2567569
feat: add deterministic bfcl tool smoke
adibarra
committed
a847479
fix: install bfcl audio dependency
adibarra
committed
9fff29e
fix: require externally grounded bfcl calls
adibarra
committed
2550733
fix: harden tool eval failure handling
adibarra
committed
2dec376
feat: add opt-in full verifier suites
adibarra
committed
592215f

Commits on Aug 16, 2026

Commits on Aug 17, 2026

chore: merge main into tool-use experiment
adibarra
committed
a4713af
feat: complete tool-use eval integration
adibarra
committed
7b61603
fix: enable tool parsing for kimi k3
adibarra
committed
f020a8e

Commits on Aug 18, 2026

Merge branch 'main' into experiment/tool-use-eval-full
adibarra
authored
b57d8bb
fix: harden tool use deployment verification
adibarra
committed
d070e3b
fix: normalize infrastructure failure sample counts
adibarra
committed
b8d681f
ci: repair amd workspace ownership
adibarra
committed
b816503
fix: preserve kimi quality failures
adibarra
committed
6cc9937
fix: harden bfcl transport retries
adibarra
committed
240a564

Commits on Aug 28, 2026

chore: merge current main branch
adibarra
committed
c63d0d2

Commits on Aug 29, 2026

fix: preserve stock tool use validators
adibarra
committed
708c8b5
feat: select exact generated deployments for eval shards
adibarra
committed
80fde1c
chore: remove unintended generator formatting churn
adibarra
committed
07bfc0d
fix: harden cached image and recipes
adibarra
committed
69e9e1d
fix: restore GB200 draft model source
adibarra
committed
c813727
fix: gate tool evaluations on readiness
adibarra
committed
aac233a
fix: require registered model before evaluation
adibarra
committed
b74fe20
fix: wait for active chat route
adibarra
committed
533a027

Commits on Aug 30, 2026

fix: gate eval dispatcher on route
adibarra
committed
8ef8107
fix: preserve selected TRT eval framework
adibarra
committed
0bfb0fb
fix: send model-aware readiness probe
adibarra
committed
b6d4085
fix: probe chat route registration
adibarra
committed
529b343
fix: stabilize registered models before eval
adibarra
committed
bc097ac
fix: accept stable OpenAI server readiness
adibarra
committed
089e707
fix: stabilize OpenAI health before eval
adibarra
committed
c12cdd6
fix: refresh expired MiniMax image
adibarra
committed
4961aec
fix: allocate hybrid DP ranks correctly
adibarra
committed
f76459e
fix: accept stock TensorRT chat requests
adibarra
committed
dcc2763
fix: keep AgentX dependency setup rootless
adibarra
committed
5aec415
fix: support packaged MiniMax evaluator imports
adibarra
committed
b4fdc49
fix: size vLLM offload allocations correctly
adibarra
committed
19bb70a
fix: split heterogeneous vLLM cache layers
adibarra
committed
64e85a8
chore: normalize vLLM offload patcher imports
adibarra
committed
6b0c5db
fix: patch B300 vLLM offload startup
adibarra
committed
bd13916

Commits on Aug 31, 2026

chore: merge main into tool evals
adibarra
committed
c75ca23
fix: update validators and repair smoke
adibarra
committed
d5bde4c
refactor: narrow tool parser evaluation scope
adibarra
committed
38dd382
docs: document runtime patch removal gates
adibarra
committed
1c0d6f2
chore: merge latest main into evaluations
adibarra
committed
dc3cd43
docs: link upstream runtime patch tracking
adibarra
committed
03caa00
fix: cut MI300X routing over to the new AMD cluster
cquil11
committed
139128e
fix: classify empty tool outputs correctly
adibarra
committed
260c5b9
chore: merge latest main before validation
adibarra
committed
8b7037d

Commits on Sep 1, 2026

fix: bound full BFCL evaluation runtime
adibarra
committed
2249e46
ci: make tool evaluation scores nonblocking
adibarra
committed
c4cb22c
feat: auto-run matching model vendor validators
adibarra
committed
3c2a9e9
fix: avoid duplicate eval matrix scoring
adibarra
committed
f967d65
fix: preserve agentic eval topology settings
adibarra
committed
3f8eaf9
fix: preserve GB200 synthetic acceptance dispatch
adibarra
committed
f672621
docs: remove unnecessary PR waiver files
adibarra
committed
1cf10bf