Skip to content

Add lightweight tool-use verifier evaluations / 添加轻量级工具调用验证评估 - #2634

Merged
adibarra merged 115 commits into
mainfrom
experiment/tool-use-eval-full
Sep 1, 2026
Merged

Add lightweight tool-use verifier evaluations / 添加轻量级工具调用验证评估#2634
adibarra merged 115 commits into
mainfrom
experiment/tool-use-eval-full

Conversation

@adibarra

@adibarra adibarra commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add pinned Kimi, MiniMax, and BFCL tool-use evaluators to the existing InferenceX eval pipeline.
  • Automatically run kimi-vendor for every kimik3 Agentic point and minimax-vendor for every minimaxm3 Agentic point. BFCL remains explicit.
  • Preserve each recipe's serving topology, router, speculative decoding, and runtime settings in eval-only jobs.
  • Enable the documented Kimi K3 and MiniMax M3 tool parsers on their existing serving paths. Eval-only jobs restore real acceptance behavior, while GB200 throughput retains its synthetic-acceptance opt-in.
  • Reuse the existing launcher, artifact, score-validation, collection, and dashboard paths.

Evaluation behavior

Model Automatic Agentic evaluation Coverage
kimik3 kimi-vendor with kimi_tool_call_schema Every generated point
minimaxm3 minimax-vendor with minimax_m3_smoke Every generated point
Other models Existing lm-eval GSM8K policy Highest eligible point per deployment group
BFCL Explicit only No automatic model mapping

Fixed-sequence GSM8K selection is unchanged. New tool-use thresholds are 0.0, so completed results report model quality without blocking. Startup, transport, timeout, malformed output, missing artifact, and zero-effective-sample failures still fail the job.

Third-party sources and SHA256 checks are pinned. Implementation details and manual invocation examples are in utils/evals/EVALS.md.

Validation

  • Current-head config parsing, process-result validation, changelog validation, and CodeQL passed.
  • Matrix and schema tests passed 267/267. Eval tests passed 223/223, with one unrelated SWE-bench dependency test skipped. Collection and reusable-artifact tests passed 64/64. Launcher compatibility tests passed 24/24.
  • Kimi topology verification matched 105/105 throughput points to eval jobs. MiniMax verification matched 113/113 points.
  • Hardware runs completed the Kimi vendor full suite with 408 outcomes, MiniMax vendor full suite with 102 cases, and MiniMax BFCL full suite with 1000 cases.
  • The Kimi BFCL full run exposed request and step bounds that are fixed in the current code. Its current-head retry was canceled before runner assignment. BFCL is not selected automatically.

中文说明

  • 将固定版本的 Kimi、MiniMax 和 BFCL 工具调用评估器接入现有 InferenceX 评估流水线。
  • 对每个 kimik3 Agentic 测试点自动运行 kimi-vendor,对每个 minimaxm3 Agentic 测试点自动运行 minimax-vendor。BFCL 仍需显式启用。
  • Eval-only 任务完整保留 recipe 的服务拓扑、router、投机解码和运行时配置。
  • 在现有服务路径上启用文档约定的 Kimi K3 和 MiniMax M3 工具解析器。Eval-only 任务恢复真实 acceptance 行为,GB200 吞吐量任务继续保留合成 acceptance 显式开关。
  • 复用现有启动器、产物、得分校验、结果汇总和看板链路。

评估行为

模型 自动 Agentic 评估 覆盖范围
kimik3 kimi-vendorkimi_tool_call_schema 每个生成的测试点
minimaxm3 minimax-vendorminimax_m3_smoke 每个生成的测试点
其他模型 现有 lm-eval GSM8K 策略 每个部署分组中符合条件的最高并发测试点
BFCL 仅显式启用 不进行自动模型映射

固定序列长度的 GSM8K 选择策略保持不变。新增工具调用评估的阈值均为 0.0,因此已完成的结果只报告模型质量,不阻塞流程。服务启动、传输、超时、输出格式、产物缺失或有效样本数为零仍会导致任务失败。

第三方源码版本和 SHA256 校验均已固定。实现细节和手动运行示例见 utils/evals/EVALS.md

验证

  • 当前头提交的配置解析结果处理校验变更日志校验CodeQL已通过。
  • 矩阵和 schema 测试 267/267 通过。评估测试 223/223 通过,另有一个无关的 SWE-bench 依赖测试跳过。结果汇总和可复用产物测试 64/64 通过。启动器兼容性测试 24/24 通过。
  • Kimi 拓扑验证中,105/105 个吞吐量测试点均匹配对应评估任务。MiniMax 验证中,113/113 个测试点完全匹配。
  • 硬件任务已完成 Kimi vendor 完整套件的 408 个结果、MiniMax vendor 完整套件的 102 个用例,以及 MiniMax BFCL 完整套件的 1000 个用例。
  • Kimi BFCL 完整任务暴露的请求超时和步骤上限问题已在当前代码中修复。当前头提交的重试在分配 runner 前被取消。BFCL 不会自动运行。

中文:添加基于 Kimi Vendor Verifier 原生实现的工具调用评估
中文:精简工具调用评估实现并接入工作流选择路径
中文:合并最新主分支以完成发布准备
中文:隔离 Kimi 验证器依赖并明确供应商适配边界,同时加入执行超时和通用结果格式标识。
中文:合并最新 main,确保发布前验证基于当前代码。
中文:区分失败结果产物写入成功与写入失败,同时保留原始安装失败退出码。
中文:将 agentic 评估矩阵的投机解码模式传递给启动器,避免 MTP 配置静默回退到 STP。
中文:合并最新 main,准备发布。
中文:补全纯评估工作流的结果收集依赖,避免所有评估任务成功后工作流仍显示失败。
The collector dependency change did not affect GitHub’s workflow conclusion and added unrelated orchestration scope.

中文:恢复纯评估结果收集的原有范围。该依赖调整未改变 GitHub 工作流结论,且扩大了无关改动范围。
中文:合并最新 main,准备进入评审。
中文:修正验证器失败样本数、共享格式契约、路径复用及双语文档链接。
中文:保留结果收集器测试的现有格式,仅添加格式契约测试。
中文:在启动失败时保留验证器计划样本数,同时将有效样本数记录为零。
中文:合并最新 main,并保留工具调用评估元数据。
中文:保留可配置的 eval 调度行为,并修正失败返回码、Bash 兼容性和 eval 文档。
中文:强化验证器评审路径。
中文:限定评估套件状态的作用域,并修正双语 AgentX 文档中的源码行号链接。
中文:将发生不可纠正 NVLink 错误的 b300-017 节点排除在 Slurm 分配之外。
中文:让 B300 启动器将 SALLOC_EXCLUDE 传递给 salloc,避免调度到已知故障节点。
中文:启用多节点 Kimi 验证器
中文:强化 Kimi 评估运行时故障处理
中文:捕获 Kimi 工具调用响应诊断数据。
中文:采集确定性的 Kimi 工具调用诊断数据。
中文:启用 Kimi 结构化工具调用约束。
中文:格式化 Kimi 配方回归测试。
中文:重试 Kimi 验证器的临时下载失败。
中文:稳定并清理 Kimi 验证器集成。
adibarra and others added 10 commits August 31, 2026 10:12
中文:将 main 合并到工具调用评估分支
中文:更新验证器版本并修复 MiniMax smoke 适配逻辑
中文:将工具解析器和服务就绪改动限制在 Kimi、MiniMax 及 eval-only 路径
中文:记录 PR 2634 运行时补丁的适用范围、验证证据和移除条件
中文:再次将最新 main 合并到工具调用评估分支
中文:补充运行时补丁对应的上游 PR 和 issue 链接
中文:正确分类空工具调用输出,将成功请求中的零调用结果保留为有效的模型质量失败,而不是集成失败。
中文:在验证前合并最新的 main。
中文:限制完整 BFCL 评估运行时间,固定请求超时和重试次数,将 Kimi 多轮步数限制为 10,并为其完整套件预留四小时。
@adibarra
adibarra marked this pull request as ready for review September 1, 2026 03:14
@adibarra
adibarra requested a review from a team September 1, 2026 03:14
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 2249e46. Configure here.

Comment thread .github/workflows/e2e-tests.yml Outdated
中文:将工具调用评估得分设为非阻塞诊断指标。

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Comment thread runners/launch_gb200-nv.sh Outdated
Comment thread .github/workflows/e2e-tests.yml Outdated
中文:自动为匹配模型运行厂商验证器
中文:避免重复计算评估任务矩阵优先级
中文:保留 agentic 评估任务的完整拓扑配置
中文:保留 GB200 吞吐量测试的合成 acceptance 调度
中文:删除本 PR 不再需要的补丁豁免文件
@SemiAnalysisAI SemiAnalysisAI deleted a comment from github-actions Bot Sep 1, 2026
@adibarra
adibarra merged commit 7ae9bf6 into main Sep 1, 2026
74 of 77 checks passed
@adibarra
adibarra deleted the experiment/tool-use-eval-full branch September 1, 2026 05:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

2 participants