Add opt-in Bug Scouter PR analysis for azure-ai-ml - #48503
Open
mohammadsheraj wants to merge 3 commits into
Open
Add opt-in Bug Scouter PR analysis for azure-ai-ml#48503mohammadsheraj wants to merge 3 commits into
mohammadsheraj wants to merge 3 commits into
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 10 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
|
Azure Pipelines: Successfully started running 1 pipeline(s). 10 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds opt-in Bug Scouter analysis for labeled azure-ai-ml pull requests using separated untrusted build and trusted invocation workflows.
Changes:
- Builds and packages exact-commit PR inputs.
- Validates provenance, artifacts, authorization, and Azure OIDC invocation.
- Adds focused preparation, publication, and failure-handling tests.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/bug-scouter-build.yml |
Builds and uploads exact-SHA inputs. |
.github/workflows/bug-scouter-invoke.yml |
Validates, invokes, and publishes results. |
.github/scripts/prepare_bugscouter.py |
Validates artifacts and assembles PR context. |
.github/scripts/run_bugscouter.py |
Invokes Bug Scouter and publishes outcomes. |
.github/scripts/test_prepare_bugscouter.py |
Tests provenance and artifact validation. |
.github/scripts/test_run_bugscouter.py |
Tests invocation and result publication. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
| build-wheel: | ||
| name: Build exact PR wheel | ||
| if: contains(github.event.pull_request.labels.*.name, 'bug-scouter') | ||
| runs-on: ubuntu-latest |
Comment on lines
+350
to
+352
| existing = next( | ||
| (comment for comment in comments if marker in str(comment.get("body") or "")), | ||
| None, |
Comment on lines
+15
to
+19
| invoke: | ||
| name: Validate, invoke, and publish | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 100 | ||
| environment: bug-scouter-prod |
Comment on lines
+285
to
+295
| def _bug_count(result: dict) -> int: | ||
| value = result.get("real_bug_count") | ||
| if isinstance(value, bool): | ||
| raise ValueError("real_bug_count must be an integer") | ||
| try: | ||
| bug_count = int(value) | ||
| except (TypeError, ValueError) as exc: | ||
| raise ValueError("real_bug_count must be an integer") from exc | ||
| if bug_count < 0 or bug_count > MAX_BUG_COUNT: | ||
| raise ValueError("real_bug_count is outside the allowed range") | ||
| return bug_count |
Comment on lines
+339
to
+350
| with zipfile.ZipFile(archive_file) as archive: | ||
| members = [member for member in archive.infolist() if not member.is_dir()] | ||
| names = {member.filename for member in members} | ||
| wheel_names = [name for name in names if WHEEL_PATTERN.fullmatch(name)] | ||
| if len(wheel_names) != 1 or names != { | ||
| "manifest.json", | ||
| "README.md", | ||
| wheel_names[0], | ||
| }: | ||
| raise ValueError( | ||
| "artifact archive contains missing, nested, or unexpected files" | ||
| ) |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (4)
.github/scripts/run_bugscouter.py:350
- When all ten pages are full, the search silently proceeds as if no marker exists. On a PR that already has at least 1,000 comments, the first Bug Scouter comment lands after the search window and every rerun creates another comment, breaking the promised idempotency. Fail closed when the pagination bound is exhausted, as
prepare_bugscouter._github_pagesalready does.
if len(batch) < 100:
break
existing = next(
.github/scripts/run_bugscouter.py:282
- The length cap is applied before escaping
@, but each replacement expands one character to five. A valid 60,000-character service summary containing many mentions can therefore become hundreds of thousands of characters and exceed GitHub's issue-comment body limit, causing result publication to fail. Escape first and truncate the escaped value.
summary = str(value or fallback)[:MAX_SUMMARY_CHARACTERS]
return summary.replace("@", "@")
.github/scripts/run_bugscouter.py:190
- The live label/SHA authorization is checked before hashing a wheel of up to 200 MiB, leaving a revocation window in which the label can be removed before staging begins but the blob is still uploaded. Compute the hash and blob name first, then perform the live check immediately before
_upload_blobso the opt-in check actually guards staging as described.
This issue also appears in the following locations of the same file:
- line 281
- line 348
_validate_live_authorization(values, github_token)
wheel_hash = hashlib.sha256(wheel_path.read_bytes()).hexdigest()
.github/workflows/bug-scouter-invoke.yml:19
- The protected environment is entered for every completed
Bug Scouter Buildrun, not only opted-in runs. The build workflow itself starts for every matching PR and only skips its sole job when the label is absent; an all-skipped workflow still completes successfully and triggers thisworkflow_run, so unlabeled PRs can queue this environment job and then fail inresolve. Split the preflight/label validation into a job without the environment and gate a second environment-scoped invocation job on its validated output (while retaining a non-environment failure-publication path).
environment: bug-scouter-prod
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an opt-in
bug-scouterPR analysis pilot for changes undersdk/ml/azure-ai-ml.pull_requestworkflowbug-scouterlabel is presentworkflow_runApplying the
bug-scouterlabel is an explicit authorization for the hosted Bug Scouter service to install and test that exact PR wheel. Remove the label to block invocation before staging; pushing a new commit invalidates an older run through the live SHA check.Context: #48502
Security boundary
The build job has read-only contents access and no Azure identity or write token. The trusted job pins action dependencies, checks the upstream repository/workflow/run/live SHA/head repository/label, rejects unexpected paths, symlinks, sizes, digest mismatches, oversized GitHub responses, and excessive pagination, then invokes Foundry. Fork PRs are supported without granting credentials to PR code in GitHub Actions.
PR-controlled wheel execution occurs in the labeled hosted Bug Scouter run. The current service isolation boundary is its hosted container and per-run virtual environment; the label must therefore be applied only to PRs approved for that execution boundary.
Validation
python -m pytest .github/scripts/test_prepare_bugscouter.py .github/scripts/test_run_bugscouter.py -q(48 passed)0 issues)actionlint1.7.12 for both workflowsgit diff --check7663ce3Administrator setup after approval
A repository administrator must create the
bug-scouter-prodenvironment and configure the six referencedBUGSCOUTER_*variables. Azure federation and least-privilege RBAC are configured separately; no cloud secrets are stored in GitHub.