enable posting to GitHub Checks API

[OmarTawfik]

Thanks for working on Bencher!

We have bencher running on our PRs, and it posts a PR comment with the result. But unfortunately we have a couple of issues with PR comments:

1. A PR comment with active alerts won't block merging the PR. GitHub checks do.
2. On large PRs with many comments/review rounds, that comment (and any alerts) are less visible/can be missed. A failed check (because of alerts) is highly visible.
3. We have a lot of benchmarks, so we end up with ⚠️ WARNING: Truncated view! The full continuous benchmarking report exceeds the maximum length allowed on this platform.

The documentation mentions posting to GitHub Checks, which would work perfectly for us. But looking at the code, it seems to be a fallback only if bencher is not able to post a comment to the PR.

Is there currently a way to override this behavior, and explicitly switch to GitHub checks instead?
If not, any thoughts/concerns on adding a flag like this? i.e. something like --github-checks

[epompeii]

@OmarTawfik thank you for using Bencher, and I'm sorry things aren't working as you would like.

1. A PR comment with active alerts won't block merging the PR. GitHub checks do.

Would setting the --error-on-alert flag work here?
You should be able to have the error fail the PR job step to block the merge.

2. On large PRs with many comments/review rounds, that comment (and any alerts) are less visible/can be missed. A failed check (because of alerts) is highly visible.

Yes, on very active PRs, comments can get lost. Especially with GitHub's comment collapsing.
If you implement 1. above you should be able to see an ❌ in your list of checks on failure.

3. We have a lot of benchmarks, so we end up with ⚠️ WARNING: Truncated view! The full continuous benchmarking report exceeds the maximum length allowed on this platform.

This is a limitation of the GitHub API. There are only so many characters allowed per comment. I've considered splitting results across multiple comments, but that can get really messy.
It's also worth noting this same limitation exists for GitHub Check output as well.

What would you like to see here?

Is there currently a way to override this behavior, and explicitly switch to GitHub checks instead?

No, there is not. I have honestly considered completely removing the current GitHub Check integration due to #536
That is, Bencher can't actually rely on the GitHub API to put a GitHub Check output where it should be.

It may be worth clarifying, Bencher's GitHub Check integration is just the output block. We don't actually create a new, distinct line item in your GitHub PR checks list. That's up to your GitHub Actions implementation.

Does that make sense?

[OmarTawfik]

Thanks for the detailed response!

Would setting the --error-on-alert flag work here?

I think the main difference is that the CI job failure will now mean either failure of the runner itself (failure to report results) or metric alerts (runner succeeded, but reported concerning numbers).

Differentiating between these two modes is important, because the former should absolutely always prevent a PR from merging (if it breaks the runner or the tests), but a raised alert can be reviewed and deemed safe to merge (e.g. adding a new feature usually increases runtime benchmarks).

Unfortunately --error-on-alert can't provide that, because it collapses both failure modes into the job exit code.

It may be worth clarifying, Bencher's GitHub Check integration is just the output block. We don't actually create a new, distinct line item in your GitHub PR checks list. That's up to your GitHub Actions implementation.

I'm not sure I'm following. My understanding is that a "Check Run" (what the code above is already creating) is a named, first-class status entry (e.g. "Bencher Report") attached to a commit SHA, with its own conclusion (success/failure/neutral) and an output block. It appears as its own row in the checks list, independent of the job, and can be independently marked as a required status check in branch protection.

No, there is not. I have honestly considered completely removing the current GitHub Check integration due to #536
That is, Bencher can't actually rely on the GitHub API to put a GitHub Check output where it should be.

From reading the docs below, looks like a check run would appear on any PR that has the same HEAD commit SHA associated with the check run? regardless of which workflow it was attached to?

https://docs.github.com/en/rest/checks/runs?create-a-check-run=&apiVersion=2026-03-10#create-a-check-run

[epompeii]

My understanding is that a "Check Run" (what the code above is already creating) is a named, first-class status entry (e.g. "Bencher Report") attached to a commit SHA, with its own conclusion (success/failure/neutral) and an output block. It appears as its own row in the checks list, independent of the job, and can be independently marked as a required status check in branch protection.

Taking a closer look, you're right. Bencher does create a first-class Check Run named Bencher Report, with its own conclusion: failure if there are active Alerts and success otherwise. So it can be marked as a required status check in branch protection, independent of the job.

From reading the docs below, looks like a check run would appear on any PR that has the same HEAD commit SHA associated with the check run? regardless of which workflow it was attached to?

Yes, that's correct. After digging back into this, #536 is narrower than I remembered. The Check Run is keyed to the head SHA, so the merge box row, its conclusion, and branch protection all work regardless of which workflow run it gets grouped under. What #536 actually breaks is where the output renders: clicking "Details" on the check shows the report nested under an arbitrary workflow run for that SHA (whichever check suite was created first https://github.com/orgs/community/discussions/24616). The report itself renders fine. It's the navigation context that's confusing.

Differentiating between these two modes is important, because the former should absolutely always prevent a PR from merging (if it breaks the runner or the tests), but a raised alert can be reviewed and deemed safe to merge (e.g. adding a new feature usually increases runtime benchmarks).

That's a fair point, and the Check Run conclusion gives you exactly that separation: the job exit code tells you whether bencher run itself succeeded, and the Bencher Report conclusion tells you whether there were any Alerts. As a bonus, if the runner fails outright then no check is created, so a required Bencher Report check would sit at "Expected" and block the merge, which matches the semantics you want.

So instead of adding a --github-checks flag, I went one step further in #895: bencher run --github-actions now always creates the Check Run, alongside the PR comment.

• The check is created on a best effort basis. If it can't be created (for example, the token is missing checks: write permissions), then bencher run logs a warning and continues.
• The check is created on the pull request head commit (pull_request.head.sha or workflow_run.head_sha), so it actually shows up on the PR. Previously it used GITHUB_SHA, which is the ephemeral merge commit for pull_request events.
• The check name is Bencher Report by default. Only if you set --ci-id <ID> is the ID appended to the check name (ie Bencher Report (<ID>)). That way multiple bencher run invocations on the same commit each get a distinct, stable name that can be used as a separate required status check.

You will need to give the GITHUB_TOKEN checks: write permissions. I plan to ship this in the next release. Does this fit your needs, or is there anything you think I should tweak?

The one thing this does not fix is your third issue: the 65,535 character limit also applies to Check Run output (#534). The full report is always available on Bencher itself, and the job summary gets the same report on the workflow run page.

[OmarTawfik]

I plan to ship this in the next release.

This is perfect! thank you for all the hard work!
Is the guidance is still to build/run bencher_cli from your main branch? or should we move to versioned releases now?

Does this fit your needs, or is there anything you think I should tweak?

The only thing I would suggest is maybe considering a stronger failure mode (or as the default) if the checks can't be created or updated after creation (missing permissions, network error, etc...). A warning in the logs is easily missed, and might lead to a PR can be merged accidentally without the check. But not a blocker. I believe there are ways around it that we can look into.

[epompeii]

Is the guidance is still to build/run bencher_cli from your main branch? or should we move to versioned releases now?

Yes, the current guidance is still to use the bencherdev/bencher@main GitHub Action for Bencher Cloud. This will use the latest Bencher CLI release automatically. In the current era of supply chain risk though, it is understandable to want to pin the GitHub Action SHA.

The only thing I would suggest is maybe considering a stronger failure mode (or as the default) if the checks can't be created or updated after creation (missing permissions, network error, etc...).

If you set the Bencher Report GitHub Check as required then I think you can hard fail on these sorts of issues.
I agree that having it get caught would be nice. I would like to avoid a breaking change here though, so I'm not inclined to make the check required at this point.

[epompeii]

@OmarTawfik Bencher v0.6.7 is now released. Please, give things a try and let me know what you think either way.

[OmarTawfik]

Thank you @epompeii!

I tried updating it in NomicFoundation/slang#1873, but the CI job silently failed with:

  Failed to create GitHub Check: JSON Error in pull_requests[0]: missing field `node_id` at line 1 column 4954

[epompeii]

@OmarTawfik thank you for giving things a go! And I'm sorry for the hiccup.

I think the GitHub Check is getting created here: https://github.com/NomicFoundation/slang/pull/1873/checks?check_run_id=81743189675

However, there is a bug in the version of Octocrab I'm currently using: XAMPPRocky/octocrab#902
This causes the successful Check Run response body to fail to deserialize. Hence the:

Error in pull_requests[0]: missing field `node_id`

The Octocrab fix was already released: https://github.com/XAMPPRocky/octocrab/releases/tag/v0.52.0

So I have a PR up to bump the version: #902