Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The policy as-is is too lax. I don't think libdatadog should be included as a subject as we already have benchmark jobs triggered internally and use a different existing policy. They shouldn't share a policy with this workflow.
I don't know how these benchmarks are going to work, but my suggestion assumes they are going to run on the default branch of the SDKs.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The current plan is to have those benchmarks run on some branch of the SDK/tracers (but not the default one, since the idea is to auto update the SDK to some version of libdatadog corresponding to a PR, so this will be on a new, dedicated branch). At some point this SDK benchmark job needs to write back a comment here to report the result, which I think is the goal of this policy change. Hopes that clarifies. But also maybe there's a better way to orchestrate all of this, so open to suggestions here!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it. If it's a dedicated branch then my suggestion would still mostly work. You'd just need to update it to match on that branch name instead of main/master.