Export BEARER_TOKEN and build kube-burner-ocp from fork for kube-apiserver pprof#82349
Conversation
|
Warning Review limit reached
Next review available in: 20 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
WalkthroughThe six density workload scripts now enable fixed-interval pprof polling, provision kube-burner profiling access, build a custom kube-burner binary with kube-apiserver targets, and export the resulting credentials and local artifact URL. ChangesDensity profiling workflows
Estimated code review effort: 3 (Moderate) | ~25 minutes Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant DensityScript
participant OpenShiftCluster
participant CustomBuild
participant kube-burner
DensityScript->>OpenShiftCluster: Provision profiling RBAC
OpenShiftCluster-->>DensityScript: Return BEARER_TOKEN
DensityScript->>CustomBuild: Clone and build kube-burner-ocp
CustomBuild-->>DensityScript: Return packaged local tarball
DensityScript->>kube-burner: Export KUBE_BURNER_URL and profiling flags
kube-burner->>OpenShiftCluster: Run workload with authenticated pprof collection
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: redhat-chai-bot The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh (2)
84-95: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winAll six scripts build
kube-burner-ocpfrom an unpinned branch on an unofficial fork with a hardcodedbin/amd64/output path. Cloningadd-kube-apiserver-pprof-targetsby branch name (not commit SHA) means the build can change or break unexpectedly if the branch is amended/force-pushed, and the hardcoded arch path assumes a fixed Makefile output layout.
ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh#L84-L95: pinFORK_BRANCH/clone to a specific commit SHA and confirm the runner image hasgit/make/Go available.ci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.sh#L71-L82: same pinning/verification.ci-operator/step-registry/openshift-qe/node-density-heavy/openshift-qe-node-density-heavy-commands.sh#L66-L77: same pinning/verification.ci-operator/step-registry/openshift-qe/node-density/openshift-qe-node-density-commands.sh#L74-L85: same pinning/verification.ci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.sh#L77-L88: same pinning/verification.ci-operator/step-registry/openshift-qe/udn-density-pods/openshift-qe-udn-density-pods-commands.sh#L87-L98: same pinning/verification.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh` around lines 84 - 95, Pin FORK_BRANCH in the kube-burner-ocp build flow to a specific commit SHA and clone that immutable revision; replace the hardcoded bin/amd64/kube-burner-ocp lookup with a validated output path or discovery compatible with the build. Confirm git, make, and Go are available before building. Apply the same changes in ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh lines 84-95, node-density-cni/openshift-qe-node-density-cni-commands.sh lines 71-82, node-density-heavy/openshift-qe-node-density-heavy-commands.sh lines 66-77, node-density/openshift-qe-node-density-commands.sh lines 74-85, cudn-density/openshift-qe-cudn-density-commands.sh lines 77-88, and udn-density-pods/openshift-qe-udn-density-pods-commands.sh lines 87-98.
79-81: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winBlanket
|| trueon ns/SA/CRB creation masks real failures, not just AlreadyExists, in all six scripts. A genuine RBAC/API failure here silently propagates into a broken/emptyBEARER_TOKENlater, making failures much harder to diagnose.
ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh#L79-L81: check for existing resources explicitly (e.g.oc get ... || oc create ...) instead of a blind|| true.ci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.sh#L66-L68: same fix.ci-operator/step-registry/openshift-qe/node-density-heavy/openshift-qe-node-density-heavy-commands.sh#L61-L63: same fix.ci-operator/step-registry/openshift-qe/node-density/openshift-qe-node-density-commands.sh#L69-L71: same fix.ci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.sh#L72-L74: same fix.ci-operator/step-registry/openshift-qe/udn-density-pods/openshift-qe-udn-density-pods-commands.sh#L82-L84: same fix.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh` around lines 79 - 81, Replace the blanket “|| true” handling for namespace, service account, and cluster role binding creation with explicit existence checks followed by creation only when absent, while allowing genuine oc/API failures to propagate. Apply this consistently at the resource-creation blocks in ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh:79-81, ci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.sh:66-68, ci-operator/step-registry/openshift-qe/node-density-heavy/openshift-qe-node-density-heavy-commands.sh:61-63, ci-operator/step-registry/openshift-qe/node-density/openshift-qe-node-density-commands.sh:69-71, ci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.sh:72-74, and ci-operator/step-registry/openshift-qe/udn-density-pods/openshift-qe-udn-density-pods-commands.sh:82-84.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh`:
- Around line 78-81: The kube-burner service account currently receives
cluster-admin solely for kube-apiserver pprof access. In all six
scripts—ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh:78-81,
node-density-cni/openshift-qe-node-density-cni-commands.sh:65-68,
node-density-heavy/openshift-qe-node-density-heavy-commands.sh:60-63,
node-density/openshift-qe-node-density-commands.sh:68-71,
cudn-density/openshift-qe-cudn-density-commands.sh:71-74, and
udn-density-pods/openshift-qe-udn-density-pods-commands.sh:81-84—replace the
cluster-admin binding with a dedicated ClusterRole granting only get on
nonResourceURLs /debug/pprof/* and a corresponding ClusterRoleBinding for the
kube-burner service account.
- Around line 78-82: The BEARER_TOKEN assignment is exposed by shell tracing and
also violates SC2155. In
ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh
lines 78-82, disable tracing before token generation, declare BEARER_TOKEN
separately from the command substitution, then re-enable tracing afterward.
Apply the same set +x/set -x wrapping to the BEARER_TOKEN assignment in
ci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.sh
lines 65-69,
ci-operator/step-registry/openshift-qe/node-density-heavy/openshift-qe-node-density-heavy-commands.sh
lines 60-64,
ci-operator/step-registry/openshift-qe/node-density/openshift-qe-node-density-commands.sh
lines 68-72,
ci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.sh
lines 71-75, and
ci-operator/step-registry/openshift-qe/udn-density-pods/openshift-qe-udn-density-pods-commands.sh
lines 81-85.
---
Nitpick comments:
In
`@ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh`:
- Around line 84-95: Pin FORK_BRANCH in the kube-burner-ocp build flow to a
specific commit SHA and clone that immutable revision; replace the hardcoded
bin/amd64/kube-burner-ocp lookup with a validated output path or discovery
compatible with the build. Confirm git, make, and Go are available before
building. Apply the same changes in
ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh
lines 84-95, node-density-cni/openshift-qe-node-density-cni-commands.sh lines
71-82, node-density-heavy/openshift-qe-node-density-heavy-commands.sh lines
66-77, node-density/openshift-qe-node-density-commands.sh lines 74-85,
cudn-density/openshift-qe-cudn-density-commands.sh lines 77-88, and
udn-density-pods/openshift-qe-udn-density-pods-commands.sh lines 87-98.
- Around line 79-81: Replace the blanket “|| true” handling for namespace,
service account, and cluster role binding creation with explicit existence
checks followed by creation only when absent, while allowing genuine oc/API
failures to propagate. Apply this consistently at the resource-creation blocks
in
ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh:79-81,
ci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.sh:66-68,
ci-operator/step-registry/openshift-qe/node-density-heavy/openshift-qe-node-density-heavy-commands.sh:61-63,
ci-operator/step-registry/openshift-qe/node-density/openshift-qe-node-density-commands.sh:69-71,
ci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.sh:72-74,
and
ci-operator/step-registry/openshift-qe/udn-density-pods/openshift-qe-udn-density-pods-commands.sh:82-84.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 28805ad0-8fd2-4224-b79d-4712c9f28e6e
📒 Files selected for processing (6)
ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.shci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.shci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.shci-operator/step-registry/openshift-qe/node-density-heavy/openshift-qe-node-density-heavy-commands.shci-operator/step-registry/openshift-qe/node-density/openshift-qe-node-density-commands.shci-operator/step-registry/openshift-qe/udn-density-pods/openshift-qe-udn-density-pods-commands.sh
| # Setup pprof secrets for kube-apiserver profiling | ||
| oc create ns benchmark-operator || true | ||
| oc create serviceaccount kube-burner -n benchmark-operator || true | ||
| oc create clusterrolebinding kube-burner-crb --clusterrole=cluster-admin --serviceaccount=benchmark-operator:kube-burner || true |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
All six scripts bind cluster-admin to the kube-burner SA solely to scrape kube-apiserver pprof. A ClusterRole scoped to nonResourceURLs: ["/debug/pprof/*"] with verb: get would be sufficient and avoids granting full cluster-admin to a token that lives in benchmark-operator.
ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh#L78-L81: replace thecluster-adminclusterrolebinding with a scoped pprof ClusterRole/binding.ci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.sh#L65-L68: same replacement.ci-operator/step-registry/openshift-qe/node-density-heavy/openshift-qe-node-density-heavy-commands.sh#L60-L63: same replacement.ci-operator/step-registry/openshift-qe/node-density/openshift-qe-node-density-commands.sh#L68-L71: same replacement.ci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.sh#L71-L74: same replacement.ci-operator/step-registry/openshift-qe/udn-density-pods/openshift-qe-udn-density-pods-commands.sh#L81-L84: same replacement.
📍 Affects 6 files
ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh#L78-L81(this comment)ci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.sh#L65-L68ci-operator/step-registry/openshift-qe/node-density-heavy/openshift-qe-node-density-heavy-commands.sh#L60-L63ci-operator/step-registry/openshift-qe/node-density/openshift-qe-node-density-commands.sh#L68-L71ci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.sh#L71-L74ci-operator/step-registry/openshift-qe/udn-density-pods/openshift-qe-udn-density-pods-commands.sh#L81-L84
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh`
around lines 78 - 81, The kube-burner service account currently receives
cluster-admin solely for kube-apiserver pprof access. In all six
scripts—ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh:78-81,
node-density-cni/openshift-qe-node-density-cni-commands.sh:65-68,
node-density-heavy/openshift-qe-node-density-heavy-commands.sh:60-63,
node-density/openshift-qe-node-density-commands.sh:68-71,
cudn-density/openshift-qe-cudn-density-commands.sh:71-74, and
udn-density-pods/openshift-qe-udn-density-pods-commands.sh:81-84—replace the
cluster-admin binding with a dedicated ClusterRole granting only get on
nonResourceURLs /debug/pprof/* and a corresponding ClusterRoleBinding for the
kube-burner service account.
| # Setup pprof secrets for kube-apiserver profiling | ||
| oc create ns benchmark-operator || true | ||
| oc create serviceaccount kube-burner -n benchmark-operator || true | ||
| oc create clusterrolebinding kube-burner-crb --clusterrole=cluster-admin --serviceaccount=benchmark-operator:kube-burner || true | ||
| export BEARER_TOKEN=$(oc create token -n benchmark-operator kube-burner --duration=6h || oc sa get-token kube-burner -n benchmark-operator) |
There was a problem hiding this comment.
🔒 Security & Privacy | 🔴 Critical | ⚡ Quick win
BEARER_TOKEN leaks into CI logs via set -x in all six workload scripts. Each script enables set -x at the top and never disables tracing before generating/exporting the SA bearer token, so the token literal is echoed into build logs — a direct credential leak, and a violation of the guideline to disable tracing around sensitive operations.
ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh#L78-L82: wrap theexport BEARER_TOKEN=$(...)line withset +x/set -xand split the declare/assign per SC2155.ci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.sh#L65-L69: apply the sameset +x/set -xwrapping around theBEARER_TOKENexport.ci-operator/step-registry/openshift-qe/node-density-heavy/openshift-qe-node-density-heavy-commands.sh#L60-L64: apply the same fix.ci-operator/step-registry/openshift-qe/node-density/openshift-qe-node-density-commands.sh#L68-L72: apply the same fix.ci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.sh#L71-L75: apply the same fix.ci-operator/step-registry/openshift-qe/udn-density-pods/openshift-qe-udn-density-pods-commands.sh#L81-L85: apply the same fix.
🧰 Tools
🪛 Shellcheck (0.11.0)
[warning] 82-82: Declare and assign separately to avoid masking return values.
(SC2155)
📍 Affects 6 files
ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh#L78-L82(this comment)ci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.sh#L65-L69ci-operator/step-registry/openshift-qe/node-density-heavy/openshift-qe-node-density-heavy-commands.sh#L60-L64ci-operator/step-registry/openshift-qe/node-density/openshift-qe-node-density-commands.sh#L68-L72ci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.sh#L71-L75ci-operator/step-registry/openshift-qe/udn-density-pods/openshift-qe-udn-density-pods-commands.sh#L81-L85
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh`
around lines 78 - 82, The BEARER_TOKEN assignment is exposed by shell tracing
and also violates SC2155. In
ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh
lines 78-82, disable tracing before token generation, declare BEARER_TOKEN
separately from the command substitution, then re-enable tracing afterward.
Apply the same set +x/set -x wrapping to the BEARER_TOKEN assignment in
ci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.sh
lines 65-69,
ci-operator/step-registry/openshift-qe/node-density-heavy/openshift-qe-node-density-heavy-commands.sh
lines 60-64,
ci-operator/step-registry/openshift-qe/node-density/openshift-qe-node-density-commands.sh
lines 68-72,
ci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.sh
lines 71-75, and
ci-operator/step-registry/openshift-qe/udn-density-pods/openshift-qe-udn-density-pods-commands.sh
lines 81-85.
Source: Coding guidelines
|
/pj-rehearse periodic-ci-openshift-eng-ocp-perfscale-main-aws-5.0-nightly-x86-payload-control-plane-6nodes |
|
@mcornea: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
f2e2f7b to
67afbaf
Compare
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
ci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.sh (1)
108-117: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winPoint the timeout recovery check at the custom binary path.
The script ships the built binary from
/tmp/kube-burner-ocp-bin/kube-burner-ocpintofile:///tmp/kube-burner-ocp-custom.tar.gz. If the wrapper leaves that tarball unpacked or extracts to a non-executable location,if /tmp/kube-burner-ocp cluster-health; thensilently fails onrun.shexit code 2 and skips the intended health-check fallback. Match the health check to the actual executable path used byKUBE_BURNER_URL.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.sh` around lines 108 - 117, Update the cluster-health invocation in the RUN_EXIT_CODE 2 recovery block to use the executable path configured through KUBE_BURNER_URL, specifically the custom binary location used by the shipped kube-burner artifact, instead of /tmp/kube-burner-ocp. Preserve the existing health-check fallback and cleanup behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In
`@ci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.sh`:
- Around line 108-117: Update the cluster-health invocation in the RUN_EXIT_CODE
2 recovery block to use the executable path configured through KUBE_BURNER_URL,
specifically the custom binary location used by the shipped kube-burner
artifact, instead of /tmp/kube-burner-ocp. Preserve the existing health-check
fallback and cleanup behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 859c8ea5-ffd9-4eff-8ca8-093ebf4b7397
📒 Files selected for processing (6)
ci-operator/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.shci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.shci-operator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.shci-operator/step-registry/openshift-qe/node-density-heavy/openshift-qe-node-density-heavy-commands.shci-operator/step-registry/openshift-qe/node-density/openshift-qe-node-density-commands.shci-operator/step-registry/openshift-qe/udn-density-pods/openshift-qe-udn-density-pods-commands.sh
|
/pj-rehearse periodic-ci-openshift-eng-ocp-perfscale-main-aws-5.0-nightly-x86-payload-control-plane-6nodes |
|
@mcornea: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Set BEARER_TOKEN=$(oc whoami -t) before kube-burner-ocp invocation in all openshift-qe step ref command scripts that use PPROF. This provides the authentication token needed by the new kube-apiserver pprof targets (PERFSCALE-5323).
67afbaf to
45bbd1f
Compare
|
[REHEARSALNOTIFIER]
A total of 737 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/pj-rehearse periodic-ci-openshift-eng-ocp-perfscale-main-aws-5.0-nightly-x86-payload-control-plane-6nodes |
|
@mcornea: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-eng-ocp-perfscale-main-aws-5.0-nightly-x86-payload-control-plane-6nodes |
|
@mcornea: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@redhat-chai-bot: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
Summary
Enable kube-apiserver CPU pprof collection for all PerfScale workloads in the
payload-control-plane-6nodesjob. This is an investigation aid for PERFSCALE-5323 — a confirmed 11-19%apiserverCPU_avgregression after the Kubernetes 1.36.2 rebase in OCP 5.0.Problem
Audit log analysis confirmed that the API call volume during node-density-cni is essentially unchanged between pre-rebase (k8s v1.35.3) and post-rebase (k8s v1.36.2) — the regression is caused by each API call being more CPU-expensive, not by more calls. CPU pprof profiles are needed to identify which code paths cost more after the rebase.
The existing
PPROF=truesetting only collects OVN profiles (ovnkube-controller, ovnk-control-plane). kube-apiserver pprof requires bearer token auth and is not included in the upstream kube-burner-ocp binary.Changes
All 6 step ref command scripts are updated with:
BEARER_TOKEN setup — creates a
kube-burnerServiceAccount inbenchmark-operatornamespace withcluster-adminClusterRoleBinding, then generates a 6-hour bearer token. This follows theget_pprof_secrets()pattern from e2e-benchmarking common.sh L134-148.Custom kube-burner-ocp build — clones the fork branch
redhat-chai-bot/kube-burner-ocp:add-kube-apiserver-pprof-targetswhich adds akube-apiserver-cpupprof target to all workload configs. Builds withmake build, tarballs the binary, and setsKUBE_BURNER_URLto the local tarball sorun.shuses it instead of downloading from GitHub releases.Recurring pprof collection — adds
--pprof-interval=1mtoEXTRA_FLAGSfor periodic 30-second CPU profile capture every minute during the workload (instead of the default single collection at end).Modified scripts
cluster-density-v2-commands.shcudn-density-commands.shnode-density-cni-commands.shnode-density-heavy-commands.shnode-density-commands.shudn-density-pods-commands.shNext steps
Once merged, trigger pre-rebase and post-rebase runs to compare kube-apiserver CPU profiles and identify which code paths regressed in k8s 1.36.2.
Cleanup
After the investigation, the fork build logic should be removed and replaced with the appropriate
KUBE_BURNER_VERSIONoverride once upstream kube-burner-ocp includes kube-apiserver pprof targets natively.@mcornea requested in Slack thread
Summary by CodeRabbit
payload-control-plane-6nodesPerfScale job (cluster-density-v2, CUDN-density, node-density-cni, node-density-heavy, node-density, and UDN-density-pods) to collect recurring kube-apiserver CPU pprof profiles by adding--pprof-interval=1mto the kube-burner flags.benchmark-operatorRBAC (createskube-burnerServiceAccount +cluster-adminbinding) and exporting a 6-hourBEARER_TOKEN.add-kube-apiserver-pprof-targets), exports it via a localfile://tarball (KUBE_BURNER_URL), and uses that artifact for the workload runs.