INTEROP-9236: Add ACS SMOKE test pipeline for OPP interop - #83102
Conversation
|
@amp-rh: This pull request references INTEROP-8003 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@amp-rh, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp4.22-interop-opp-aws |
|
@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds ACS OPP readiness validation and smoke testing to the OCP 4.22 AWS interop workflow. The changes add a runner image, readiness checks, shared credentials, test execution, and report collection. ChangesACS OPP testing
Estimated code review effort: 3 (Moderate) | ~30 minutes Sequence Diagram(s)sequenceDiagram
participant AWSInterop
participant ReadinessStep
participant OpenShift
participant Central
participant SharedDir
participant SmokeStep
participant ArtifactDir
AWSInterop->>ReadinessStep: Run ACS readiness
ReadinessStep->>OpenShift: Discover namespaces and check sensors
ReadinessStep->>Central: Validate API, clusters, and policies
ReadinessStep->>SharedDir: Write credentials and connection details
AWSInterop->>SmokeStep: Run ACS smoke tests
SmokeStep->>SharedDir: Load readiness outputs
SmokeStep->>Central: Execute testSMOKE
SmokeStep->>ArtifactDir: Publish test reports
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 2 warnings)
✅ Passed checks (12 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: amp-rh The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/stolostron/policy-collection/stolostron-policy-collection-main__ocp4.22.yaml`:
- Around line 56-58: Pin all externally executed artifacts to immutable
revisions: in
ci-operator/config/stolostron/policy-collection/stolostron-policy-collection-main__ocp4.22.yaml
lines 56-58, replace the stable OpenShift client URL with a release-specific URL
and validate it against a fixed expected checksum; in
ci-operator/step-registry/stackrox/opp-smoke/stackrox-opp-smoke-commands.sh
lines 31-32, replace mutable StackRox and Scanner main-branch references with
immutable revisions and verify each checked-out commit before running Gradle.
In
`@ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.sh`:
- Around line 87-90: Enable TLS certificate verification for all Central
requests by removing curl’s -k option in
ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.sh
at lines 87-90 (health request), 101-105 (authenticated cluster request), and
165-171 (authenticated policy request); rely on the system trust bundle or
provide the ingress CA via --cacert.
In
`@ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-ref.yaml`:
- Line 9: Align the readiness gate’s total timeout with its retry budget: in
ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-ref.yaml
at lines 9-9, increase the step timeout if each wait_for check retains its
independent budget; alternatively, in
ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.sh
at lines 30-30, replace the per-check ELAPSED reset with one deadline covering
all seven checks.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 9bf41049-e10a-4671-acca-4f75e94c5a38
📒 Files selected for processing (9)
ci-operator/config/stolostron/policy-collection/stolostron-policy-collection-main__ocp4.22.yamlci-operator/step-registry/stackrox/opp-readiness/OWNERSci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.shci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-ref.metadata.jsonci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-ref.yamlci-operator/step-registry/stackrox/opp-smoke/OWNERSci-operator/step-registry/stackrox/opp-smoke/stackrox-opp-smoke-commands.shci-operator/step-registry/stackrox/opp-smoke/stackrox-opp-smoke-ref.metadata.jsonci-operator/step-registry/stackrox/opp-smoke/stackrox-opp-smoke-ref.yaml
|
/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp4.22-interop-opp-aws |
|
@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@amp-rh: This pull request references INTEROP-9236 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.sh (1)
30-43: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winUse one deadline for the readiness sequence.
Line 30 resets
ELAPSEDfor everyWaitForcall. The seven calls at Lines 66, 69, 79, 94, 125, 159, and 175 can each consumeTIMEOUT. The full sequence can run for approximately7 * TIMEOUT, not the single polling budget stated for this step.Set one deadline before the first wait. Calculate each retry budget from that deadline.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.sh` around lines 30 - 43, Use a single readiness deadline initialized before the sequence of WaitFor calls instead of resetting ELAPSED inside each WaitFor invocation. Update WaitFor and its retry/timeout checks to calculate remaining time from that shared deadline, so all readiness checks collectively remain within one TIMEOUT budget.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.sh`:
- Around line 151-156: Update the sensor pod readiness check that builds
notRunning from podJson to require both status.phase == "Running" and a pod
status.conditions entry with type "Ready" and status "True" for every pod.
Preserve the existing name-and-phase reporting for pods that fail either
requirement and the final empty-result readiness check.
- Line 2: Update the shell options near CheckCentralApi so the script starts
without xtrace enabled, while retaining errexit and pipefail. Enable tracing
only in a narrowly scoped region that cannot expand or log the Central URL, and
disable it before the URL-consuming commands.
---
Outside diff comments:
In
`@ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.sh`:
- Around line 30-43: Use a single readiness deadline initialized before the
sequence of WaitFor calls instead of resetting ELAPSED inside each WaitFor
invocation. Update WaitFor and its retry/timeout checks to calculate remaining
time from that shared deadline, so all readiness checks collectively remain
within one TIMEOUT budget.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 4afc50ba-93ee-42f1-942c-3bfde202dcdf
📒 Files selected for processing (3)
ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.shci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-ref.yamlci-operator/step-registry/stackrox/opp-smoke/stackrox-opp-smoke-commands.sh
🚧 Files skipped from review as they are similar to previous changes (1)
- ci-operator/step-registry/stackrox/opp-smoke/stackrox-opp-smoke-commands.sh
|
/pj-rehearse |
|
@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp4.22-interop-opp-aws |
|
@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.sh (1)
103-129: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winRetry the
central-htpasswdlookup.The admin-password extraction runs once outside
WaitFor. Withset -euo pipefail, a transientoc,jq, orbase64failure can terminate readiness before later checks retry.Move the lookup and non-empty validation into a
WaitForcheck function. Keep xtrace disabled for the complete secret-handling scope.Proposed fix
+function CheckAdminPassword () { + # Disable xtrace while reading the admin password. + set +x + typeset password="" + password="$(oc get secret -n "${CENTRAL_NS}" central-htpasswd \ + -o json | jq -r '.data.password' | base64 -d)" || { + set -x + return 1 + } + [[ -n "${password}" ]] || { + set -x + return 1 + } + ROX_ADMIN_PASSWORD="${password}" + set -x +} + +WaitFor "admin password extraction" CheckAdminPassword🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.sh` around lines 103 - 129, Move the central-htpasswd lookup and empty-password validation into a dedicated WaitFor-compatible check function, preserving the existing ROX_ADMIN_PASSWORD assignment and fatal validation behavior. Ensure the function returns failure when oc, jq, base64, or the resulting password is unavailable so WaitFor retries transient failures. Keep xtrace disabled throughout the complete secret-handling operation, and invoke this check before CheckClustersConnected.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.sh`:
- Around line 81-83: Keep xtrace disabled through every command that writes or
expands CENTRAL_URL, including the corresponding scopes at the referenced
credential and later URL blocks; move each set -x until after the protected
command completes. Add a comment immediately before each set +x documenting
whether the scope protects the Central URL or a credential, while preserving
tracing for all unrelated commands.
---
Outside diff comments:
In
`@ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.sh`:
- Around line 103-129: Move the central-htpasswd lookup and empty-password
validation into a dedicated WaitFor-compatible check function, preserving the
existing ROX_ADMIN_PASSWORD assignment and fatal validation behavior. Ensure the
function returns failure when oc, jq, base64, or the resulting password is
unavailable so WaitFor retries transient failures. Keep xtrace disabled
throughout the complete secret-handling operation, and invoke this check before
CheckClustersConnected.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: cebc18db-28f2-461b-b7f6-6ead74fe3c00
📒 Files selected for processing (4)
ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.shci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-ref.metadata.jsonci-operator/step-registry/stackrox/opp-smoke/stackrox-opp-smoke-commands.shci-operator/step-registry/stackrox/opp-smoke/stackrox-opp-smoke-ref.metadata.json
🚧 Files skipped from review as they are similar to previous changes (3)
- ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-ref.metadata.json
- ci-operator/step-registry/stackrox/opp-smoke/stackrox-opp-smoke-ref.metadata.json
- ci-operator/step-registry/stackrox/opp-smoke/stackrox-opp-smoke-commands.sh
|
@amp-rh: |
|
/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp4.22-interop-opp-aws |
|
@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.sh`:
- Around line 162-171: Update the notReady assignment in the Sensor readiness
check to preserve and propagate failures from the oc get pods pipeline instead
of masking them with || true. Ensure the surrounding readiness condition returns
failure when the query fails, allowing WaitFor to retry, while retaining the
existing notReady handling for successful queries.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 41b4439a-0b9a-4062-93f7-068421398bc3
📒 Files selected for processing (3)
ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.shci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-ref.metadata.jsonci-operator/step-registry/stackrox/opp-smoke/stackrox-opp-smoke-ref.metadata.json
🚧 Files skipped from review as they are similar to previous changes (2)
- ci-operator/step-registry/stackrox/opp-smoke/stackrox-opp-smoke-ref.metadata.json
- ci-operator/step-registry/stackrox/opp-readiness/stackrox-opp-readiness-ref.metadata.json
|
/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp4.22-interop-opp-aws |
|
@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp4.22-interop-opp-aws |
|
@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
Chai Bot Review Summary (Gate 2: Approach Validation) Chai Bot reviewed the PR and config approach. Findings:
Overall verdict: Architecture is sound. Step naming, Dockerfile, ci-operator wiring, and inter-step communication all follow established patterns correctly. The two-step readiness-gate + smoke-test pattern is the right design for OPP interop testing. Most actionable item: Address the xtrace/sensitive-data-in-logs issue in the readiness script's SHARED_DIR write section (finding #3). |
|
/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp4.22-interop-opp-aws |
|
@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@amp-rh: your |
|
/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp4.22-interop-opp-aws |
|
@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse ack |
|
@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp4.22-interop-opp-aws |
|
@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp4.22-interop-opp-aws |
|
@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@amp-rh: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Readiness gate for ACS Central and SecuredCluster before running SMOKE tests. Discovers namespaces dynamically via CRs, polls Central API health, secured-cluster connectivity, sensor pod readiness (with OOMKilled detection), and default policy count. Writes credentials and connection details to SHARED_DIR.
Runs the ACS qa-tests-backend SMOKE suite against a live ACS instance. Sparse-clones stackrox/stackrox and stackrox/scanner, materializes proto symlinks, injects Gradle init script for Gradle 9 task dependency, patches DEFAULT_CLUSTER_NAME for OPP cluster naming, and exports required credentials from Vault. JUnit XML results are copied to ARTIFACT_DIR.
Add acs-smoke-runner image (UBI9/openjdk-17 + git + oc) and insert stackrox-opp-readiness and stackrox-opp-smoke refs into the interop-opp-aws test chain.
The SMOKE step now exits 0 if tests actually ran and JUnit results were captured, even if some tests failed. Individual test failures (e.g. scanner timing issues in fresh clusters) are reported via JUnit XML artifacts for Sippy/Firewatch consumption rather than hard-failing the entire interop pipeline.
079559a to
198c8a7
Compare
|
@amp-rh, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/test all |
|
/pj-rehearse ack |
|
@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@amp-rh: This pull request references INTEROP-9236 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.1.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/pj-rehearse ack Both new ACS steps validated in the batch PR rehearsal (#83405, build 2089358099696586752 https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/83405/rehearse-83405-periodic-ci-stolostron-policy-collection-main-ocp4.22-interop-opp-aws/2089358099696586752):
|
|
@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
Blocker: GitHub API outage (2026-08-17) ci/prow/check-gh-automation failing on all openshift/release PRs due to GitHub returning HTTP 503 to Prow permission checks. See https://www.githubstatus.com/. Unrelated to PR changes; will pass on /retest once GitHub recovers. |
TL;DR
Adds ACS (StackRox) SMOKE test integration to the OPP interop pipeline (OCP 4.22). Two new step-registry refs (
stackrox-opp-readiness,stackrox-opp-smoke) and one config change; ~390 lines across 9 files.Why This Change
ACS (Advanced Cluster Security / StackRox) is one of the four core products in the OPP bundle, but until now the interop pipeline had zero automated validation that ACS was functional after installation. The pipeline installs the rhacs-operator and deploys Central + SecuredCluster, but never checks whether Central's API is healthy, whether the scanner is operational, or whether security policies are being enforced.
This gap means:
acm-opp-app, which relies on ACS scanning images pushed to Quay) may fail with misleading errors that look like ACM or Quay problems.This PR adds two steps: a readiness gate that confirms ACS is fully operational before any tests run, and a smoke test step that runs the upstream StackRox SMOKE suite against the live OPP cluster. This is a Critical-priority epic (INTEROP-8003) for Q3.
What changed
stackrox-opp-readinessstep: Polls ACS Central and SecuredCluster CRs for readiness, extracts credentials, and writes connection details toSHARED_DIR.stackrox-opp-smokestep: Sparse-clonesstackrox/stackrox+stackrox/scanner, materializes proto symlinks, patches cluster name, and runs./gradlew testSMOKE.acs-smoke-runnerimage (UBI9 + JDK 17 + oc) and inserts both steps before the existing ACM test steps.Core files (review these first)
step-registry/stackrox/opp-readiness/stackrox-opp-readiness-commands.shstep-registry/stackrox/opp-smoke/stackrox-opp-smoke-commands.shconfig/stolostron/policy-collection/stolostron-policy-collection-main__ocp4.22.yamlMechanical/config files (routine)
OWNERS(x2): Standard cspi-qe-ocp-lp team alias*-ref.yaml(x2): Step metadata (resources, timeout, from-image, docs)*-ref.metadata.json(x2): Auto-consumed by registry toolingRisk areas
opp-smoke-commands.sh:36-44): Replaces proto symlinks with directory copies. If upstream adds new symlinked dirs, this list needs updating.opp-smoke-commands.sh:46-50): Hardcodeslocal-cluster; verified withgrep -qpost-patch. If upstream renames the constant, this will fail loudly.opp-smoke-commands.sh:80-87): Required for Gradle 9.6 task dependency ordering. May need removal when upstream fixes their build scripts.Key design decisions
qa-tests-backend/andproto/.TEST_TARGET=smoke-test: Activates upstream@IgnoreIfannotations to skip known-flaky CVE test (ROX-34393).stackrox-stackrox-e2e-testsfor Quay.io and GCR/GAR registry credentials needed by image-scanning tests.stackroxorrhacs-operator; discovers via Central/SecuredCluster CRs.Expected runtime
The readiness gate polls for ~2-3 minutes. The SMOKE suite runs 28 tests (26 pass, 2 skipped via upstream flaky-test annotations) in approximately 10-15 minutes. Total step runtime: ~15-20 minutes.
Testing
Jira
Part of OPP Q3 interop improvements (umbrella: #83405).
/cc @cspi-qe-ocp-lp
Rehearsal Validation
Both new steps validated in the batch PR rehearsal (#83405, build 2089358099696586752):
stackrox-opp-readinessstackrox-opp-smoke