INTEROP-9337: Fix OCP 5.0 upgrade config and harden upgrade script

[amp-rh]

Summary

Fix multiple issues in the OCP 5.0 upgrade interop test configuration and harden the upgrade script. Replaces #82182 with expanded scope.

Changes

1. Remove deprecated operators from 5.0 upgrade config

• Remove rhacs-operator (absent from v5.0 catalog)
• Remove odf-operator (absent from v5.0 catalog)
• Keep quay-operator, fix channel from stable-3.18 to stable-3.17 (present in v5.0 catalog; stable-3.18 does not exist)

2. Add missing backup and preflight steps

The 5.0 upgrade config had only 4 test steps while the 4.22 config has 6. Added interop-opp-backup and interop-opp-preflight before the upgrade step.

3. Add proxy-conf.sh sourcing

Source ${SHARED_DIR}/proxy-conf.sh when present, matching the pattern used by other interop step scripts.

4. Fix CSV matching from substring to prefix

Change index($2, op) > 0 to index($2, op) == 1 in ValidateOppOperators() to avoid false CSV matches.

Verification

• Operator catalog presence verified using Red Hat operator catalog index (synced Jun-Jul 2026)
• rhacs-operator: absent from v5.0 catalog
• odf-operator: absent from v5.0 catalog
• quay-operator: present with channels up to stable-3.17

References

• INTEROP-9337
• LPINTEROP-6885: original failure bug
• Replaces INTEROP-9337: Remove deprecated rhacs-operator from OCP 5.0 upgrade interop config #82182

Summary by CodeRabbit

• Updates the OCP 5.0 policy-collection interop workflow by removing rhacs-operator and odf-operator, switching quay-operator to stable-3.17, and adding backup and preflight steps.
• Hardens the upgrade script by sourcing optional proxy settings, replacing unavailable jq usage with oc templates and awk, and preserving health, release, admin-gate, and CSV validation checks.

[openshift-ci-robot]

@amp-rh: This pull request references INTEROP-9337 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Fix multiple issues in the OCP 5.0 upgrade interop test configuration and harden the upgrade script. Replaces #82182 with expanded scope.

Changes

1. Remove deprecated operators from 5.0 upgrade config

• Remove rhacs-operator (absent from v5.0 catalog)
• Remove odf-operator (absent from v5.0 catalog)
• Keep quay-operator, fix channel from stable-3.18 to stable-3.17 (present in v5.0 catalog; stable-3.18 does not exist)

2. Add missing backup and preflight steps

The 5.0 upgrade config had only 4 test steps while the 4.22 config has 6. Added interop-opp-backup and interop-opp-preflight before the upgrade step.

3. Add proxy-conf.sh sourcing

Source ${SHARED_DIR}/proxy-conf.sh when present, matching the pattern used by other interop step scripts.

4. Fix CSV matching from substring to prefix

Change index($2, op) > 0 to index($2, op) == 1 in ValidateOppOperators() to avoid false CSV matches.

Verification

• Operator catalog presence verified using Red Hat operator catalog index (synced Jun-Jul 2026)
• rhacs-operator: absent from v5.0 catalog
• odf-operator: absent from v5.0 catalog
• quay-operator: present with channels up to stable-3.17

References

• INTEROP-9337
• LPINTEROP-6885: original failure bug
• Replaces INTEROP-9337: Remove deprecated rhacs-operator from OCP 5.0 upgrade interop config #82182

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Warning

Review limit reached

@amp-rh, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 30 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 535610bc-988d-4e8b-a836-d10e7e16f1b2

📥 Commits

Reviewing files that changed from the base of the PR and between 5c7d2ee and 1a5ea35.

📒 Files selected for processing (2)

• ci-operator/config/stolostron/policy-collection/stolostron-policy-collection-main__ocp5.0-upgrade.yaml
• ci-operator/step-registry/interop/opp/upgrade/interop-opp-upgrade-commands.sh

Walkthrough

The OCP 5.0 upgrade policy adds backup and preflight steps, changes operator configuration, sources optional proxy settings, and replaces several jq-based queries in the OPP upgrade script with go-template and awk parsing.

Changes

OPP upgrade workflow

Layer / File(s)	Summary
Upgrade policy flow ci-operator/config/stolostron/policy-collection/stolostron-policy-collection-main__ocp5.0-upgrade.yaml	The Quay operator channel changes to stable-3.17, RHACS and ODF are removed, and backup and preflight steps are added before upgrade.
Upgrade runtime parsing ci-operator/step-registry/interop/opp/upgrade/interop-opp-upgrade-commands.sh	Optional proxy configuration is sourced, and digest, admin-gate, and target-version extraction no longer uses jq.
Cluster and operator validation ci-operator/step-registry/interop/opp/upgrade/interop-opp-upgrade-commands.sh	Diagnostics and platform or OPP operator validation use go-template and awk pipelines while preserving existing failure checks.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• openshift/release#82182: Updates the same OCP 5.0 upgrade policy and replaces similar jq parsing in the OPP upgrade script.
• openshift/release#82372: Changes the related OCP 4.22 upgrade policy’s Quay operator channel to stable-3.17.

---

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 1 warning)

Check name	Status	Explanation	Resolution
No-Sensitive-Data-In-Logs	❌ Error	The new source "${SHARED_DIR}/proxy-conf.sh" runs under set -x, and that file exports proxy_auth@host URLs, which can leak credentials/internal hostnames.	Disable xtrace before sourcing proxy-conf.sh (or source it after set +x) so the exported proxy values are not echoed into logs.
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (13 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: OCP 5.0 upgrade config fixes and upgrade script hardening.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	No Ginkgo-style test titles were added or modified; the diff only touches YAML config and a shell script.
Test Structure And Quality	✅ Passed	This PR only changes a ci-operator YAML and a shell script; no Ginkgo test code is modified, so the test-structure check is not applicable.
Microshift Test Compatibility	✅ Passed	The PR changes only CI config and a shell script; no new Ginkgo test definitions or MicroShift-unsupported APIs/features were added.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	No new Ginkgo e2e tests were added; the PR only changes CI config and a shell upgrade script, so the SNO test check is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	No deployment manifests, operator code, or controllers were changed; the PR only edits CI config and a shell script, with no scheduling constraints introduced.
Ote Binary Stdout Contract	✅ Passed	PR only changes ci-operator YAML and a shell step script; no OTE Go main/init/TestMain/suite stdout writes were introduced.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No new Ginkgo e2e tests or IPv4-specific logic were added; the only public curl is pre-existing in the upgrade script, not introduced by this PR.
No-Weak-Crypto	✅ Passed	Changed files contain no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB or custom crypto; only SHA-256/signature checks and non-secret string comparisons.
Container-Privileges	✅ Passed	Changed files contain only CI config and shell logic; no privileged, hostPID/hostNetwork/hostIPC, SYS_ADMIN, root, or allowPrivilegeEscalation settings found.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: amp-rh
Once this PR has been reviewed and has the lgtm label, please assign jan-law for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• ci-operator/config/stolostron/policy-collection/OWNERS
• ci-operator/step-registry/interop/opp/upgrade/OWNERS [amp-rh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/config/stolostron/policy-collection/stolostron-policy-collection-main__ocp5.0-upgrade.yaml`:
- Line 60: Update the `interop-opp-backup`, `interop-opp-preflight`, and
`interop-opp-upgrade` steps to pass an explicit `OPP_OPERATORS` value matching
the policy’s current operator list, excluding `rhacs-operator` and
`odf-operator`; alternatively, restore those operators to the list so all
workflow checks use the same scope.

In
`@ci-operator/step-registry/interop/opp/upgrade/interop-opp-upgrade-commands.sh`:
- Line 373: Update the awk predicate in the oppNamespaces derivation to require
each CSV name field ($2) to start with the operator name, matching the prefix
check used at line 350, rather than matching the operator name anywhere in the
field. Keep the existing namespace collection and output behavior unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: fb503146-12f3-4ac8-9cd1-618be28be59e

📥 Commits

Reviewing files that changed from the base of the PR and between 1b1b4a8 and 5c7d2ee.

📒 Files selected for processing (2)

• ci-operator/config/stolostron/policy-collection/stolostron-policy-collection-main__ocp5.0-upgrade.yaml
• ci-operator/step-registry/interop/opp/upgrade/interop-opp-upgrade-commands.sh

[amp-rh]

/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp4.22-upgrade-interop-opp-upgrade-aws

[openshift-merge-bot]

@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[redhat-chai-bot]

PR Review: INTEROP-9337 — OCP 5.0 Upgrade Config & Script Hardening

Overall: Approach is sound, but has one critical gap that will cause runtime failures.

The goals are well-motivated — removing unavailable operators from the 5.0 config, fixing the Quay channel, aligning steps with 4.22, and eliminating the jq dependency. CI is clean. However, a few issues should be addressed before merge.

---

🚨 Critical: OPP_OPERATORS default mismatch — will cause test failures

The 5.0 YAML config now only installs 2 operators (ACM, Quay), but all three OPP steps (interop-opp-backup, interop-opp-preflight, interop-opp-upgrade) default OPP_OPERATORS to all 4 operators including rhacs-operator and odf-operator.

At runtime:

• interop-opp-preflight will check compatibility for RHACS/ODF and likely exit with code 3
• interop-opp-upgrade's ValidateOppOperators() will fail when it can't find CSVs for those operators
• interop-opp-backup will try to capture state for non-existent operators

The 4.22 config doesn't have this problem because it installs all 4 operators, matching the defaults.

Fix: Add an OPP_OPERATORS env override in the 5.0 YAML config at the steps.env level:

env:
  OPP_OPERATORS: "advanced-cluster-management,quay-operator"

This scopes the override to this config without changing the shared step defaults that the 4.22 config relies on.

---

⚠️ Major: Prefix matching inconsistency in namespace derivation (line 373)

ValidateOppOperators() at line 350 correctly uses index($2, op) == 1 (prefix match) to find CSVs. But the namespace derivation at line 373 still uses index($2, arr[i]) > 0 (substring match anywhere). An unrelated CSV whose name happens to contain an operator name substring could incorrectly add its namespace to the readiness checks, causing a false failure.

Fix: Change > 0 to == 1 in the namespace awk command:

oppNamespaces="$(echo "${allCsvsJson}" | awk -F'\t' -v ops="${OPP_OPERATORS}" \
  'BEGIN{n=split(ops,arr,",")} {for(i=1;i<=n;i++) if(index($2,arr[i])==1){ns[$1]=1;break}} END{for(k in ns) print k}')"

---

ℹ️ Minor: Proxy credential leak under set -x

The script runs with set -x for debug tracing. Sourcing proxy-conf.sh will echo the exported proxy URLs (which may contain proxy_auth@host credentials) into CI build logs.

Fix: Suppress tracing around the source:

if [[ -f "${SHARED_DIR}/proxy-conf.sh" ]]; then
    set +x
    source "${SHARED_DIR}/proxy-conf.sh"
    set -x
fi

---

ℹ️ Minor: Fragile awk-based JSON parsing

Several jq replacements use positional field parsing (e.g. awk -F'"' '/\"digest\"/{print $4; exit}'). While oc output is generally predictable, these are more fragile than jq and could break if output formatting changes. For this PR's scope (removing the jq dependency) it's a reasonable tradeoff — the go-template replacements for oc get are solid. Worth noting for future maintainers.

---

Summary

Priority	Issue	Fix
Critical	OPP_OPERATORS mismatch	Add env override in 5.0 YAML
Major	Prefix match inconsistency	Change > 0 to == 1 on line 373
Minor	Proxy credential leak	Wrap source with set +x / set -x
Minor	Fragile awk JSON parsing	Acceptable for now, note for future

I'd recommend addressing at least the critical and major items before merge. The rest of the approach looks solid. 👍

---

Review by Chai Bot ([SHIP Public])

[redhat-chai-bot]

Follow-up Review: All Flagged Items Addressed ✅

Re-checked the current diff against the four items from my earlier review. All have been addressed.

---

🚨 Critical: OPP_OPERATORS mismatch → ✅ Fixed

An explicit OPP_OPERATORS override was added in the 5.0 YAML config:

OPP_OPERATORS: advanced-cluster-management,quay-operator

This correctly scopes the OPP steps to only the operators actually installed by this config, without affecting the 4.22 config's defaults.

⚠️ Major: Prefix matching inconsistency → ✅ Fixed

The namespace derivation awk command now uses index($2,arr[i])==1 (prefix match), consistent with ValidateOppOperators() at line 350:

for ns in $(awk -F'\t' -v ops="${OPP_OPERATORS}" \
  'BEGIN{n=split(ops,arr,",")} {for(i=1;i<=n;i++) if(index($2,arr[i])==1){ns[$1]=1;break}} END{for(k in ns) print k}' <<< "${allCsvsJson}"); do

ℹ️ Minor: Proxy credential leak → ✅ Fixed

The proxy-conf.sh sourcing is now wrapped with set +x / set -x:

if [[ -f "${SHARED_DIR}/proxy-conf.sh" ]]; then
    set +x
    source "${SHARED_DIR}/proxy-conf.sh"
    set -x
fi

ℹ️ Minor: Fragile awk JSON parsing → Acknowledged (no change needed)

As noted previously, this is an acceptable tradeoff for eliminating the jq dependency. The go-template replacements are solid.

---

Verdict: All critical and major items are resolved. LGTM from a code correctness standpoint. 👍

---

Follow-up review by Chai Bot ([SHIP Public])

[openshift-merge-bot]

[REHEARSALNOTIFIER]
@amp-rh: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
periodic-ci-stolostron-policy-collection-main-ocp5.0-upgrade-interop-opp-upgrade-aws	N/A	periodic	Ci-operator config changed
periodic-ci-stolostron-policy-collection-main-ocp4.22-upgrade-interop-opp-upgrade-aws	N/A	periodic	Registry content changed

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[amp-rh]

/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp5.0-upgrade-interop-opp-upgrade-aws

[amp-rh]

/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp5.0-upgrade-interop-opp-upgrade-aws

[openshift-merge-bot]

@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-merge-bot]

@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[amp-rh]

/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp4.22-upgrade-interop-opp-upgrade-aws

[openshift-merge-bot]

@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[amp-rh]

/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp4.22-upgrade-interop-opp-upgrade-aws

[openshift-merge-bot]

@amp-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci]

@amp-rh: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/periodic-ci-stolostron-policy-collection-main-ocp4.22-upgrade-interop-opp-upgrade-aws	3667882	link	unknown	/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp4.22-upgrade-interop-opp-upgrade-aws
ci/rehearse/periodic-ci-stolostron-policy-collection-main-ocp5.0-upgrade-interop-opp-upgrade-aws	3667882	link	unknown	/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp5.0-upgrade-interop-opp-upgrade-aws

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.