OKD-418: Prevent cross-version package contamination in SCOS 5.0 builds

[pskrbasu]

Summary

The CentOS SIG okd-5.0 repo on mirror.stream.centos.org contains 4.22/el9 packages (kubelet, openshift-clients) alongside the correct 5.0/el10s packages. This causes dnf to install el9 binaries into the CentOS 10 SCOS image, producing kubelet version skew (k8s 1.35 vs 1.36) and 5+ consecutive rejected OKD SCOS 5.0 nightly payloads since July 24.

Affected packages in the failing payloads:

• openshift-kubelet: 5.0.0 (el10) → 4.22.0 (el9)
• openshift-clients: 5.0.0 (el10) → 4.22.0 (el9)
• cri-o: 1.36.2 (el10s) → 1.35.5 (el10s) — not addressed by this PR (see below)
• cri-tools: 1.36.0 → 1.35.0

Two defensive fixes:

• c10s.repo: Exclude el9 packages from the SIG Cloud OKD repo. A CentOS 10 image must never install el9 binaries regardless of upstream repo contents.
• build-node-image.sh: Pin openshift-clients and openshift-kubelet to >= $OPENSHIFT_VERSION (5.0) so dnf refuses to install 4.22 packages even if rebuilt for el10.

What this does NOT fix

The cri-o downgrade (1.36.2 → 1.35.5, both el10s) cannot be fixed defensively here — cri-o uses a different versioning scheme (1.36 for OKD 5.0) with no derivable relationship to OPENSHIFT_VERSION. The upstream CBS tag for the okd-5.0 repo needs to be cleaned by the CentOS SIG Cloud maintainers to remove the 4.22 cri-o builds.

Jira

• OKD-418 — OKD SCOS 5.0 nightlies broken - CentOS SIG okd-5.0 repo contains 4.22 packages causing RPM downgrades
• Related: OKD-387 — Fixed the reverse problem (4.22 getting 5.0 packages) via OKD-387: Switch to released (mirror) repos for SCOS builds #1947

[openshift-ci-robot]

@pskrbasu: This pull request references OKD-418 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

The CentOS SIG okd-5.0 repo on mirror.stream.centos.org contains 4.22/el9 packages (kubelet, openshift-clients) alongside the correct 5.0/el10s packages. This causes dnf to install el9 binaries into the CentOS 10 SCOS image, producing kubelet version skew (k8s 1.35 vs 1.36) and 5+ consecutive rejected OKD SCOS 5.0 nightly payloads since July 24.

Affected packages in the failing payloads:

• openshift-kubelet: 5.0.0 (el10) → 4.22.0 (el9)
• openshift-clients: 5.0.0 (el10) → 4.22.0 (el9)
• cri-o: 1.36.2 (el10s) → 1.35.5 (el10s) — not addressed by this PR (see below)
• cri-tools: 1.36.0 → 1.35.0

Two defensive fixes:

• c10s.repo: Exclude el9 packages from the SIG Cloud OKD repo. A CentOS 10 image must never install el9 binaries regardless of upstream repo contents.
• build-node-image.sh: Pin openshift-clients and openshift-kubelet to >= $OPENSHIFT_VERSION (5.0) so dnf refuses to install 4.22 packages even if rebuilt for el10.

What this does NOT fix

The cri-o downgrade (1.36.2 → 1.35.5, both el10s) cannot be fixed defensively here — cri-o uses a different versioning scheme (1.36 for OKD 5.0) with no derivable relationship to OPENSHIFT_VERSION. The upstream CBS tag for the okd-5.0 repo needs to be cleaned by the CentOS SIG Cloud maintainers to remove the 4.22 cri-o builds.

Jira

• OKD-418 — OKD SCOS 5.0 nightlies broken - CentOS SIG okd-5.0 repo contains 4.22 packages causing RPM downgrades
• Related: OKD-387 — Fixed the reverse problem (4.22 getting 5.0 packages) via OKD-387: Switch to released (mirror) repos for SCOS builds #1947

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: pskrbasu
Once this PR has been reviewed and has the lgtm label, please assign mike-nguyen for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[pskrbasu]

@angelcerveraroldan / @gursewak1997 Could you help me out with what is required to get these CI checks passing? The failures do not look as if they were caused by the changes in this PR:

  Problem: package NetworkManager-ovs-1:1.58.0-1.el10 from c10s-appstream
    requires NetworkManager(x86-64) = 1:1.58.0-1.el10, but none of the
    providers can be installed
    - package NetworkManager-1:1.58.0-1.el10 from c10s-baseos is filtered
    out by exclude filtering

[angelcerveraroldan]

Yes, the issue is unrelated to this PR. Will look into it.

[angelcerveraroldan]

I think that #1952 and coreos/rhel-coreos-config#296 should help this issues from happening again

[pskrbasu]

@angelcerveraroldan So I would need to retest after those PRs get merged?

[angelcerveraroldan]

@pskrbasu Those PRs should fix the issue. I am also trying to build a new image with that issue fixed, so the test may start passing before those PRs come through. I'll update this thread once the image is built / released.

[aaradhak]

/retest

[openshift-ci]

@pskrbasu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/images	51cf939	link	true	/test images
ci/prow/okd-scos-images	51cf939	link	true	/test okd-scos-images

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.