Skip to content

GCP-429: Add e2e tests for CNCC GCP credential authentication methods#30832

Open
apahim wants to merge 1 commit intoopenshift:mainfrom
apahim:cncc-tests
Open

GCP-429: Add e2e tests for CNCC GCP credential authentication methods#30832
apahim wants to merge 1 commit intoopenshift:mainfrom
apahim:cncc-tests

Conversation

@apahim
Copy link

@apahim apahim commented Mar 4, 2026

Validates the three-tier GCP credential fallback chain (WIF → SA → GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke verification, credential rotation, WIF priority, invalid credentials, secret deletion/recovery, and IP management persistence across rotation.

Summary by CodeRabbit

  • Tests
    • Added comprehensive end-to-end CNCC GCP credential tests covering credential rotation, WIF vs service-account JSON priority, invalid-credential behavior, secret deletion/recovery, and system resilience.
    • Validates CNCC readiness, pod restart and log checks, and verifies Cloud IP management via an egress-IP workflow — ensuring CloudPrivateIPConfig creation, assignment, and persistence through credential rotations and restarts.

@openshift-ci-robot
Copy link

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 4, 2026
@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 4, 2026

@apahim: This pull request references GCP-429 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Validates the three-tier GCP credential fallback chain (WIF → SA → GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke verification, credential rotation, WIF priority, invalid credentials, secret deletion/recovery, and IP management persistence across rotation.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested review from martinkennelly and tssurya March 4, 2026 14:52
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 4, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: apahim
Once this PR has been reviewed and has the lgtm label, please assign martinkennelly for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai
Copy link

coderabbitai bot commented Mar 4, 2026

Walkthrough

Introduces a new end-to-end Go test for CNCC on GCP that snapshots/restores the cloud-credentials secret, detects credential type (WIF/SA), verifies CNCC readiness and restart behavior, exercises credential rotation and invalid-secret handling, tests secret deletion/recovery, and validates EgressIP-driven CloudPrivateIPConfig workflows.

Changes

Cohort / File(s) Summary
CNCC GCP Credentials Test Suite
test/extended/networking/cncc_gcp_credentials.go
Adds a ~500-line end-to-end test implementing CNCC GCP credential scenarios: platform detection, CRD/secret checks, cloud-credentials secret snapshot/restore, CNCC readiness and pod restart/crash checks, helper utilities (logs, restart counts), tests for WIF vs SA credential prioritization, credential rotation and invalid-credential behavior, secret deletion/recovery, and an EgressIP-based CloudPrivateIPConfig assignment workflow. Introduces an unexported helper createEgressIPFromJSON(); no exported API changes.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Test Structure And Quality ⚠️ Warning Test file has three unaddressed quality issues: improper error handling at line 282, fixed EgressIP name instead of randomized at line 432, and use of apply instead of create at line 507. Replace generic error check with apierrors.IsNotFound() at line 282, randomize EgressIP name at line 432, and change createEgressIPFromJSON to use create instead of apply at line 507.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly and accurately summarizes the main change: adding end-to-end tests for CNCC GCP credential authentication methods, which matches the changeset that introduces a comprehensive test file for this exact purpose.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Stable And Deterministic Test Names ✅ Passed All test titles use stable, descriptive static strings with no dynamic values like timestamps, UUIDs, or IP addresses. The single fmt.Sprintf usage is correctly placed in the test body, not in any test title.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.5.0)

Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions
The command is terminated due to an error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions


Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 4, 2026

@apahim: This pull request references GCP-429 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Validates the three-tier GCP credential fallback chain (WIF → SA → GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke verification, credential rotation, WIF priority, invalid credentials, secret deletion/recovery, and IP management persistence across rotation.

Summary by CodeRabbit

  • Tests
  • Added comprehensive end-to-end test suite for CNCC GCP credentials, validating credential rotation, WIF and service account JSON support, failure recovery, and IP configuration resilience.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@test/extended/networking/cncc_gcp_credentials.go`:
- Line 329: The test fixture embeds a PEM-like private key in the dummySA string
which triggers secret scanners; update the dummySA fixture (and the similar
occurrence at the other spot) to remove any real-looking PEM content and replace
the "private_key" value with a non-secret placeholder (e.g. a short
"<REDACTED_PRIVATE_KEY>" or clearly fake token) or build the JSON
programmatically without PEM text so the variable dummySA no longer contains PEM
markers like "-----BEGIN" / "-----END"; keep the same JSON keys and structure so
tests still parse but do not include PEM-looking secrets.
- Around line 70-87: The restore flow currently treats any error from
clientset.CoreV1().Secrets(cnccNamespace).Get(...) as a deleted secret; change
it to first check if the error is a NotFound (use apierrors.IsNotFound(err)) and
only then create the new secret with cnccSecretName/cnccNamespace; for all other
errors return/fail the test (o.Expect(err).NotTo(o.HaveOccurred(...)) or
propagate the error) instead of attempting Create — keep the existing Update
path for the else case where Get succeeds, and reference the same
clientset.CoreV1().Secrets(...).Get, Create and Update calls and the o.Expect
assertions to place the checks.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 23cc057b-129f-4c25-b816-2e98201af7cc

📥 Commits

Reviewing files that changed from the base of the PR and between aa56844 and d992119.

📒 Files selected for processing (1)
  • test/extended/networking/cncc_gcp_credentials.go

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 4, 2026

@apahim: This pull request references GCP-429 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Validates the three-tier GCP credential fallback chain (WIF → SA → GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke verification, credential rotation, WIF priority, invalid credentials, secret deletion/recovery, and IP management persistence across rotation.

Summary by CodeRabbit

  • Tests
  • Added comprehensive end-to-end tests for CNCC GCP credentials covering credential rotation, WIF vs service-account JSON priority, invalid-credential behaviors, secret deletion/recovery, and system resilience.
  • Includes validation of CNCC readiness, IP management via an egress IP workflow, and verification that Cloud IP assignments persist through credential rotations and restarts.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (2)
test/extended/networking/cncc_gcp_credentials.go (2)

50-57: ⚠️ Potential issue | 🟠 Major

Replace PEM-like private key literals with non-secret placeholders.

The dummySAJSON and invalidSAJSON constants contain PEM-formatted private key strings that trigger secret scanners (Gitleaks flagged both). While these are test fixtures, they will cause CI noise and security triage overhead.

Replace the private_key values with clearly fake placeholders:

Suggested fix
-	dummySAJSON = `{"type":"service_account","project_id":"dummy-project","private_key_id":"dummy","private_key":"-----BEGIN RSA PRIVATE KEY-----\nMIIBogIBAAJBALRo/7FBMHpMx2ls9k5A5feDteQa4ARWV5jNz9/v3mPpM4CPYB/m\nmivnN4g3a3+0kqe5W9USNWQ/lC3qLNjdXnECAwEAAQJAC/wG4sB/YxYILqLywb11\nNpviLqQgeKi7ZXSXT1OBF0gU8bB0OP8OLjkKGfhSW3niqxoM1RRoLvNsL7A0mBp\nwQIhAO0/GNXSdA98JxmNxVdIjPTCSAP6XEoH3R8WFiJ0JBSvAiEAxjn0EDBFKOI7\nhBk1RXlTfMHbDQHrPKFmL7y2GIPBYU8CIGl9a3PJ2QeB2Nz3IlJjSPjQ5U0ASNS\nSN1GMH2eJwYnAiEAoRxE6QUCRzBL7F3l/Dq+IYbwxZkN95Nf8CPm8GjYHD8CIQC8\nuqbJMxbRbhsPNi88e1ZYayVjPhFKEXZfCihqT/YE1Q==\n-----END RSA PRIVATE KEY-----\n","client_email":"dummy@dummy-project.iam.gserviceaccount.com","client_id":"123456789","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token"}`
+	dummySAJSON = `{"type":"service_account","project_id":"dummy-project","private_key_id":"dummy","private_key":"not-a-real-key","client_email":"dummy@dummy-project.iam.gserviceaccount.com","client_id":"123456789","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token"}`

-	invalidSAJSON = `{"type":"service_account","project_id":"invalid-project","private_key_id":"invalid","private_key":"-----BEGIN RSA PRIVATE KEY-----\nMIIBogIBAAJBALRo/7FBMHpMx2ls9k5A5feDteQa4ARWV5jNz9/v3mPpM4CPYB/m\nmivnN4g3a3+0kqe5W9USNWQ/lC3qLNjdXnECAwEAAQJAC/wG4sB/YxYILqLywb11\nNpviLqQgeKi7ZXSXT1OBF0gU8bB0OP8OLjkKGfhSW3niqxoM1RRoLvNsL7A0mBp\nwQIhAO0/GNXSdA98JxmNxVdIjPTCSAP6XEoH3R8WFiJ0JBSvAiEAxjn0EDBFKOI7\nhBk1RXlTfMHbDQHrPKFmL7y2GIPBYU8CIGl9a3PJ2QeB2Nz3IlJjSPjQ5U0ASNS\nSN1GMH2eJwYnAiEAoRxE6QUCRzBL7F3l/Dq+IYbwxZkN95Nf8CPm8GjYHD8CIQC8\nuqbJMxbRbhsPNi88e1ZYayVjPhFKEXZfCihqT/YE1Q==\n-----END RSA PRIVATE KEY-----\n","client_email":"invalid@invalid-project.iam.gserviceaccount.com","client_id":"000000000","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token"}`
+	invalidSAJSON = `{"type":"service_account","project_id":"invalid-project","private_key_id":"invalid","private_key":"not-a-real-key","client_email":"invalid@invalid-project.iam.gserviceaccount.com","client_id":"000000000","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token"}`
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 50 - 57, The
dummySAJSON and invalidSAJSON test fixtures include real-looking PEM private_key
literals that trigger secret scanners; update the JSON string values for the
"private_key" fields in both dummySAJSON and invalidSAJSON to use clearly fake
placeholders (e.g., a short non-sensitive token or a clearly marked placeholder
like "<PRIVATE_KEY_PLACEHOLDER>" or "-----BEGIN PRIVATE
KEY-----\nFAKE-KEY\n-----END PRIVATE KEY-----") so the JSON remains
syntactically valid but contains no PEM-like secret material; ensure you only
change the private_key content within the dummySAJSON and invalidSAJSON
constants.

78-98: ⚠️ Potential issue | 🟠 Major

Differentiate NotFound from other secret-read errors in restore flow.

The code treats any Get() error as if the secret was deleted (line 79), without distinguishing NotFound from actual failures like permission denied or API server issues. This hides real failures and triggers incorrect remediation.

Suggested fix
+import (
+    apierrors "k8s.io/apimachinery/pkg/api/errors"
+)
...
 existing, err := clientset.CoreV1().Secrets(cnccNamespace).Get(context.Background(), cnccSecretName, metav1.GetOptions{})
 if err != nil {
+    if !apierrors.IsNotFound(err) {
+        o.Expect(err).NotTo(o.HaveOccurred(), "failed to read CNCC secret during restore")
+        return
+    }
     // Secret was deleted; recreate it.
     newSecret := &corev1.Secret{
         ...
     }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 78 - 98, The
restore flow currently treats any error from
clientset.CoreV1().Secrets(cnccNamespace).Get(...) as a missing secret; change
it to distinguish a NotFound error using apierrors.IsNotFound(err) (from
k8s.io/apimachinery/pkg/api/errors) and only recreate the secret (newSecret with
Name cnccSecretName, Namespace cnccNamespace, Annotations/Labels/Data/Type from
originalSecret) in that case; for any other Get() error, fail the test or
surface the error via o.Expect(err).NotTo(o.HaveOccurred(), ...) (or return the
error) instead of attempting to recreate, ensuring existing is only used when
Get() succeeds and originalSecret/existing are preserved as before.
🧹 Nitpick comments (2)
test/extended/networking/cncc_gcp_credentials.go (2)

447-449: Clarify node label value intention.

Line 448 sets k8s.ovn.org/egress-assignable= with an empty value. While this may work (OVN-Kubernetes typically just checks for label presence), explicitly setting =true is more conventional and self-documenting.

Suggested fix
-		_, err = oc.AsAdmin().Run("label").Args("node", workerNodes[0].Name, "k8s.ovn.org/egress-assignable=", "--overwrite").Output()
+		_, err = oc.AsAdmin().Run("label").Args("node", workerNodes[0].Name, "k8s.ovn.org/egress-assignable=true", "--overwrite").Output()
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 447 - 449, The
label call that presently sets "k8s.ovn.org/egress-assignable=" with an empty
value should explicitly set a boolean value; update the
oc.AsAdmin().Run("label").Args invocation that targets workerNodes[0].Name to
use "k8s.ovn.org/egress-assignable=true" (and keep --overwrite) so the label
value is explicit and self-documenting while preserving the existing error check
after Output().

200-219: Use defer for logStream.Close() to prevent resource leaks.

If logStream.Read() panics, the stream won't be closed. Using defer ensures cleanup regardless of how the function exits.

Suggested fix
 		logStream, err := req.Stream(context.Background())
 		if err != nil {
 			framework.Logf("Failed to get logs for pod %s: %v", pod.Name, err)
 			continue
 		}
+		defer logStream.Close()
 		buf := new(strings.Builder)
 		_, _ = fmt.Fprintf(buf, "--- Logs from pod %s ---\n", pod.Name)
 		b := make([]byte, 4096)
 		for {
 			n, readErr := logStream.Read(b)
 			if n > 0 {
 				buf.Write(b[:n])
 			}
 			if readErr != nil {
 				break
 			}
 		}
-		logStream.Close()
 		allLogs.WriteString(buf.String())
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 200 - 219, The
log stream opened by req.Stream(...) is closed only at the end and can leak if
Read panics or returns early; immediately after acquiring logStream in the block
using clientset.CoreV1().Pods(cnccNamespace).GetLogs(...).Stream, check err as
you already do and then call defer logStream.Close() (instead of the later
explicit logStream.Close()) so the stream is always closed; remove or avoid the
later explicit Close call to prevent double-close and ensure buf and allLogs
logic remains unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@test/extended/networking/cncc_gcp_credentials.go`:
- Around line 50-57: The dummySAJSON and invalidSAJSON test fixtures include
real-looking PEM private_key literals that trigger secret scanners; update the
JSON string values for the "private_key" fields in both dummySAJSON and
invalidSAJSON to use clearly fake placeholders (e.g., a short non-sensitive
token or a clearly marked placeholder like "<PRIVATE_KEY_PLACEHOLDER>" or
"-----BEGIN PRIVATE KEY-----\nFAKE-KEY\n-----END PRIVATE KEY-----") so the JSON
remains syntactically valid but contains no PEM-like secret material; ensure you
only change the private_key content within the dummySAJSON and invalidSAJSON
constants.
- Around line 78-98: The restore flow currently treats any error from
clientset.CoreV1().Secrets(cnccNamespace).Get(...) as a missing secret; change
it to distinguish a NotFound error using apierrors.IsNotFound(err) (from
k8s.io/apimachinery/pkg/api/errors) and only recreate the secret (newSecret with
Name cnccSecretName, Namespace cnccNamespace, Annotations/Labels/Data/Type from
originalSecret) in that case; for any other Get() error, fail the test or
surface the error via o.Expect(err).NotTo(o.HaveOccurred(), ...) (or return the
error) instead of attempting to recreate, ensuring existing is only used when
Get() succeeds and originalSecret/existing are preserved as before.

---

Nitpick comments:
In `@test/extended/networking/cncc_gcp_credentials.go`:
- Around line 447-449: The label call that presently sets
"k8s.ovn.org/egress-assignable=" with an empty value should explicitly set a
boolean value; update the oc.AsAdmin().Run("label").Args invocation that targets
workerNodes[0].Name to use "k8s.ovn.org/egress-assignable=true" (and keep
--overwrite) so the label value is explicit and self-documenting while
preserving the existing error check after Output().
- Around line 200-219: The log stream opened by req.Stream(...) is closed only
at the end and can leak if Read panics or returns early; immediately after
acquiring logStream in the block using
clientset.CoreV1().Pods(cnccNamespace).GetLogs(...).Stream, check err as you
already do and then call defer logStream.Close() (instead of the later explicit
logStream.Close()) so the stream is always closed; remove or avoid the later
explicit Close call to prevent double-close and ensure buf and allLogs logic
remains unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 85ed3122-0f3b-4687-9c74-7eb01c306201

📥 Commits

Reviewing files that changed from the base of the PR and between d992119 and d90ffc5.

📒 Files selected for processing (1)
  • test/extended/networking/cncc_gcp_credentials.go

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 4, 2026

@apahim: This pull request references GCP-429 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Validates the three-tier GCP credential fallback chain (WIF → SA → GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke verification, credential rotation, WIF priority, invalid credentials, secret deletion/recovery, and IP management persistence across rotation.

Summary by CodeRabbit

  • Tests
  • Added comprehensive end-to-end CNCC GCP credential tests covering credential rotation, WIF vs service-account JSON priority, invalid-credential behavior, secret deletion/recovery, and system resilience.
  • Validates CNCC readiness, restart/log checks, and verifies Cloud IP management via an egress-IP workflow — ensuring Cloud IP assignments persist through credential rotations and restarts.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
test/extended/networking/cncc_gcp_credentials.go (2)

205-223: Consider using defer for stream cleanup.

The logStream.Close() call could be missed if a panic occurs between opening and closing. Using defer would ensure proper cleanup.

♻️ Suggested improvement
 		logStream, err := req.Stream(context.Background())
 		if err != nil {
 			framework.Logf("Failed to get logs for pod %s: %v", pod.Name, err)
 			continue
 		}
+		defer logStream.Close()
 		buf := new(strings.Builder)
 		_, _ = fmt.Fprintf(buf, "--- Logs from pod %s ---\n", pod.Name)
 		b := make([]byte, 4096)
 		for {
 			n, readErr := logStream.Read(b)
 			if n > 0 {
 				buf.Write(b[:n])
 			}
 			if readErr != nil {
 				break
 			}
 		}
-		logStream.Close()
 		allLogs.WriteString(buf.String())
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 205 - 223, The
log stream opened via req.Stream(...) is closed explicitly later which can be
skipped on panic—after successfully acquiring logStream (the variable returned
from req.Stream), immediately schedule cleanup with defer logStream.Close() so
the stream is always closed; remove the later explicit logStream.Close() call
and keep the read loop as-is (references: req.Stream, logStream, and the read
loop writing into buf/allLogs).

451-453: Clarify the egress-assignable label value.

The label is set with an empty value (k8s.ovn.org/egress-assignable=). While this works for presence-based checks, consider using an explicit value like "true" for clarity and consistency with common label conventions.

♻️ Optional clarification
-		_, err = oc.AsAdmin().Run("label").Args("node", workerNodes[0].Name, "k8s.ovn.org/egress-assignable=", "--overwrite").Output()
+		_, err = oc.AsAdmin().Run("label").Args("node", workerNodes[0].Name, "k8s.ovn.org/egress-assignable=true", "--overwrite").Output()
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 451 - 453, The
label on the first worker node is being set with an empty value; change the
label call that currently sets "k8s.ovn.org/egress-assignable=" to use an
explicit value (e.g., "k8s.ovn.org/egress-assignable=true") in the
oc.AsAdmin().Run("label") invocation that references workerNodes[0].Name, and
update any related presence-based assertions elsewhere in the test to check for
the explicit value if needed.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@test/extended/networking/cncc_gcp_credentials.go`:
- Around line 205-223: The log stream opened via req.Stream(...) is closed
explicitly later which can be skipped on panic—after successfully acquiring
logStream (the variable returned from req.Stream), immediately schedule cleanup
with defer logStream.Close() so the stream is always closed; remove the later
explicit logStream.Close() call and keep the read loop as-is (references:
req.Stream, logStream, and the read loop writing into buf/allLogs).
- Around line 451-453: The label on the first worker node is being set with an
empty value; change the label call that currently sets
"k8s.ovn.org/egress-assignable=" to use an explicit value (e.g.,
"k8s.ovn.org/egress-assignable=true") in the oc.AsAdmin().Run("label")
invocation that references workerNodes[0].Name, and update any related
presence-based assertions elsewhere in the test to check for the explicit value
if needed.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: ff3f78ea-3aff-4ca8-9452-79cc6b816d38

📥 Commits

Reviewing files that changed from the base of the PR and between d90ffc5 and ed8cc01.

📒 Files selected for processing (1)
  • test/extended/networking/cncc_gcp_credentials.go

Validates the three-tier GCP credential fallback chain (WIF → SA →
GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke
verification, credential rotation, WIF priority, invalid credentials,
secret deletion/recovery, and IP management persistence across rotation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Amador Pahim <apahim@redhat.com>
@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 4, 2026

@apahim: This pull request references GCP-429 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Validates the three-tier GCP credential fallback chain (WIF → SA → GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke verification, credential rotation, WIF priority, invalid credentials, secret deletion/recovery, and IP management persistence across rotation.

Summary by CodeRabbit

  • Tests
  • Added comprehensive end-to-end CNCC GCP credential tests covering credential rotation, WIF vs service-account JSON priority, invalid-credential behavior, secret deletion/recovery, and system resilience.
  • Validates CNCC readiness, pod restart and log checks, and verifies Cloud IP management via an egress-IP workflow — ensuring CloudPrivateIPConfig creation, assignment, and persistence through credential rotations and restarts.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
test/extended/networking/cncc_gcp_credentials.go (1)

97-101: Restore secret metadata on update path too.

When the secret exists, only Data and Type are restored. Restoring labels/annotations as well keeps “original state” behavior consistent with the recreate path.

Proposed fix
 		} else {
 			// Secret exists; update data back to original.
 			existing.Data = originalSecret.Data
 			existing.Type = originalSecret.Type
+			existing.Labels = originalSecret.Labels
+			existing.Annotations = originalSecret.Annotations
 			_, err = clientset.CoreV1().Secrets(cnccNamespace).Update(context.Background(), existing, metav1.UpdateOptions{})
 			o.Expect(err).NotTo(o.HaveOccurred(), "failed to restore CNCC secret data")
 		}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 97 - 101, When
restoring an existing secret in the update path, also copy metadata labels and
annotations from originalSecret so the secret's original state matches the
recreate path; update the block that currently sets existing.Data =
originalSecret.Data and existing.Type = originalSecret.Type to also set
existing.Labels = originalSecret.Labels and existing.Annotations =
originalSecret.Annotations before calling
clientset.CoreV1().Secrets(cnccNamespace).Update(...), and keep the same error
check using o.Expect(err).NotTo(o.HaveOccurred()).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@test/extended/networking/cncc_gcp_credentials.go`:
- Around line 432-433: The EgressIP setup uses a fixed name (egressIPName) and
then applies an EgressIP object (type EgressIP), which can silently reuse a
stale object from prior runs; change the test to generate a unique name per run
(e.g., append a timestamp/UUID) or explicitly ensure idempotence by deleting any
existing EgressIP with that name before creating/applying it; update all
occurrences around egressIPName and the EgressIP creation blocks (including the
other locations noted) so the test always creates a fresh resource or cleans up
the old one prior to apply.
- Around line 281-284: When reading the cloud-credentials secret (originalSecret
via clientset.CoreV1().Secrets(cnccNamespace).Get(...)) only treat a NotFound
error as a reason to skipper.Skipf; for all other errors (API/auth/transient
failures) fail the test instead of skipping by returning or calling the test
failure helper and include the error details. Use apierrors.IsNotFound(err)
(import "k8s.io/apimachinery/pkg/api/errors" as apierrors) to distinguish
NotFound from other errors and keep the existing skipper.Skipf(cnccNamespace,
cnccSecretName...) branch for the NotFound case.

---

Nitpick comments:
In `@test/extended/networking/cncc_gcp_credentials.go`:
- Around line 97-101: When restoring an existing secret in the update path, also
copy metadata labels and annotations from originalSecret so the secret's
original state matches the recreate path; update the block that currently sets
existing.Data = originalSecret.Data and existing.Type = originalSecret.Type to
also set existing.Labels = originalSecret.Labels and existing.Annotations =
originalSecret.Annotations before calling
clientset.CoreV1().Secrets(cnccNamespace).Update(...), and keep the same error
check using o.Expect(err).NotTo(o.HaveOccurred()).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 5878b4a7-11d4-4310-98b0-bf338906f908

📥 Commits

Reviewing files that changed from the base of the PR and between ed8cc01 and 6a54273.

📒 Files selected for processing (1)
  • test/extended/networking/cncc_gcp_credentials.go

Comment on lines +281 to +284
originalSecret, err = clientset.CoreV1().Secrets(cnccNamespace).Get(context.Background(), cnccSecretName, metav1.GetOptions{})
if err != nil {
skipper.Skipf("Skipping CNCC credential tests: cloud-credentials secret not found (possible HCP cluster): %v", err)
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Avoid skipping on non-NotFound secret read failures.

At Line 282, any Get error is treated as “secret not found” and skipped. That can hide real API/auth/transient failures as skipped tests.

Proposed fix
-		originalSecret, err = clientset.CoreV1().Secrets(cnccNamespace).Get(context.Background(), cnccSecretName, metav1.GetOptions{})
-		if err != nil {
-			skipper.Skipf("Skipping CNCC credential tests: cloud-credentials secret not found (possible HCP cluster): %v", err)
-		}
+		originalSecret, err = clientset.CoreV1().Secrets(cnccNamespace).Get(context.Background(), cnccSecretName, metav1.GetOptions{})
+		if apierrors.IsNotFound(err) {
+			skipper.Skipf("Skipping CNCC credential tests: cloud-credentials secret not found (possible HCP cluster)")
+		}
+		o.Expect(err).NotTo(o.HaveOccurred(), "failed to read CNCC cloud-credentials secret")
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 281 - 284,
When reading the cloud-credentials secret (originalSecret via
clientset.CoreV1().Secrets(cnccNamespace).Get(...)) only treat a NotFound error
as a reason to skipper.Skipf; for all other errors (API/auth/transient failures)
fail the test instead of skipping by returning or calling the test failure
helper and include the error details. Use apierrors.IsNotFound(err) (import
"k8s.io/apimachinery/pkg/api/errors" as apierrors) to distinguish NotFound from
other errors and keep the existing skipper.Skipf(cnccNamespace,
cnccSecretName...) branch for the NotFound case.

Comment on lines +432 to +433
egressIPName := "cncc-cred-test-egressip"
egressIPObj := &EgressIP{
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Prevent stale-object reuse in EgressIP setup.

Using a fixed name plus oc apply can silently reuse an old object from a prior interrupted run, making this test non-deterministic.

Proposed fix
-		egressIPName := "cncc-cred-test-egressip"
+		egressIPName := fmt.Sprintf("cncc-cred-test-egressip-%d", time.Now().UnixNano())
...
-		err = createEgressIPFromJSON(oc, egressIPJSON)
+		err = createEgressIPFromJSON(oc, egressIPJSON)
 func createEgressIPFromJSON(oc *exutil.CLI, jsonData []byte) error {
-	return oc.AsAdmin().Run("apply").Args("-f", "-").InputString(string(jsonData)).Execute()
+	return oc.AsAdmin().Run("create").Args("-f", "-").InputString(string(jsonData)).Execute()
 }

Also applies to: 458-459, 507-509

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 432 - 433, The
EgressIP setup uses a fixed name (egressIPName) and then applies an EgressIP
object (type EgressIP), which can silently reuse a stale object from prior runs;
change the test to generate a unique name per run (e.g., append a
timestamp/UUID) or explicitly ensure idempotence by deleting any existing
EgressIP with that name before creating/applying it; update all occurrences
around egressIPName and the EgressIP creation blocks (including the other
locations noted) so the test always creates a fresh resource or cleans up the
old one prior to apply.

@apahim
Copy link
Author

apahim commented Mar 4, 2026

/payload-job-with-prs periodic-ci-openshift-release-main-ci-4.22-e2e-gcp-ovn openshift/cloud-network-config-controller#206

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 4, 2026

@apahim: it appears that you have attempted to use some version of the payload command, but your comment was incorrectly formatted and cannot be acted upon. See the docs for usage info.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 4, 2026

@apahim: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-4.22-e2e-gcp-ovn

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/67ccb430-1813-11f1-9f44-422dfd494bcd-0

@openshift-ci-robot
Copy link

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 5, 2026

@apahim: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-vsphere-ovn 6a54273 link true /test e2e-vsphere-ovn
ci/prow/e2e-aws-ovn-fips 6a54273 link true /test e2e-aws-ovn-fips
ci/prow/e2e-vsphere-ovn-upi 6a54273 link true /test e2e-vsphere-ovn-upi
ci/prow/e2e-metal-ipi-ovn-ipv6 6a54273 link true /test e2e-metal-ipi-ovn-ipv6
ci/prow/e2e-gcp-ovn 6a54273 link true /test e2e-gcp-ovn

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@ricky-rav
Copy link

The new should have a working CNCC deployment with valid GCP credentials" is exactly what's failing :) https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-cloud-network-config-controller-206-openshift-origin-30832-ci-4.22-e2e-gcp-ovn/2029312221011513344

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants