WIP: Increase etcd pod-disruption severity to simulate a real failure#82272
WIP: Increase etcd pod-disruption severity to simulate a real failure#82272oharan2 wants to merge 5 commits into
Conversation
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
WalkthroughThe AWS hub tests job environment now includes disruption count, expected pod count, pod label, and wait duration variables while preserving existing component and telemetry settings. ChangesAWS hub chaos configuration
Estimated code review effort: 1 (Trivial) | ~2 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
/pj-rehearse periodic-ci-redhat-chaos-lp-chaos-main-ocp-4.22-ocp-chaos-cr--ocp--krkn-hub-tests--aws |
|
@oharan2: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@oharan2, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/redhat-chaos/pod-scenarios/etcd-disruption/redhat-chaos-pod-scenarios-etcd-disruption-ref.yaml`:
- Around line 38-39: Fix the WAIT_DURATION entry in the pod scenario reference
by closing its quoted default value, ensuring the YAML parses correctly while
preserving the existing value of 90.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 01becfde-0625-43ae-8883-ad1365030b42
📒 Files selected for processing (1)
ci-operator/step-registry/redhat-chaos/pod-scenarios/etcd-disruption/redhat-chaos-pod-scenarios-etcd-disruption-ref.yaml
|
/pj-rehearse periodic-ci-redhat-chaos-lp-chaos-main-ocp-4.22-ocp-chaos-cr--ocp--krkn-hub-tests--aws |
|
@oharan2: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: oharan2 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/redhat-chaos/lp-chaos/redhat-chaos-lp-chaos-main__ocp-4.22-ocp-chaos.yaml`:
- Around line 55-60: Remove the intentionally failing POC overrides from the
recurring cr--ocp--krkn-hub-tests--aws job configuration, including the
environment block containing DISRUPTION_COUNT, DR__RP__CR_COMP_NAME,
EXPECTED_POD_COUNT, POD_LABEL, TELEMETRY_GROUP, and WAIT_DURATION. Move them to
a dedicated manual/POC job or gate them so scheduled runs retain only
steady-state defaults.
- Line 58: Update the POD_LABEL setting in the redhat-chaos configuration to
explicitly select etcd pods with the k8s-app=etcd label instead of leaving it
empty. Preserve the existing etcd disruption scenario behavior and ensure the
selector is passed to the pod-scenario step.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: b2551d39-37cf-4140-b908-54b46c259e9a
📒 Files selected for processing (1)
ci-operator/config/redhat-chaos/lp-chaos/redhat-chaos-lp-chaos-main__ocp-4.22-ocp-chaos.yaml
| DISRUPTION_COUNT: "6" | ||
| DR__RP__CR_COMP_NAME: lp-chaos--OCP | ||
| EXPECTED_POD_COUNT: "6" | ||
| POD_LABEL: "" | ||
| TELEMETRY_GROUP: ocp4.22-nightly--aws | ||
| WAIT_DURATION: "15" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Keep the intentionally failing POC out of the recurring job.
This environment block belongs to the twice-daily cr--ocp--krkn-hub-tests--aws job. Since the PR objective says this scenario is expected to fail and is not a steady-state default, these overrides should move to a dedicated manual/POC job or be explicitly gated; otherwise every scheduled run will fail. (github.com)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/config/redhat-chaos/lp-chaos/redhat-chaos-lp-chaos-main__ocp-4.22-ocp-chaos.yaml`
around lines 55 - 60, Remove the intentionally failing POC overrides from the
recurring cr--ocp--krkn-hub-tests--aws job configuration, including the
environment block containing DISRUPTION_COUNT, DR__RP__CR_COMP_NAME,
EXPECTED_POD_COUNT, POD_LABEL, TELEMETRY_GROUP, and WAIT_DURATION. Move them to
a dedicated manual/POC job or gate them so scheduled runs retain only
steady-state defaults.
| DISRUPTION_COUNT: "6" | ||
| DR__RP__CR_COMP_NAME: lp-chaos--OCP | ||
| EXPECTED_POD_COUNT: "6" | ||
| POD_LABEL: "" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the target CI config and the referenced step-registry ref.
sed -n '1,140p' ci-operator/config/redhat-chaos/lp-chaos/redhat-chaos-lp-chaos-main__ocp-4.22-ocp-chaos.yaml
printf '\n--- step registry search ---\n'
rg -n "POD_LABEL|NAME_PATTERN|etcd-disruption|k8s-app=etcd" ci-operator/step-registry/redhat-chaos -S
printf '\n--- broader search for Krkn pod label handling ---\n'
rg -n "POD_LABEL|NAME_PATTERN" -S .Repository: openshift/release
Length of output: 8567
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the relevant step-registry file before reading its contents.
fd -a "redhat-chaos-pod-scenarios-etcd-disruption-ref.yaml" ci-operator/step-registryRepository: openshift/release
Length of output: 295
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Read the step-registry file once located.
FILE="$(fd -a "redhat-chaos-pod-scenarios-etcd-disruption-ref.yaml" ci-operator/step-registry | head -n 1)"
echo "FILE=$FILE"
if [ -n "$FILE" ]; then
wc -l "$FILE"
sed -n '1,220p' "$FILE"
fiRepository: openshift/release
Length of output: 2789
🏁 Script executed:
#!/bin/bash
set -euo pipefail
pwd
lsRepository: openshift/release
Length of output: 336
🌐 Web query:
Krkn POD_LABEL NAME_PATTERN empty falls back to NAME_PATTERN=.* etcd disruption
💡 Result:
In Krkn, the NAME_PATTERN parameter serves as a regex pattern to match pods within a target namespace when a specific POD_LABEL is not provided [1][2]. According to official Krkn documentation and scenario configurations, if NAME_PATTERN is left empty or not explicitly defined, it defaults to., which causes the scenario to target all pods matching the namespace criteria [1][2]. Regarding the context of etcd disruption, Krkn includes predefined scenarios (e.g., scenarios/openshift/etcd.yml) specifically designed to disrupt etcd components [3]. If a user executes a generic pod disruption scenario without specifying a POD_LABEL or a restrictive NAME_PATTERN, the default behavior (NAME_PATTERN=.) will attempt to disrupt all pods in the targeted namespace, which could inadvertently include etcd pods if the namespace is set to one where etcd runs (such as openshift-etcd or kube-system) [1][2][4]. Users should exercise caution when defining the NAMESPACE and NAME_PATTERN parameters to avoid unintended disruption of critical cluster infrastructure [1].
Citations:
- 1: https://github.com/krkn-chaos/krkn-hub/blob/main/docs/pod-scenarios.md
- 2: https://krkn-chaos.dev/docs/scenarios/pod-scenario/
- 3: https://krkn-chaos.dev/docs/krkn/config/
- 4: https://krkn-chaos.dev/docs/getting-started/getting-started-krkn/
Keep the etcd pod selector explicit. POD_LABEL: "" drops the k8s-app=etcd default from redhat-chaos-pod-scenarios-etcd-disruption-ref.yaml, so the step can fall back to a broad pod match and hit non-etcd pods in openshift-etcd.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/config/redhat-chaos/lp-chaos/redhat-chaos-lp-chaos-main__ocp-4.22-ocp-chaos.yaml`
at line 58, Update the POD_LABEL setting in the redhat-chaos configuration to
explicitly select etcd pods with the k8s-app=etcd label instead of leaving it
empty. Preserve the existing etcd disruption scenario behavior and ensure the
selector is passed to the pod-scenario step.
|
/pj-rehearse periodic-ci-redhat-chaos-lp-chaos-main-ocp-4.22-ocp-chaos-cr--ocp--krkn-hub-tests--aws |
|
@oharan2: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@oharan2: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
1 participant
Summary
This PR intentionally increases the severity of the
redhat-chaos-pod-scenarios-etcd-disruptionstep so that a CI run actually exercises Krkn's post-chaos gating checks, instead of completing "clean" every time.Today, with the default
DISRUPTION_COUNT=1,EXPECTED_RECOVERY_TIME=101, andWAIT_DURATION=300, etcd has enough time to recover a single killed pod before Krkn ever checks recovery status or Prometheus alerts. As a result, the step almost always passes and the gating logic (recovery check +CHECK_CRITICAL_ALERTS) is never really put to the test in this pipeline.Changes
In
redhat-chaos-pod-scenarios-etcd-disruption-ref.yaml:DISRUPTION_COUNT: Kills 6 of 6 etcd pods, breaking quorum instead of a single, easily-tolerated pod loss.WAIT_DURATION: (Decreased to15) key change - shrinks the pause before the post-chaos alert check, so Krkn queries Prometheus while etcd is still degraded, givingCHECK_CRITICAL_ALERTS(alreadyTrue) a real window to catch actual firing critical alerts instead of checking after everything has already self-healed.Why this matters
CHECK_CRITICAL_ALERTSis a post-chaos gate: Krkn queries Prometheus after the scenario and exits non-zero if anyseverity: criticalalert is firing. With the previous defaults, the longWAIT_DURATIONlet etcd fully recover before that check ran, so the gate was effectively never exercised in this job — a real quorum-loss/critical-alert scenario could pass through undetected. This change is meant to prove out (and validate) that gating actually fires when etcd is genuinely unhealthy.Expected outcome
This is expected to fail the etcd-disruption step (the point of this POC). It's a controlled way to confirm:
CHECK_CRITICAL_ALERTScorrectly catches and gates on real critical alerts fired during quorum loss.Summary by CodeRabbit
Updates the OpenShift 4.22 AWS Krkn hub test configuration to override disruption settings, including the number of disrupted pods, expected pod count, pod label, and wait duration. This increases the etcd failure scenario’s severity to exercise recovery and critical-alert handling.