Skip to content

WIP: Increase etcd pod-disruption severity to simulate a real failure #82272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
oharan2 wants to merge 8 commits into
base: main
Choose a base branch
from
Open

WIP: Increase etcd pod-disruption severity to simulate a real failure #82272

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
fadfb86
Apply breakage to the etcd pod disruption step
oharan2 Jul 22, 2026
0b14849
revert modifying recovery time, increase wait time for critical alerts
oharan2 Jul 22, 2026
14cc02d
Fix typo
oharan2 Jul 22, 2026
0dee041
Revert modifying vars in step ref, override in config
oharan2 Jul 23, 2026
480e426
Kill 6/6 pods, wait duration of 15
oharan2 Jul 23, 2026
cc8a5e3
Increase wait duration for 45s
oharan2 Aug 6, 2026
4f254f6
Fix artifact collection method
oharan2 Aug 6, 2026
053057b
Add missing grace_period for all affected steps
oharan2 Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions ci-operator/config/redhat-chaos/lp-chaos/redhat-chaos-lp-chaos-main__ocp-4.22-ocp-chaos.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,12 @@ tests:
BASE_DOMAIN: chaos.lp.devcluster.openshift.com
CHAOS_TEAM_NAME: lp-chaos
CONTAINER_ETCD_RECOVERY_TIME: "120"
DISRUPTION_COUNT: "6"
DR__RP__CR_COMP_NAME: lp-chaos--OCP
EXPECTED_POD_COUNT: "6"
POD_LABEL: ""

@coderabbitai coderabbitai Bot Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the target CI config and the referenced step-registry ref.
sed -n '1,140p' ci-operator/config/redhat-chaos/lp-chaos/redhat-chaos-lp-chaos-main__ocp-4.22-ocp-chaos.yaml

printf '\n--- step registry search ---\n'
rg -n "POD_LABEL|NAME_PATTERN|etcd-disruption|k8s-app=etcd" ci-operator/step-registry/redhat-chaos -S

printf '\n--- broader search for Krkn pod label handling ---\n'
rg -n "POD_LABEL|NAME_PATTERN" -S .

Repository: openshift/release

Length of output: 8567

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Map the relevant step-registry file before reading its contents.
fd -a "redhat-chaos-pod-scenarios-etcd-disruption-ref.yaml" ci-operator/step-registry

Repository: openshift/release

Length of output: 295

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Read the step-registry file once located.
FILE="$(fd -a "redhat-chaos-pod-scenarios-etcd-disruption-ref.yaml" ci-operator/step-registry | head -n 1)"
echo "FILE=$FILE"
if [ -n "$FILE" ]; then
  wc -l "$FILE"
  sed -n '1,220p' "$FILE"
fi

Repository: openshift/release

Length of output: 2789

🏁 Script executed:

#!/bin/bash
set -euo pipefail
pwd
ls

Repository: openshift/release

Length of output: 336

🌐 Web query:

Krkn POD_LABEL NAME_PATTERN empty falls back to NAME_PATTERN=.* etcd disruption

💡 Result:

In Krkn, the NAME_PATTERN parameter serves as a regex pattern to match pods within a target namespace when a specific POD_LABEL is not provided [1][2]. According to official Krkn documentation and scenario configurations, if NAME_PATTERN is left empty or not explicitly defined, it defaults to., which causes the scenario to target all pods matching the namespace criteria [1][2]. Regarding the context of etcd disruption, Krkn includes predefined scenarios (e.g., scenarios/openshift/etcd.yml) specifically designed to disrupt etcd components [3]. If a user executes a generic pod disruption scenario without specifying a POD_LABEL or a restrictive NAME_PATTERN, the default behavior (NAME_PATTERN=.) will attempt to disrupt all pods in the targeted namespace, which could inadvertently include etcd pods if the namespace is set to one where etcd runs (such as openshift-etcd or kube-system) [1][2][4]. Users should exercise caution when defining the NAMESPACE and NAME_PATTERN parameters to avoid unintended disruption of critical cluster infrastructure [1].

Citations:

Keep the etcd pod selector explicit. POD_LABEL: "" drops the k8s-app=etcd default from redhat-chaos-pod-scenarios-etcd-disruption-ref.yaml, so the step can fall back to a broad pod match and hit non-etcd pods in openshift-etcd.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/config/redhat-chaos/lp-chaos/redhat-chaos-lp-chaos-main__ocp-4.22-ocp-chaos.yaml`
at line 58, Update the POD_LABEL setting in the redhat-chaos configuration to
explicitly select etcd pods with the k8s-app=etcd label instead of leaving it
empty. Preserve the existing etcd disruption scenario behavior and ensure the
selector is passed to the pod-scenario step.

TELEMETRY_GROUP: ocp4.22-nightly--aws
WAIT_DURATION: "45"
post:
- ref: mpiit-data-router-reporter
- chain: ipi-aws-post
Expand Down
1 change: 1 addition & 0 deletions ...tep-registry/redhat-chaos/pod-scenarios/custom/redhat-chaos-pod-scenarios-custom-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ ref:
name: prow-scripts
tag: latest
commands: redhat-chaos-pod-scenarios-custom-commands.sh
grace_period: 10m
credentials:
- namespace: test-credentials
name: cluster-secrets-aws-chaos
Expand Down
41 changes: 24 additions & 17 deletions ...haos/pod-scenarios/etcd-disruption/redhat-chaos-pod-scenarios-etcd-disruption-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,24 +56,31 @@ else
export HEALTH_CHECK_URL=https://$console_url
echo "Using console health check URL: $HEALTH_CHECK_URL"
fi
set -o nounset
set -o pipefail
set -x

./pod-scenarios/prow_run.sh
rc=$?
echo "Done running the test!"
oc get pods -n $TARGET_NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.status.conditions[?(@.type=="Ready")].lastTransitionTime}{"\t"}{.status.startTime}{"\t"}{.metadata.creationTimestamp}{"\n"}{end}'

cat /tmp/*.log
if [[ $TELEMETRY_EVENTS_BACKUP == "True" ]]; then
cp /tmp/events.json ${ARTIFACT_DIR}/events.json
fi
collect_artifacts() {
local rc=$?
# Disable errexit here: prow_run.sh may have exited non-zero and triggered
# this trap, so none of the collection commands below should abort the trap.
set +o errexit
echo "Done running the test!"
oc get pods -n "$TARGET_NAMESPACE" -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.status.conditions[?(@.type=="Ready")].lastTransitionTime}{"\t"}{.status.startTime}{"\t"}{.metadata.creationTimestamp}{"\n"}{end}'
cat /tmp/*.log 2>/dev/null
if [[ "${TELEMETRY_EVENTS_BACKUP:-}" == "True" && -f /tmp/events.json ]]; then
cp /tmp/events.json "${ARTIFACT_DIR}/events.json"
fi
if [[ -f /tmp/report.out.pdf ]]; then
cp /tmp/report.out.pdf "${ARTIFACT_DIR}/kraken.report.pdf"
fi
echo "Return code: $rc"
exit "$rc"
}

# errexit will abort the script the instant prow_run.sh fails, before any
# artifact-collection commands can run. A trap on EXIT fires regardless of
# how the shell exits, so it's used here to guarantee artifacts (including
# the PDF report) are always copied to ARTIFACT_DIR.
trap collect_artifacts EXIT

if [[ -f /tmp/report.out.pdf ]]; then
cp /tmp/report.out.pdf ${ARTIFACT_DIR}/kraken.report.pdf
fi
set -euxo pipefail; shopt -s inherit_errexit

echo "Return code: $rc"
exit $rc
./pod-scenarios/prow_run.sh
1 change: 1 addition & 0 deletions ...t-chaos/pod-scenarios/etcd-disruption/redhat-chaos-pod-scenarios-etcd-disruption-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ ref:
name: prow-scripts
tag: latest
commands: redhat-chaos-pod-scenarios-etcd-disruption-commands.sh
grace_period: 10m
credentials:
- namespace: test-credentials
name: cluster-secrets-aws-chaos
Expand Down
1 change: 1 addition & 0 deletions ...s/kube-apiserver-disruption/redhat-chaos-pod-scenarios-kube-apiserver-disruption-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ ref:
name: prow-scripts
tag: latest
commands: redhat-chaos-pod-scenarios-kube-apiserver-disruption-commands.sh
grace_period: 10m
credentials:
- namespace: test-credentials
name: cluster-secrets-aws-chaos
Expand Down
1 change: 1 addition & 0 deletions ...aos/pod-scenarios/ovn-cp-disruption/redhat-chaos-pod-scenarios-ovn-cp-disruption-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ ref:
name: prow-scripts
tag: latest
commands: redhat-chaos-pod-scenarios-ovn-cp-disruption-commands.sh
grace_period: 10m
credentials:
- namespace: test-credentials
name: cluster-secrets-aws-chaos
Expand Down
1 change: 1 addition & 0 deletions ...hat-chaos/pod-scenarios/ovn-disruption/redhat-chaos-pod-scenarios-ovn-disruption-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ ref:
name: prow-scripts
tag: latest
commands: redhat-chaos-pod-scenarios-ovn-disruption-commands.sh
grace_period: 10m
credentials:
- namespace: test-credentials
name: cluster-secrets-aws-chaos
Expand Down
1 change: 1 addition & 0 deletions ...scenarios/prometheus-disruption/redhat-chaos-pod-scenarios-prometheus-disruption-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ ref:
name: prow-scripts
tag: latest
commands: redhat-chaos-pod-scenarios-prometheus-disruption-commands.sh
grace_period: 10m
credentials:
- namespace: test-credentials
name: cluster-secrets-aws-chaos
Expand Down
1 change: 1 addition & 0 deletions ...s/pod-scenarios/random-system-pods/redhat-chaos-pod-scenarios-random-system-pods-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ ref:
name: prow-scripts
tag: latest
commands: redhat-chaos-pod-scenarios-random-system-pods-commands.sh
grace_period: 10m
credentials:
- namespace: test-credentials
name: cluster-secrets-aws-chaos
Expand Down