OLS-3670 - adding lseval presubmits evaluation to ols

[JoaoFula]

Summary

Adds a new ols-lseval-presubmit presubmit
job for openshift/lightspeed-service that
runs the lseval short dataset (10 questions)
across 6 providers sequentially.

• Trigger: run_if_changed on eval
  configs, test files, provider code, prompts,
  RAG, and tools
• Cluster: AWS us-east-1 FIPS 4.22,
  1h30m timeout
• Credentials: openai, azure_openai,
  watsonx, vertex, bedrock-iam (all from
  test-credentials namespace)
• Script:
  tests/scripts/test-lseval-presubmit.sh

Dependencies

• Companion lightspeed-service PR must merge
  first (adds the test script and configs
  this job invokes)

Test plan

• make update regenerates cleanly
  (already run, generated jobs included in
  this PR)
• Job triggers on PRs touching
  eval/provider files after both PRs merge

Summary by CodeRabbit

Adds an ols-lseval-presubmit OpenShift CI job for openshift/lightspeed-service. The job runs the 10-question lseval short dataset sequentially across six providers on an AWS us-east-1 FIPS OpenShift 4.22 cluster, with a 1h30m timeout.

It triggers for changes to evaluation configuration, tests, provider code, prompts, RAG, or tools; mounts provider credentials from test-credentials; depends on lightspeed-service-api; and executes tests/scripts/test-lseval-presubmit.sh.

[openshift-merge-bot]

[REHEARSALNOTIFIER]
@JoaoFula: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-lightspeed-service-main-ols-lseval-presubmit	openshift/lightspeed-service	presubmit	Presubmit changed

Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: JoaoFula
Once this PR has been reviewed and has the lgtm label, please assign bparees for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• ci-operator/config/openshift/lightspeed-service/OWNERS
• ci-operator/jobs/openshift/lightspeed-service/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 1b5485bf-88d4-41a2-8f51-01bf9c2c76dd

📥 Commits

Reviewing files that changed from the base of the PR and between 02c0c92 and 05e94ce.

⛔ Files ignored due to path filters (1)

• ci-operator/jobs/openshift/lightspeed-service/openshift-lightspeed-service-main-presubmits.yaml is excluded by !ci-operator/jobs/**

📒 Files selected for processing (1)

• ci-operator/config/openshift/lightspeed-service/openshift-lightspeed-service-main.yaml

---

Walkthrough

Adds an ols-lseval-presubmit OpenShift CI test job targeting an AMD64 AWS FIPS cluster, conditionally triggered by relevant path changes, and configured with provider credentials and the LSEval presubmit script.

Changes

LSEval presubmit workflow

Layer / File(s)	Summary
Configure LSEval presubmit execution ci-operator/config/openshift/lightspeed-service/openshift-lightspeed-service-main.yaml	Adds the cluster claim, path-based trigger, generic workflow steps, provider credential mounts and environment variables, lightspeed-service-api image dependency, resource request, and test-lseval-presubmit.sh execution.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is clearly related to the main change: adding an lseval presubmit evaluation job to OLS.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	PASS: The only changed artifact is a CI YAML job config; no Ginkgo It/Describe/Context/When titles were added or modified, so there are no unstable test names.
Test Structure And Quality	✅ Passed	No Ginkgo test code was changed; this PR only adds CI/job YAML, so the test-structure rules don’t apply.
Microshift Test Compatibility	✅ Passed	Only CI config/trigger wiring changed; no new Ginkgo tests or MicroShift-sensitive APIs were added in this PR.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	Only CI/prowgen YAML changed; no new Ginkgo test bodies or SNO-sensitive test logic were added.
Topology-Aware Scheduling Compatibility	✅ Passed	Only ci-operator config/jobs changed; no deployment manifests/controllers, and no nodeSelector/anti-affinity/topology-spread scheduling fields were added.
Ote Binary Stdout Contract	✅ Passed	PR only adds ci-operator YAML; no binary/process-level code or stdout writes changed, so the OTE stdout contract isn't impacted.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PASS: This PR only adds CI job YAML; no new Ginkgo e2e test code or network logic is introduced here, so the IPv6/disconnected check is not applicable.
No-Weak-Crypto	✅ Passed	PASS: The diff only adds CI job config and secret env vars; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons are present.
Container-Privileges	✅ Passed	The new CI job adds no privileged flags, host namespace settings, SYS_ADMIN, or allowPrivilegeEscalation; repo search found none.
No-Sensitive-Data-In-Logs	✅ Passed	The new CI job only mounts secrets and exports them for the script; no added echo/print/logging of secret values or PII appears in the changed files.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

@JoaoFula: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.