MOB-43135: EFT failed_transactions.json for browser-based tests - #2017
Open
pjain05 wants to merge 12 commits into
Open
MOB-43135: EFT failed_transactions.json for browser-based tests#2017pjain05 wants to merge 12 commits into
pjain05 wants to merge 12 commits into
Conversation
…owser-based Selenium tests Opt-in via modules.blazemeter.generate-failed-transactions (default off). Extract ERRTYPE_* classification into bzt/modules/eft.py; generate in BlazeMeterUploader.post_process; reuse existing artifact upload. JTLErrorsReader unchanged; EFT not gated in taurus. Includes brownfield-context, spec, plan, tasks, test-scenarios, design-contract, evidence-index. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… (classify/recover/build) Executor-agnostic EFT helper for browser-based (Selenium/Apiritif) tests: - classify_failure() maps a failure to an error_item_skel-shaped dict, typing ERRTYPE_ASSERT when an assertion name is present else ERRTYPE_ERROR (T011). - recover_assertion_name() returns a synthetic assert::<label> for assertion failures whose real name is not recoverable, None for general errors (T012, A-ASSERT). - build_failed_transactions() assembles the reportInfo + three-way-split artifact (T007). - Graceful degradation: empty responseBodies, null rc tolerated (T016). Classification extracted from JTLErrorsReader.find_failure; jmeter.py unchanged (FR-008). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…post_process - Read modules.blazemeter.generate-failed-transactions (default off) and skip all generation when falsy — zero behaviour change (T014, FR-006). - When on, iterate FunctionalAggregator.cumulative_results, classify each FAILED/BROKEN sample via bzt.modules.eft, and write failed_transactions.json to engine.artifacts_dir before __upload_artifacts bundles it into artifacts.zip (T008, FR-001/FR-002). - Whole path is defensively wrapped so a build/write error is logged and never breaks post_process (Failure Modes contract). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ons flag Add BlazeMeter reporter docs for the opt-in failed_transactions.json artifact for browser-based tests, matching specs/mob-43135/quickstart.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an opt-in EFT (Exclude Failed Transactions) flow to generate a failed_transactions.json artifact for browser-based (Selenium/Apiritif) runs, so it can be uploaded via the existing artifacts.zip upload path and consumed by BlazeMeter for error categorization.
Changes:
- Introduces
bzt/modules/eft.pyto classify functional failures and assemble thefailed_transactions.jsonartifact structure. - Hooks artifact generation into
BlazeMeterUploader.post_process()behindmodules.blazemeter.generate-failed-transactions(default off), and documents the new flag. - Adds unit/integration-style tests plus a real fixture for assertion-vs-general-error behavior.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
bzt/modules/eft.py |
New EFT helper module: assertion detection, assertion-name recovery (with synthetic fallback), and artifact assembly. |
bzt/modules/blazemeter/blazemeter_reporter.py |
Adds flag-gated generation of failed_transactions.json during post_process() so it’s included in artifact upload. |
tests/unit/modules/test_eft.py |
Unit tests for EFT artifact shape and classification behavior (including null rc / empty responseBodies expectations). |
tests/unit/modules/test_eft_assertion_probe.py |
“Probe” test validating real-fixture assertion-name recoverability assumptions and fallback behavior. |
tests/unit/modules/blazemeter/test_blazemeter_eft.py |
Integration-style unit tests ensuring the artifact is written (or not) depending on the opt-in flag. |
tests/resources/selenium/eft_assertion_failure.ldjson |
Realistic functional-result fixture rows used by EFT tests/probe. |
site/dat/docs/BlazemeterReporter.md |
Documents generate-failed-transactions config flag and the emitted artifact semantics. |
CLAUDE.md |
Adds repository guidance describing the EFT helper and where it’s invoked. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…t gaps - Fix Counter() not JSON-serializable in classify_failure() → use plain dict - Fix except BaseException → except Exception (don't swallow KeyboardInterrupt) - Remove dead _sample_is_assertion() method and unreachable fallback - Remove unreachable ERRTYPE_SUBSAMPLE branch in _transaction_from_item() - Remove broken schema file reference in test (fixes CI FileNotFoundError) - Add integration tests for general error and mixed failure scenarios - Clean up phantom spec.md/tasks.md references in test docstrings - Remove internal CLAUDE.md addition from open-source repo Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Collaborator
Copilot Review Items — Resolved in 46fe5ce1. Schema file 2. Additional fixes in the same commit:
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2017 +/- ##
==========================================
+ Coverage 88.11% 88.17% +0.06%
==========================================
Files 73 74 +1
Lines 20964 21051 +87
==========================================
+ Hits 18471 18559 +88
+ Misses 2493 2492 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Cover _find_functional_aggregator() returning None when engine uses ConsolidatingAggregator (load test) instead of FunctionalAggregator. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
nose2 requires __init__.py to discover tests in subdirectories. Without it, tests/unit/modules/blazemeter/test_blazemeter_eft.py was not being run by CI's coverage command, causing 0% patch coverage on blazemeter_reporter.py EFT code. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
test_engine_metrics.py used bare `from modules.blazemeter...` and `from unit import...` instead of fully qualified `from bzt.modules...` and `from tests.unit import...`. Never caught because the missing __init__.py prevented nose2 from discovering it. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When no FunctionalAggregator is present (BBT / performance mode), _generate_failed_transactions_artifact now reads from ConsolidatingAggregator.cumulative (KPISets) and classifies errors by ERRTYPE_* — same three-way split as DatapointSerializer.__add_errors. Adds 4 new unit tests covering: assertion failure, general error, passing labels excluded, and Counter JSON serialization. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Taurus-side generation of a generic
failed_transactions.jsonartifact (EFT — Exclude Failed Transactions) for browser-based (Selenium/Apiritif) runs. Each failed transaction is categorized into general errors vs. failed assertions vs. failed embedded resources, reusing the existingERRTYPE_*classification, and written toengine.artifacts_dirso it uploads via the standardartifacts.zippath. The feature is opt-in via a new config flagmodules.blazemeter.generate-failed-transactions(default off — zero behaviour change when unset).JTLErrorsReader/bzt/modules/jmeter.pyare intentionally left unchanged; the classification logic is extracted, not modified. Gating stays server-side (a.blazemeter.com) — Taurus generates the artifact unconditionally when the flag is set.Tasks completed (10)
bzt/modules/eft.pypublic surface (4e4737f3)build_failed_transactionsthree-way split +reportInfo(4e4737f3)classify_failureextracting ERRTYPE_* logic fromJTLErrorsReader.find_failure(4e4737f3)recover_assertion_name+ syntheticassert::<label>fallback (4e4737f3)responseBodies, nullrctolerated (4e4737f3)BlazeMeterUploader.post_process()(f252ad34)f252ad34)site/dat/docs/BlazemeterReporter.md(0e79cea1)Stage validation
Regression validation
Implementer note: full nose2 -s tests/unit suite compared before/after — 91 pre-existing environmental failures, 0 regressions introduced, all 8 target EFT tests red→green (9/9 including S9 flag-off guard).
Links
AI-SDLC run: mob-43135-20260825T082749Z