Skip to content

Quarantine or fix test_utah hang and other known-broken tests on master #3478

@MaxGhenis

Description

@MaxGhenis

Summary

Several test failures are known-broken on main across repos but are neither fixed nor quarantined. Every PR body ends up noting "these failures predate this PR" instead of the suite being green. This makes real regressions harder to spot.

What goes wrong

policyengine-api

  • tests/to_refactor/python/test_us_policy_macro.py::test_utah hangs locally; module references policyengine_api/worker.py that no longer exists. Confirmed on master by multiple agents.

policyengine-api-v2 (simulation project)

  • tests/test_household_calculation.py — macOS-only Blosc2 import failures.
  • tests/test_agent.pyAskRequest ImportError.

policyengine-app-v2

  • 40 failures in app/src/tests/unit/libs/migration/* from localStorage.clear is not a function.

Suggested fix

Per repo, either:

  1. Fix the underlying issue (preferred), or
  2. Quarantine with an explicit marker so the default suite is green:
# pytest
@pytest.mark.skip(reason="Broken on main — tracked in #NNNN, fix pending")
def test_utah(): ...

# vitest
it.skip('migration test — localStorage.clear not a function, tracked in #NNNN', ...)
  1. Surface via a separate CI job (pytest -m broken) that's allowed to fail, so the status is visible without masking real regressions.

The anti-pattern to eliminate: PR authors noting "these failures predate this PR" in every review. If they predate every PR, they should be marked so CI reflects that.

Severity

Medium (dev-experience). Hides real regressions, creates review friction, breaks the "CI green = merge" invariant.

Scope

Filing this in three repos: policyengine-api, policyengine-api-v2, policyengine-app-v2.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions