Skip to content

Bug: Fix start of week CI#558

Merged
mtcarlone merged 2 commits into
mainfrom
bug/ci-rd2-fix
Jun 2, 2026
Merged

Bug: Fix start of week CI#558
mtcarlone merged 2 commits into
mainfrom
bug/ci-rd2-fix

Conversation

@mtcarlone

Copy link
Copy Markdown
Contributor

Fix Monday regression: schema collision bug + drop EOL dbt versions from CI

The first weekly Tier 3 regression against main failed broadly. Triage of all 43 matrix jobs found two independent root causes — one real bug, one environmental. This PR fixes both.


Root cause 1 — DBT_VERSION schema collision (real bug)

This is why supported versions (BigQuery 1.9, 1.10) failed.

scripts/ci/test.sh read the positional dbt_version arg into a local variable used only to pick the tox env name — but never assigned it to the DBT_VERSION env var that profiles.yml interpolates into the schema/dataset name. _lib.sh's ensure_dbt_version() then defaulted it to empty.

Result: every matrix job within a warehouse wrote to the same schema (dbt_artifacts_test_commit__<sha> — note the empty version segment). Under max-parallel: 8, parallel jobs on the shared cloud warehouses collided:

409 Already Exists: Table ...microbatch_seed; reason: duplicate
429 Exceeded rate limits: too many table update operations for this table
Not found: Table ...microbatch_seed was not found in location US

BigQuery failed hardest (strict table semantics + rate limits); Snowflake survived this run on luck but was equally exposed. Container-isolated warehouses (postgres/trino/sqlserver) were unaffected — they use fixed schemas. Invisible locally because we only ever ran one job at a time.

Fix: test.sh now does export DBT_VERSION="${dbt_version}"; removed the misleading ensure_dbt_version() helper. Each pinned version gets a distinct schema, so parallel jobs no longer collide.

Root cause 2 — EOL versions can't run on the runner

Every failure on 1.3–1.8 is environmental, not a package bug:

  • 1.3–1.7 import distutils (removed in Python 3.12) → die at dbt clean.
  • 1.3–1.8 predate the microbatch incremental strategy (a dbt 1.9 feature) that the test fixture uses.

Keeping them green would require a legacy Python 3.11 lane plus per-version gates — tech debt for versions dbt Labs has already EOL'd.

Decision: drop dbt 1.3–1.8 from the CI matrix. Matrix trimmed 42 → 15 slots (1.9 / 1.10 / 1.11 / latest).


⚠ This is a CI-coverage change, NOT a support drop

  • require-dbt-version is unchanged (>=1.3.0) — older-dbt consumers can still install the package; it's simply no longer verified each release.
  • tox.ini envs for 1.3–1.8 are kept — a maintainer can still run tox -e integration_snowflake_1_5_0 on a local Python 3.11 to debug a consumer report.
  • The formal support-floor bump (require-dbt-version>=1.9, removing the tox envs, public announcement) remains a separate 3.0.0 decision. This is Phase 1 (CI descope) brought forward, not a pre-emption of that.
  • A consumer-facing note was added to README.md ("Supported dbt versions").

Files changed

File Change
scripts/ci/test.sh Export DBT_VERSION from the version arg (the fix)
scripts/ci/_lib.sh Remove misleading ensure_dbt_version() helper
.github/workflows/release.yml Matrix 42 → 15 slots; explanatory comment
README.md "Supported dbt versions" note
specs/ci-rework/README.md §12.3.0 records both root causes + decision

mtcarlone added 2 commits June 1, 2026 17:04
test.sh read the positional dbt_version arg into a local variable used
only to pick the tox env name, but never assigned it to the DBT_VERSION
env var that profiles.yml interpolates into the schema/dataset name.
Every matrix job within a warehouse therefore wrote to the same schema
(dbt_artifacts_test_commit__<sha>), and parallel cloud-warehouse jobs
collided — BigQuery surfaced 409 Already Exists / 429 rate limits /
"table not found" on supported versions 1.9 and 1.10.

Now exports DBT_VERSION from the arg; removed the misleading
ensure_dbt_version() default from _lib.sh. Invisible locally because we
only ever ran one job at a time.
They can't run on the runner's Python 3.12 (distutils, removed in 3.12)
and predate the microbatch strategy the fixture uses. Matrix trimmed
42 -> 15 slots (1.9/1.10/1.11/latest).

CI-coverage change only: require-dbt-version stays >=1.3.0 (still
installs), tox envs for old versions kept for local debugging. Formal
support-floor bump remains a separate 3.0.0 decision. README note added;
reasoning recorded in specs/ci-rework/README.md §12.3.0.
@mtcarlone mtcarlone merged commit dc8dabb into main Jun 2, 2026
3 checks passed
@mtcarlone mtcarlone deleted the bug/ci-rd2-fix branch June 2, 2026 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant