Skip to content

fix: use the current schema when reading matched rows in upsert - #3850

Open
JooyoungJeong wants to merge 1 commit into
apache:mainfrom
JooyoungJeong:fix-upsert-schema-evolution
Open

fix: use the current schema when reading matched rows in upsert#3850
JooyoungJeong wants to merge 1 commit into
apache:mainfrom
JooyoungJeong:fix-upsert-schema-evolution

Conversation

@JooyoungJeong

Copy link
Copy Markdown

Closes #2467

Rationale for this change

Transaction.upsert() pins the target branch with use_ref(branch), which sets a snapshot id, so TableScan.projection() resolves the schema recorded on that snapshot rather than the table's current schema. A schema-only update creates no data snapshot, so the branch tip can still carry the pre-update schema, and the matched rows are then read with a schema that no longer matches the input dataframe.

An added column surfaces as ValueError: Target schema's field names are not matching the table's field names in get_rows_to_update(). A renamed non-key column fails the same way, and a renamed join column fails earlier still, in has_duplicate_rows(), with KeyError.

The change is one call site in pyiceberg/table/__init__.py: the scan that reads the matched rows now projects the current table schema. File planning is untouched and still uses the pinned snapshot, so time-travel and ref scan semantics do not change. ArrowScan already binds the row filter against table_metadata.schema(), so the projection was the only part still tied to the snapshot's historical schema.

Are these changes tested?

Yes. Four cases added to tests/table/test_upsert.py, each of which fails on main:

  • test_upsert_after_adding_column
  • test_upsert_after_renaming_column
  • test_upsert_after_renaming_join_column
  • test_upsert_after_adding_column_in_transaction

uv run pytest tests/table/test_upsert.py -q → 27 passed.

Are there any user-facing changes?

No.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upsert fails after adding in a new column, target field not found

1 participant