[spark] Fix MERGE INTO silently skipping WHEN NOT MATCHED BY SOURCE actions

[kerwin-zk]

Purpose

MergeIntoPaimonTable extracts the target-only conjuncts of the merge condition and uses them to prune the target table before the full outer join (targetOnlyCondition / filteredTargetPlan, MergeIntoPaimonTable.scala:63-69).

That pruning is sound for WHEN MATCHED and WHEN NOT MATCHED: a target row that fails a target-only conjunct can never satisfy the whole merge condition, so it can never be matched, and dropping it cannot change the outcome of those actions.

It is not sound for WHEN NOT MATCHED BY SOURCE. The pruned-away rows are exactly the population that clause is defined over, so their actions are silently skipped — no error, no warning, just fewer rows changed.

Repro on a partitioned table holding pt = 'p1' and pt = 'p2' rows, with a source that only contains a = 1:

MERGE INTO target t
USING source s
ON t.a = s.a AND t.pt = 'p1'
WHEN MATCHED THEN UPDATE SET t.b = s.b
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET t.c = 'stale'

Every pt = 'p2' row should be updated to stale (no source row can ever match it), but none of them is:

Expected                Actual
 [1,100,c1,p1]          [1,100,c1,p1]
 [2,20,stale,p1]        [2,20,stale,p1]
![3,30,stale,p2]        [3,30,c3,p2]
![4,40,stale,p2]        [4,40,c4,p2]

The pruning was introduced together with MERGE INTO itself in #2331 (2023-11-17), one month before WHEN NOT MATCHED BY SOURCE was added in #2517 (2023-12-22), and its safety argument was never revisited.

Note the V2 row-level paths (ReplaceData / WriteDelta) are rewritten by Spark and are not affected, so today the same statement produces different results depending on whether the table qualifies for SparkTable.supportsV2RowLevelOps. Primary key tables never qualify, so they always take the affected V1 path.

Fix: disable the pruning when the merge has any WHEN NOT MATCHED BY SOURCE action. Setting targetOnlyCondition to None covers all three places it feeds — filteredTargetPlan, findCandidateDataSplits and targetDSWithFilePathCol — so there is no path left that can drop those rows.

Tests

CI