[DNM][VL] Take Arrow out of the bundle for Spark 4.x and drop dead arrow-dataset - #12737
Draft
jackylee-ch wants to merge 1 commit into
Draft
[DNM][VL] Take Arrow out of the bundle for Spark 4.x and drop dead arrow-dataset#12737jackylee-ch wants to merge 1 commit into
jackylee-ch wants to merge 1 commit into
Conversation
jackylee-ch
force-pushed
the
arrow-unbundle-dnm
branch
from
August 10, 2026 07:57
2ac5922 to
7aeb578
Compare
jackylee-ch
changed the title
|
Run Gluten Clickhouse CI on x86 |
…row-dataset
Test-only, two independent parts.
1) Spark 4.x uses the Arrow that Spark itself ships, instead of gluten
bundling its own copy. Two new properties drive it:
arrow.deps.scope compile -> provided (spark-4.0 / 4.1)
spark.arrow.exclusion.groupId org.apache.arrow -> none
The second one parameterizes the `<exclusion>` groupId on the Spark
dependencies in dependencyManagement, so those exclusions stop matching and
Spark's own Arrow flows through transitively - including into the gluten-ut
test classpath, which is what `provided` alone cannot do.
arrow.version now matches what each Spark actually ships:
spark-4.0 -> 18.1.0, spark-4.1 -> 18.3.0.
Spark 3.3 / 3.4 / 3.5 keep 15.0.0 at `compile` scope, unchanged.
package/pom.xml needs no change: under `provided` the Arrow artifacts never
enter the shade artifactSet, so the org.apache.arrow relocation becomes a
no-op on Spark 4.x while still applying on 3.x.
2) Drops arrow-dataset. `ArrowNativeMemoryPool` and `ArrowReservationListener`
were its only main-source users and have had no callers since apache#12130 removed
the Arrow-CSV scan path. The three tests under backends-velox .../fs/ that
used it exercise Arrow's own FileSystemDatasetFactory, not gluten code.
Bundle size:
spark-4.1 138.6 MB -> 56.4 MB compressed (600.6 -> 299.6 uncompressed)
spark-3.5 138.6 MB -> 68.6 MB compressed
Arrow left in the spark-4.1 bundle: arrow-c-data only, 35 classes plus 0.9 MB
of libarrow_cdata_jni - Spark does not ship arrow-c-data and its JNI symbols
bind to the original package names, so it cannot be provided or relocated.
Generated-by: Claude claude-opus-4.7
jackylee-ch
force-pushed
the
arrow-unbundle-dnm
branch
from
August 10, 2026 08:44
7aeb578 to
1393c1c
Compare
|
Run Gluten Clickhouse CI on x86 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test-only, two independent parts.
1) Spark 4.x uses the Arrow that Spark itself ships. Two new properties drive it:
arrow.deps.scopecompileprovidedspark.arrow.exclusion.groupIdorg.apache.arrownoneThe second parameterizes the
<exclusion>groupId on the Spark dependencies independencyManagement, so those exclusions stop matching and Spark's own Arrow flows through transitively — including into thegluten-uttest classpath, whichprovidedalone cannot reach.arrow.versionnow matches what each Spark actually ships: spark-4.0 → 18.1.0, spark-4.1 → 18.3.0. Spark 3.3/3.4/3.5 keep 15.0.0 atcompile, unchanged.package/pom.xmlneeds no change: underprovidedthe Arrow artifacts never enter the shade artifactSet, so theorg.apache.arrowrelocation is a no-op on 4.x while still applying on 3.x.2) Drops arrow-dataset.
ArrowNativeMemoryPool/ArrowReservationListenerwere its only main-source users and have had no callers since #12130 removed the Arrow-CSV scan path. The three tests underbackends-velox/.../fs/that used it exercise Arrow's ownFileSystemDatasetFactory, not gluten code.Arrow left in the spark-4.1 bundle:
arrow-c-dataonly — 35 classes + 0.9 MB oflibarrow_cdata_jni. Spark does not shiparrow-c-data, and its JNI symbols bind to the original package names, so it can be neitherprovidednor relocated.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-4.7