feat: Add support for explode_outer function for arrays

[athlcode]

Which issue does this PR close?

• Closes Add support for Spark-compatible explode_outer function for arrays #19053.

Rationale for this change

DataFusion's 'unnest' had no way to express Spark 'explode_outer' semantics, empty input lists were silently dropped, even with 'preserve_nulls' = true.

What changes are included in this PR?

Adds a third unnest behavior that produces a NULL row for empty input lists, by replacing UnnestOptions.preserve_nulls: bool with a NullHandling { Drop, Preserve, PreserveAndExpandEmpty } enum. The with_preserve_nulls(bool) builder is kept as a backward-compat shim.

Are these changes tested?

Yes, new unit test for the empty-list case, extended longest-length and DataFrame unnest_column_nulls tests, and existing proto round-trip coverage.

Are there any user-facing changes?

The preserve_nulls field on UnnestOptions is renamed to null_handling. The with_preserve_nulls(bool) builder is preserved, so most callers are unaffected. Add the api change label for the field rename.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v54.1.0 (current)
       Built [ 107.213s] (current)
     Parsing datafusion v54.1.0 (current)
      Parsed [   0.036s] (current)
    Building datafusion v54.1.0 (baseline)
       Built [ 107.156s] (baseline)
     Parsing datafusion v54.1.0 (baseline)
      Parsed [   0.035s] (baseline)
    Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.636s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 217.447s] datafusion
    Building datafusion-common v54.1.0 (current)
       Built [  35.269s] (current)
     Parsing datafusion-common v54.1.0 (current)
      Parsed [   0.064s] (current)
    Building datafusion-common v54.1.0 (baseline)
       Built [  36.088s] (baseline)
     Parsing datafusion-common v54.1.0 (baseline)
      Parsed [   0.065s] (baseline)
    Checking datafusion-common v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.739s] 223 checks: 221 pass, 2 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field UnnestOptions.null_handling in /home/runner/work/datafusion/datafusion/datafusion/common/src/unnest.rs:104

--- failure struct_pub_field_missing: pub struct's pub field removed or renamed ---

Description:
A publicly-visible struct has at least one public field that is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/struct_pub_field_missing.ron

Failed in:
  field preserve_nulls of struct UnnestOptions, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/a2a52463feed322b7cebb716ed4aac2d9e64ccba/datafusion/common/src/unnest.rs:72

     Summary semver requires new major version: 2 major and 0 minor checks failed
    Finished [  73.346s] datafusion-common
    Building datafusion-expr v54.1.0 (current)
       Built [  28.071s] (current)
     Parsing datafusion-expr v54.1.0 (current)
      Parsed [   0.078s] (current)
    Building datafusion-expr v54.1.0 (baseline)
       Built [  28.583s] (baseline)
     Parsing datafusion-expr v54.1.0 (baseline)
      Parsed [   0.079s] (baseline)
    Checking datafusion-expr v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.263s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field Unnest.outer in /home/runner/work/datafusion/datafusion/datafusion/expr/src/expr.rs:684

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  59.124s] datafusion-expr
    Building datafusion-physical-plan v54.1.0 (current)
       Built [  38.597s] (current)
     Parsing datafusion-physical-plan v54.1.0 (current)
      Parsed [   0.148s] (current)
    Building datafusion-physical-plan v54.1.0 (baseline)
       Built [  38.119s] (baseline)
     Parsing datafusion-physical-plan v54.1.0 (baseline)
      Parsed [   0.150s] (baseline)
    Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.633s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  78.975s] datafusion-physical-plan
    Building datafusion-proto v54.1.0 (current)
       Built [  61.643s] (current)
     Parsing datafusion-proto v54.1.0 (current)
      Parsed [   0.019s] (current)
    Building datafusion-proto v54.1.0 (baseline)
       Built [  63.068s] (baseline)
     Parsing datafusion-proto v54.1.0 (baseline)
      Parsed [   0.020s] (baseline)
    Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.259s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 126.440s] datafusion-proto
    Building datafusion-proto-models v54.1.0 (current)
       Built [  25.848s] (current)
     Parsing datafusion-proto-models v54.1.0 (current)
      Parsed [   0.136s] (current)
    Building datafusion-proto-models v54.1.0 (baseline)
       Built [  25.367s] (baseline)
     Parsing datafusion-proto-models v54.1.0 (baseline)
      Parsed [   0.135s] (baseline)
    Checking datafusion-proto-models v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.562s] 223 checks: 221 pass, 2 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field UnnestOptions.null_handling in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:674
  field UnnestOptions.null_handling in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:674
  field Unnest.outer in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1015
  field Unnest.outer in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1015

--- failure struct_pub_field_missing: pub struct's pub field removed or renamed ---

Description:
A publicly-visible struct has at least one public field that is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/struct_pub_field_missing.ron

Failed in:
  field preserve_nulls of struct UnnestOptions, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/a2a52463feed322b7cebb716ed4aac2d9e64ccba/datafusion/proto-models/src/generated/prost.rs:674
  field preserve_nulls of struct UnnestOptions, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/a2a52463feed322b7cebb716ed4aac2d9e64ccba/datafusion/proto-models/src/generated/prost.rs:674

     Summary semver requires new major version: 2 major and 0 minor checks failed
    Finished [  55.016s] datafusion-proto-models
    Building datafusion-sql v54.1.0 (current)
       Built [  43.035s] (current)
     Parsing datafusion-sql v54.1.0 (current)
      Parsed [   0.032s] (current)
    Building datafusion-sql v54.1.0 (baseline)
       Built [  44.130s] (baseline)
     Parsing datafusion-sql v54.1.0 (baseline)
      Parsed [   0.032s] (baseline)
    Checking datafusion-sql v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.221s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  89.173s] datafusion-sql
    Building datafusion-sqllogictest v54.1.0 (current)
       Built [ 186.554s] (current)
     Parsing datafusion-sqllogictest v54.1.0 (current)
      Parsed [   0.025s] (current)
    Building datafusion-sqllogictest v54.1.0 (baseline)
       Built [ 187.149s] (baseline)
     Parsing datafusion-sqllogictest v54.1.0 (baseline)
      Parsed [   0.022s] (baseline)
    Checking datafusion-sqllogictest v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.101s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 376.389s] datafusion-sqllogictest

[comphead]

Thanks @athlcode explode_outer is not like other builtin function so it has to be in the core.

Please add tests, more tests the better, including mix with nulls, nested combinations with explode and explode_outer and supported datatypes

[athlcode]

Thanks @athlcode explode_outer is not like other builtin function so it has to be in the core.

Please add tests, more tests the better, including mix with nulls, nested combinations with explode and explode_outer and supported datatypes

thank you, added more test cases

[comphead]

Thanks @athlcode my bad, I just realized I haven't clarified tests needs to be added to slt file, lets remove rust tests unless they are not reproducible with slt. Ideally to create a separate .slt file for explode_outer to localize tests

[athlcode]

Thanks @athlcode my bad, I just realized I haven't clarified tests needs to be added to slt file, lets remove rust tests unless they are not reproducible with slt. Ideally to create a separate .slt file for explode_outer to localize tests

Thanks @comphead for the clarification. None of the test cases are currently reproducible via SLT because there's no SQL surface for NullHandling::PreserveAndExpandEmpty. To enable SLT coverage we need to add SQL recognition of explode_outer(col) (and probably explode(col) for symmetry) and map them to UNNEST with the right NullHandling variant.

[comphead]

we would still need to have .slt tests in datafusion-spark crate to test explode_outer works properly with SQL dialect

[athlcode]

we would still need to have .slt tests in datafusion-spark crate to test explode_outer works properly with SQL dialect

Added slt tests at datafusion/sqllogictest/test_files/spark/generator/explode_outer.slt, exercises explode_outer via the Spark SQL dialect path

[kazuyukitanimura]

Thanks @athlcode would you mind resolving the conflicts?

@comphead any further comments?

[athlcode]

Thanks @athlcode would you mind resolving the conflicts?

@comphead any further comments?

resolved the conflicts, thank you

[comphead]

I think we should not be adding spark into the core, it would inverse the dependencies

[comphead]

Tests are fine, thanks @athlcode
Please help me to understand why we need to keep anything spark related in the core?

[comphead]

we might also update the SELECT doc like in #22672

[athlcode]

Tests are fine, thanks @athlcode Please help me to understand why we need to keep anything spark related in the core?

The mechanism has to live in core, but it isn't Spark-specific. NullHandling::PreserveAndExpandEmpty is just a new option on UnnestOptions in datafusion-common, where UnnestOptions already lives, and UnnestExec in datafusion-physical-plan is the only place the per-row logic can run. Expr::Unnest.outer is a new field on a core Expr variant, and downstream crates can't add fields to a pub enum variant in Rust, so the flag has to live where the variant is defined.

Neither references Spark by name. They're the generic mechanism for "unnest that preserves empty arrays as NULL rows," which Hive EXPLODE OUTER, Snowflake FLATTEN(OUTER => true), and BigQuery UNNEST ... WITH OFFSET all need too.

The Spark-flavored surface (the explode / explode_outer SQL aliases and the allow_multiple_generators config) is the only part that's truly dialect-specific. Happy to move those into datafusion-spark as a planner extension and a custom config registered via Extensions if you'd prefer the core diff to be purely the mechanism.

[comphead]

Tests are fine, thanks @athlcode Please help me to understand why we need to keep anything spark related in the core?

The mechanism has to live in core, but it isn't Spark-specific. NullHandling::PreserveAndExpandEmpty is just a new option on UnnestOptions in datafusion-common, where UnnestOptions already lives, and UnnestExec in datafusion-physical-plan is the only place the per-row logic can run. Expr::Unnest.outer is a new field on a core Expr variant, and downstream crates can't add fields to a pub enum variant in Rust, so the flag has to live where the variant is defined.

Neither references Spark by name. They're the generic mechanism for "unnest that preserves empty arrays as NULL rows," which Hive EXPLODE OUTER, Snowflake FLATTEN(OUTER => true), and BigQuery UNNEST ... WITH OFFSET all need too.

The Spark-flavored surface (the explode / explode_outer SQL aliases and the allow_multiple_generators config) is the only part that's truly dialect-specific. Happy to move those into datafusion-spark as a planner extension and a custom config registered via Extensions if you'd prefer the core diff to be purely the mechanism.

Thanks @athlcode sorry I really missed this comments, it makes sense to me, lets remove Spark from comments in the core, as this would be confusing, and other than that, the PR would be good to go

[athlcode]

@comphead thanks, I have removed spark from the comments,

[comphead]

Thanks @athlcode.
Since the code in the core it would actually benefit the DF to have outer unnest. For example unnest can be represented as a built in function or via joins #13824

I checked some of engines for unnest_outer as a built in function and the same functionality represented mostly via joins. However if we support unnest it would prob makes sense to support unnest_outer as a builtin function.

Lets move tests from explode.slt into unnest.slt referring it as unnest_outer

[codecov-commenter]

Codecov Report

❌ Patch coverage is 77.15931% with 119 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.75%. Comparing base (f7aef23) to head (d01c70f).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/proto-models/src/generated/pbjson.rs	14.28%	58 Missing and 2 partials ⚠️
datafusion/expr/src/expr.rs	25.00%	21 Missing ⚠️
datafusion/physical-plan/src/unnest.rs	93.73%	18 Missing and 3 partials ⚠️
datafusion/proto-models/src/generated/prost.rs	0.00%	13 Missing ⚠️
datafusion/proto/src/logical_plan/from_proto.rs	80.00%	2 Missing ⚠️
datafusion/proto/src/logical_plan/to_proto.rs	85.71%	1 Missing ⚠️
datafusion/sql/src/select.rs	96.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #22100      +/-   ##
==========================================
- Coverage   80.76%   80.75%   -0.01%     
==========================================
  Files        1089     1089              
  Lines      368797   369268     +471     
  Branches   368797   369268     +471     
==========================================
+ Hits       297850   298210     +360     
- Misses      53201    53309     +108     
- Partials    17746    17749       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[athlcode]

Thanks @athlcode. Since the code in the core it would actually benefit the DF to have outer unnest. For example unnest can be represented as a built in function or via joins #13824

I checked some of engines for unnest_outer as a built in function and the same functionality represented mostly via joins. However if we support unnest it would prob makes sense to support unnest_outer as a builtin function.

Lets move tests from explode.slt into unnest.slt referring it as unnest_outer

thanks @comphead. moved tests into unnest.slt.