select distinct on map types yields ArrowError NotYetImplemented

[gz]

Describe the bug

querying a table/parquet file with the following schema and query fails:

create table t (
    m MAP<VARCHAR, REAL> NOT NULL
);

let df = ctx.sql("SELECT distinct m FROM t LIMIT 10").await?;

fails with:

Error: ArrowError(NotYetImplemented("not yet implemented: Map(Field { name: \"entries\", data_type: Struct([Field { name: \"key\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"value\", data_type: Float32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false)"), None)

it runs into the unimplemented error that comes from arrow-rs: https://github.com/apache/arrow-rs/blob/15db256bcdd739ce3233d90b3242af15ae3945fa/arrow-row/src/lib.rs#L434

To Reproduce

parquet file with the given format that fails the query attached for convenience
results_20250325_003444.parquet.zip

Expected behavior

query returns correct results

Additional context

No response

[LSUDOKO]

take

[Jefffrey]

Upstream:

• Support MapArray in RowConverter arrow-rs#7879

[Jefffrey]

Rowconverter support is available on datafusion main now; to progress this issue we should add some test coverage in our sqllogictests with different distinct/aggregation queries (testing with group by too, etc.)

e.g. in here https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/test_files/map.slt

[PG1204]

take

Hi team, I'd like to pick this up. Following up on Jefffrey's note that RowConverter support is now on main, I'll add sqllogictest coverage in map.slt for:

SELECT DISTINCT on Map columns
GROUP BY with Map columns as grouping keys
Aggregations (COUNT, etc.) over Map-typed inputs

Planning to include edge cases like empty maps, maps with null values, and duplicate keys. Will confirm scope if there are specific scenarios you'd like prioritized.