Skip to content

StreamingTable does not project physical sort expressions #19717

@timsaucer

Description

@timsaucer

Describe the bug

When performing a scan using a StreamingTable that contains sort expressions, if the projection does not include one of the sort columns it leads to an error.

PhysicalExpr Column references column 'b' at index 1 (zero-based) but input schema only has 1 columns: [\"a\"]

To Reproduce


#[tokio::test]
async fn test_sort_with_streaming_table() -> Result<()> {
    let batch = record_batch!(
            ("a", Int32, [1, 2, 3]),
            ("b", Int32, [1, 2, 3])
    )?;

    let ctx = SessionContext::new();

    let sort_order = vec![
        SortExpr::new(
            Expr::Column(datafusion_common::Column::new(
                Option::<TableReference>::None,
                "a",
            )),
            true,
            false,
        ),
        SortExpr::new(
            Expr::Column(datafusion_common::Column::new(
                Option::<TableReference>::None,
                "b",
            )),
            true,
            false,
        ),
    ];
    let schema = batch.schema();
    let batches = Arc::new(DummyStreamPartition {
        schema: schema.clone(),
        batches: vec![batch],
    }) as _;
    let provider = StreamingTable::try_new(schema.clone(), vec![batches])?
        .with_sort_order(sort_order);
    ctx.register_table("test_table", Arc::new(provider))?;

    let sql = "SELECT a FROM test_table group by a";
    ctx.sql(sql).await?.show().await?;

    Ok(())
}

Expected behavior

The above test should pass.

Additional context

I think this can probably be resolved in StreamingTable::scan. We can project the schema before calling create_physical_sort_exprs

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions