[VL] Support format_number function - #12754
Conversation
|
Run Gluten Clickhouse CI on x86 |
There was a problem hiding this comment.
Pull request overview
This PR wires Spark SQL format_number(number, decimalPlaces) through Gluten’s Substrait expression mapping so Velox can execute it natively where supported, and adds coverage plus documentation/restriction metadata.
Changes:
- Add
format_numbertoExpressionNamesand registerFormatNumberinExpressionMappings. - Document Velox support as partial (numeric primitives only; DecimalType excluded) and add a corresponding restriction entry for doc generation.
- Add Velox backend validation tests for
format_number, including expected Spark-side execution for DecimalType input.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| shims/common/src/main/scala/org/apache/gluten/expression/ExpressionNames.scala | Adds the FORMAT_NUMBER function name constant used across shims/mappings. |
| gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionMappings.scala | Maps Spark’s FormatNumber expression to Substrait function name format_number. |
| docs/velox-backend-scalar-function-support.md | Updates function support table to reflect partial support + DecimalType restriction note. |
| backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala | Adds UTs covering offload for supported numeric types and non-offload for DecimalType input. |
| backends-velox/src/main/scala/org/apache/gluten/expression/ExpressionRestrictions.scala | Adds a FormatNumberRestrictions entry for doc/restriction reporting. |
| backends-clickhouse/src/main/scala/org/apache/gluten/utils/CHExpressionUtil.scala | Blacklists format_number for ClickHouse backend to avoid incorrect pushdown. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Run Gluten Clickhouse CI on x86 |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (1)
backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala:768
- The added test comments say this block exercises HALF_EVEN rounding and thousands separators, but the current inputs from TPCH columns are not deterministic for those edge cases (e.g., they may not hit tie-to-even cases, and values may not exceed 999 to guarantee separators). Adding deterministic cases (using
datatab.double_field1which already includes 1.025/1.035/1.045, and a scaled integer) will make the intent of the test reliably true.
// Floating-point input, exercising HALF_EVEN rounding and thousands separators.
runQueryAndCompare(
"SELECT format_number(cast(l_quantity as double), 1) FROM lineitem limit 50") {
checkGlutenPlan[ProjectExecTransformer]
}
20b7e3d to
ba2e2b4
Compare
|
Run Gluten Clickhouse CI on x86 |
What changes are proposed in this pull request?
Velox already registers the sparksql
format_number(number, decimalPlaces)function (thousands separators, fixed decimal digits, HALF_EVEN rounding), so wire it up on the Gluten side.Unlike CAST(number AS VARCHAR), this adds thousands separators and fixed decimal places. Supports tinyint, smallint, integer, bigint, float, and double.
How was this patch tested?
New UTs.
Was this patch authored or co-authored using generative AI tooling?
Yes, Assisted-by: Claude:claude-opus-4-8