fix(cubesql): Support DATE_ADD SQL pushdown - #11539
Conversation
|
Claude finished @MazterQyou's task in 3m 33s —— View job Review: templates look right for
|
| Dialect | Native signature | Template | Verdict |
|---|---|---|---|
| Postgres / DuckDB | ts + interval |
({{ args[0] }} + '{{ interval }} {{ date_part }}'::interval) |
✅ |
| MySQL | DATE_ADD(date, INTERVAL n unit) |
matches | ✅ order, |
| MSSQL / Snowflake / Redshift | DATEADD(part, n, date) |
matches | ✅ |
| Presto/Trino | date_add(unit, value, ts) |
matches (unit strings are case-insensitive) | ✅ |
| ClickHouse | addDate(date, INTERVAL n unit) |
matches | ✅, |
| Databricks | ts + INTERVAL '{n}' unit |
matches ANSI form |
The Postgres change also correctly un-comments as DATE_ADD with args[0] (the old commented-out DATEADD variant used args[2], which is the right index for the 3-arg dateadd/datediff shape but wrong here). interval is always an integer stringified from a literal and date_part for date_add comes from a closed set, so there's no injection surface introduced.
Findings
Medium
- MySQL cannot express
MILLISECOND(MysqlQuery.ts:191). Sub-day intervals are emitted asMILLISECOND, soINTERVAL '2 hours'rendersINTERVAL 7200000 MILLISECOND→ MySQL syntax error, replacing today's silent fallback to post-processing with a hard query failure. Suggested unit mapping inline. - Databricks quoted ANSI interval rejects
MILLISECOND/NANOSECOND(DatabricksQuery.ts:179).INTERVAL '{{ interval }}' {{ date_part }}is the typed form, limited toYEAR…SECOND; the unquoted multi-unit form is broader. Suggestion inline.
Low
3. NANOSECOND is unrepresentable in Postgres, DuckDB and Trino (PostgresQuery.ts:91). If the IntervalMonthDayNano-with-nanos branch is reachable, normalizing nanos → micros/seconds on the Rust side fixes all dialects at once instead of nine independent guards. Also note this template is inherited by CrateQuery, FireboltQuery and QuestQuery.
4. ClickHouse addDate requires 23.9+ (ClickHouseQuery.ts:269); the + INTERVAL operator form is version-agnostic.
5. The new Rust test doesn't guard this PR (test_wrapper.rs:2891-2897). functions/DATE_ADD is already defined in the test transport on master (compile/test/mod.rs:699) as DATE_ADD({{ args_concat }}), so both assertions pass on origin/master unchanged — and because the fixture is args_concat-based, it never exercises date_part/interval at all. An arg-order or variable-name typo in any of the nine templates would still pass. compile/mod.rs:15371 shows the existing pattern for overriding the template with a realistic dialect body.
Notes
- No docs update needed as far as I can tell —
DATEADDpushdown isn't enumerated per-dialect in/docs-mintlify. - I did not run the Rust test suite or
yarn test(cubesql build cost); the review is static. - No prior review threads existed on this PR, so nothing to resolve or dedupe against.
| templates.functions.UTCTIMESTAMP = '(NOW() AT TIME ZONE \'UTC\')'; | ||
| // DATEADD is being rewritten to DATE_ADD | ||
| // templates.functions.DATEADD = '({{ args[2] }} + \'{{ interval }} {{ date_part }}\'::interval)'; | ||
| templates.functions.DATE_ADD = '({{ args[0] }} + \'{{ interval }} {{ date_part }}\'::interval)'; |
There was a problem hiding this comment.
NANOSECOND is a unit the generator can emit (the IntervalMonthDayNano branch in wrapper.rs:3168-3181), and Postgres has no nanosecond interval unit — '7200000000000 NANOSECOND'::interval errors out. Same for DuckDB (microsecond granularity) and Trino.
Worth checking whether that branch is reachable for the interval literals DataFusion actually produces here; if it is, converting nanos to microseconds/seconds on the Rust side (before it reaches any dialect template) would fix all nine dialects at once rather than each template guarding independently. Note this also inherits into CrateQuery/FireboltQuery/QuestQuery, which extend PostgresQuery.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #11539 +/- ##
===========================================
+ Coverage 59.35% 79.62% +20.26%
===========================================
Files 223 480 +257
Lines 17989 99700 +81711
Branches 3641 3641
===========================================
+ Hits 10677 79382 +68705
- Misses 6793 19799 +13006
Partials 519 519
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Signed-off-by: Alex Qyoun-ae <4062971+MazterQyou@users.noreply.github.com>
4c56809 to
f01a2b1
Compare
|
Claude finished @MazterQyou's task in 2m 55s —— View job Re-review after force-push: all four actionable findings addressed; one new finding on negative sub-day intervals (1 medium, 1 low carried over)Full review
What changed since the last review
Each fix also carries a comment explaining why the non-obvious form was chosen, which is the right call — those templates are otherwise unreadable without the Rust context. I resolved the four stale threads; the The new test is the substantive improvement: it's table-driven over the five distinct template shapes × the three units the rewrite can emit ( New finding (medium) — negative sub-day intervalsPosted inline on
Not introduced by this PR — the same literal is what DataFusion evaluates in the post-processing path today — but this test is the natural place to catch it, and negative offsets are the common real-world shape. Suggested construction-site fix is in the inline comment. Remaining low-severity notes
Notes
|
| ]; | ||
|
|
||
| for (template, expected) in dialects { | ||
| for (unit, expected) in ["hour", "week", "year"].iter().zip(expected) { |
There was a problem hiding this comment.
This matrix is a big improvement over the args_concat fixture — it genuinely pins arg order and both context variables per dialect. One gap: every case uses a positive interval (2), which is exactly where the sub-day encoding happens to work.
DATEADD('hour', -2, ...) goes through redshift-dateadd-to-interval → transform_interval_parts_to_interval (rust/cubesql/cubesql/src/compile/rewrite/rules/dates.rs:539-541), which builds the literal as IntervalDayTime(3_600_000 * n). IntervalDayTime is a packed (days: i32, millis: i32) pair, not a scalar, so a plain multiply only round-trips for non-negative values that fit in 31 bits:
n = -2→-7_200_000i64=0xFFFF_FFFF_FF91_7A00→days = -1,ms = -7_200_000.wrapper.rs:3151then hits the mixed branch and returnsUnsupported mixed IntervalDayTime: days = -1, ms = -7200000.n = 700(hours) →2_520_000_000>i32::MAX→ms as i32wraps to-1_774_967_296, so the pushed-down SQL silently gets a negative interval of the wrong magnitude.
Neither is introduced by this PR — the same literal is what DataFusion evaluates in the post-processing path today, so the value is equally wrong without pushdown — but this test is the natural place to catch it, and negative offsets (DATEADD('hour', -1, ...), DATEADD('minute', -30, ...)) are the common shape in real queries. Adding -2 alongside 2 to the unit loop would fail today.
The underlying fix is a one-liner at the construction site — build the two halves separately instead of multiplying:
Some("hour") => {
let ms = i32::try_from(3_600_000i64 * i64::from(interval_int)).ok()?; // bail if it overflows
ScalarValue::IntervalDayTime(Some(i64::from(ms) & 0xFFFF_FFFF))
}(and the same shape for millisecond / second / minute; day / week are exact multiples of 2^32 so they already sign-extend correctly). Happy to be wrong here if there's a normalization step I've missed — but if it holds, it's worth either fixing in this PR or filing separately, since this PR is what makes the value visible in generated SQL.
Check List
Description of Changes Made
This PR adds a
functions/DATE_ADDSQL template to the Postgres, Snowflake, Redshift, MSSQL, MySQL, ClickHouse, Presto/Trino, Databricks, and DuckDB dialects so that expressions usingDATEADDcan be pushed down instead of falling back to post-processing. Related test is included.