-
Notifications
You must be signed in to change notification settings - Fork 1.9k
feat(spark): add unix date and timestamp functions #19892
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
| /// Returns the number of microseconds since epoch (1970-01-01 00:00:00 UTC) for the given timestamp. | ||
| /// <https://spark.apache.org/docs/latest/api/sql/index.html#unix_micros> | ||
| #[derive(Debug, PartialEq, Eq, Hash)] | ||
| pub struct SparkUnixMicros { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can consider making these (micros, millis, seconds) a unified struct with an inner enum field to differentiate the behaviour, like so:
datafusion/datafusion/spark/src/function/bitwise/bit_shift.rs
Lines 189 to 201 in e5e7636
| #[derive(Debug, Hash, Copy, Clone, Eq, PartialEq)] | |
| enum BitShiftType { | |
| Left, | |
| Right, | |
| RightUnsigned, | |
| } | |
| #[derive(Debug, Hash, Eq, PartialEq)] | |
| pub struct SparkBitShift { | |
| signature: Signature, | |
| name: &'static str, | |
| bit_shift_type: BitShiftType, | |
| } |
We'd then have different constructors:
datafusion/datafusion/spark/src/function/bitwise/bit_shift.rs
Lines 253 to 263 in e5e7636
| pub fn left() -> Self { | |
| Self::new("shiftleft", BitShiftType::Left) | |
| } | |
| pub fn right() -> Self { | |
| Self::new("shiftright", BitShiftType::Right) | |
| } | |
| pub fn right_unsigned() -> Self { | |
| Self::new("shiftrightunsigned", BitShiftType::RightUnsigned) | |
| } |
Which can be used like so:
datafusion/datafusion/spark/src/function/bitwise/mod.rs
Lines 27 to 41 in e5e7636
| make_udf_function!( | |
| bit_shift::SparkBitShift, | |
| shiftleft, | |
| bit_shift::SparkBitShift::left | |
| ); | |
| make_udf_function!( | |
| bit_shift::SparkBitShift, | |
| shiftright, | |
| bit_shift::SparkBitShift::right | |
| ); | |
| make_udf_function!( | |
| bit_shift::SparkBitShift, | |
| shiftrightunsigned, | |
| bit_shift::SparkBitShift::right_unsigned | |
| ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sounds good, updated
Which issue does this PR close?
unix_date/micros/millis/secondsfunctions #19891datafusion-sparkSpark Compatible Functions #15914Rationale for this change
Add spark unix date and timestamp functions.
What changes are included in this PR?
New spark functions:
Are these changes tested?
yes in slt
Are there any user-facing changes?
yes