You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "gokselk (via GitHub)" <gi...@apache.org> on 2026/01/06 09:31:24 UTC
[PR] Add support for additional numeric types in to_timestamp functions [datafusion]
gokselk opened a new pull request, #19663:
URL: https://github.com/apache/datafusion/pull/19663
## Which issue does this PR close?
Closes #19117.
## Rationale for this change
The `to_timestamp` function family lacks consistency in supported argument types. While `to_timestamp` accepts Float64 and Decimal128 types, the related functions (`to_timestamp_seconds`, `to_timestamp_millis`, `to_timestamp_micros`, `to_timestamp_nanos`) don't offer the same support. Additionally, the documentation claims support for "unsigned integer" types, but this isn't fully implemented.
## What changes are included in this PR?
Standardizes all `to_timestamp` variants to uniformly support:
- All signed integer types: Int8, Int16, Int32, Int64
- All unsigned integer types: UInt8, UInt16, UInt32, UInt64
- Float32 and Float64
- Decimal128 (for millis/micros/nanos variants)
## Are these changes tested?
Yes, added comprehensive SQL logic tests in `datafusion/sqllogictest/test_files/datetime/timestamps.slt`.
## Are there any user-facing changes?
Users can now pass additional numeric types to `to_timestamp` functions. This is a backward-compatible enhancement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "gokselk (via GitHub)" <gi...@apache.org>.
gokselk commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2667395865
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
Review Comment:
Should be resolved in 8134491.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "gokselk (via GitHub)" <gi...@apache.org>.
gokselk commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2675135612
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -380,13 +451,14 @@ impl ScalarUDFImpl for ToTimestampFunc {
let tz = self.timezone.clone();
match args[0].data_type() {
- Int32 | Int64 => args[0]
+ Int8 | Int16 | Int32 | Int64 | UInt8 | UInt16 | UInt32 | UInt64 => args[0]
.cast_to(&Timestamp(Second, None), None)?
.cast_to(&Timestamp(Nanosecond, tz), None),
Null | Timestamp(_, _) => args[0].cast_to(&Timestamp(Nanosecond, tz), None),
- Float64 => {
+ Float16 | Float32 | Float64 => {
+ let arg = args[0].cast_to(&Float64, None)?;
Review Comment:
Turns out I was wrong, f16 does implement `ArrowNativeTypeOp`. Will update to handle all float types natively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2672674199
##########
datafusion/sqllogictest/test_files/datetime/timestamps.slt:
##########
@@ -4422,6 +4422,714 @@ FROM (VALUES
1970-01-01T00:00:00.000000005Z
+##########
+## to_timestamp functions with all numeric types
+##########
+
+# Test to_timestamp with all integer types
+# Int8
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:01:40
+
+# Int16
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:16:40
+
+# Int32
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'Int32'));
+----
+1970-01-02T00:00:00
+
+# Int64
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'Int64'));
+----
+1970-01-02T00:00:00
+
+# UInt8
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:01:40
+
+# UInt16
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:16:40
+
+# UInt32
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'UInt32'));
+----
+1970-01-02T00:00:00
+
+# UInt64
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'UInt64'));
+----
+1970-01-02T00:00:00
+
+# Float16
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float16'));
+----
+1970-01-01T00:00:01.500
+
+# Float32
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float32'));
+----
+1970-01-01T00:00:01.500
+
+# Float64
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float64'));
+----
+1970-01-01T00:00:01.500
+
+# Test to_timestamp_seconds with all integer types
+# Int8
+query P
+SELECT to_timestamp_seconds(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_seconds(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:01:40
+
+# Int16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:16:40
+
+# UInt8
+query P
+SELECT to_timestamp_seconds(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:01:40
+
+# UInt16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:16:40
+
+# UInt32
+query P
+SELECT to_timestamp_seconds(arrow_cast(86400, 'UInt32'));
+----
+1970-01-02T00:00:00
+
+# UInt64
+query P
+SELECT to_timestamp_seconds(arrow_cast(86400, 'UInt64'));
+----
+1970-01-02T00:00:00
+
+# Float16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float16'));
+----
+1970-01-01T00:00:01
+
+# Float32
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float32'));
+----
+1970-01-01T00:00:01
+
+# Float64
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float64'));
+----
+1970-01-01T00:00:01
+
+# Test to_timestamp_millis with all integer types
Review Comment:
Thanks, I think it applies to all the to_timestamp_xyx functions though from what I recall.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "gokselk (via GitHub)" <gi...@apache.org>.
gokselk commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2667396533
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -380,13 +380,14 @@ impl ScalarUDFImpl for ToTimestampFunc {
let tz = self.timezone.clone();
match args[0].data_type() {
- Int32 | Int64 => args[0]
+ Int8 | Int16 | Int32 | Int64 | UInt8 | UInt16 | UInt32 | UInt64 => args[0]
.cast_to(&Timestamp(Second, None), None)?
.cast_to(&Timestamp(Nanosecond, tz), None),
Null | Timestamp(_, _) => args[0].cast_to(&Timestamp(Nanosecond, tz), None),
- Float64 => {
+ Float32 | Float64 => {
Review Comment:
Resolved in cdf7f12.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2674652479
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -380,13 +451,14 @@ impl ScalarUDFImpl for ToTimestampFunc {
let tz = self.timezone.clone();
match args[0].data_type() {
- Int32 | Int64 => args[0]
+ Int8 | Int16 | Int32 | Int64 | UInt8 | UInt16 | UInt32 | UInt64 => args[0]
.cast_to(&Timestamp(Second, None), None)?
.cast_to(&Timestamp(Nanosecond, tz), None),
Null | Timestamp(_, _) => args[0].cast_to(&Timestamp(Nanosecond, tz), None),
- Float64 => {
+ Float16 | Float32 | Float64 => {
+ let arg = args[0].cast_to(&Float64, None)?;
Review Comment:
> Float16 still needs to be cast to Float64
Could you elaborate on this?
I was mainly hoping to eliminate unnecessary casts where possible since they technically allocate new arrays
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -325,6 +327,75 @@ impl_to_timestamp_constructors!(ToTimestampMillisFunc);
impl_to_timestamp_constructors!(ToTimestampMicrosFunc);
impl_to_timestamp_constructors!(ToTimestampNanosFunc);
+fn decimal_to_nanoseconds(value: i128, scale: i8) -> i64 {
+ let nanos_exponent = 9_i16 - scale as i16;
+ let timestamp_nanos = if nanos_exponent >= 0 {
+ value * 10_i128.pow(nanos_exponent as u32)
+ } else {
+ value / 10_i128.pow(nanos_exponent.unsigned_abs() as u32)
+ };
+ timestamp_nanos as i64
+}
+
+fn decimal128_to_timestamp_nanos(
Review Comment:
I notice for decimal256 we try to get the i128 value; is there any edge case if it doesn't fit in an i128, or is it not a concern as that value would be too large for timestamp output anyway?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #19663:
URL: https://github.com/apache/datafusion/pull/19663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2675463412
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -325,6 +327,75 @@ impl_to_timestamp_constructors!(ToTimestampMillisFunc);
impl_to_timestamp_constructors!(ToTimestampMicrosFunc);
impl_to_timestamp_constructors!(ToTimestampNanosFunc);
+fn decimal_to_nanoseconds(value: i128, scale: i8) -> i64 {
+ let nanos_exponent = 9_i16 - scale as i16;
+ let timestamp_nanos = if nanos_exponent >= 0 {
+ value * 10_i128.pow(nanos_exponent as u32)
+ } else {
+ value / 10_i128.pow(nanos_exponent.unsigned_abs() as u32)
+ };
+ timestamp_nanos as i64
+}
+
+fn decimal128_to_timestamp_nanos(
Review Comment:
Perhaps we should just bundle decimal256 with the same path that decimal32/decimal64 takes; that is we do the cast to decimal128 and only have conversion code for that path, so we can let the arrow kernel handle those checks for us
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2665149884
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -380,13 +380,14 @@ impl ScalarUDFImpl for ToTimestampFunc {
let tz = self.timezone.clone();
match args[0].data_type() {
- Int32 | Int64 => args[0]
+ Int8 | Int16 | Int32 | Int64 | UInt8 | UInt16 | UInt32 | UInt64 => args[0]
.cast_to(&Timestamp(Second, None), None)?
.cast_to(&Timestamp(Nanosecond, tz), None),
Null | Timestamp(_, _) => args[0].cast_to(&Timestamp(Nanosecond, tz), None),
- Float64 => {
+ Float32 | Float64 => {
Review Comment:
I think we should include Float16 whilst we're at it
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
Review Comment:
I was checking the nearby code and found this oddity; we support only scalar decimal128s? thats interesting 🤔
Might be worth correcting this so it works with array types too (like the other arms) and expanding support to the other decimal types for completeness
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2666266459
##########
datafusion/sqllogictest/test_files/datetime/timestamps.slt:
##########
@@ -4422,6 +4422,616 @@ FROM (VALUES
1970-01-01T00:00:00.000000005Z
+##########
+## to_timestamp functions with all numeric types
+##########
+
+# Test to_timestamp with all integer types
+# Int8
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:01:40
+
+# Int16
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:16:40
+
+# Int32
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'Int32'));
+----
+1970-01-02T00:00:00
+
+# Int64
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'Int64'));
+----
+1970-01-02T00:00:00
+
+# UInt8
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:01:40
+
+# UInt16
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:16:40
+
+# UInt32
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'UInt32'));
+----
+1970-01-02T00:00:00
+
+# UInt64
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'UInt64'));
+----
+1970-01-02T00:00:00
+
+# Float32
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float32'));
+----
+1970-01-01T00:00:01.500
+
+# Float64
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float64'));
+----
+1970-01-01T00:00:01.500
+
+# Test to_timestamp_seconds with all integer types
+# Int8
+query P
+SELECT to_timestamp_seconds(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_seconds(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:01:40
+
+# Int16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:16:40
+
+# UInt8
+query P
+SELECT to_timestamp_seconds(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:01:40
+
+# UInt16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:16:40
+
+# UInt32
+query P
+SELECT to_timestamp_seconds(arrow_cast(86400, 'UInt32'));
+----
+1970-01-02T00:00:00
+
+# UInt64
+query P
+SELECT to_timestamp_seconds(arrow_cast(86400, 'UInt64'));
+----
+1970-01-02T00:00:00
+
+# Float32
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float32'));
+----
+1970-01-01T00:00:01
+
+# Float64
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float64'));
+----
+1970-01-01T00:00:01
+
+# Test to_timestamp_millis with all integer types
+# Int8
+query P
+SELECT to_timestamp_millis(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_millis(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:00:00.100
+
+# Int16
+query P
+SELECT to_timestamp_millis(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:00:01
+
+# UInt8
+query P
+SELECT to_timestamp_millis(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:00:00.100
+
+# UInt16
+query P
+SELECT to_timestamp_millis(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:00:01
+
+# UInt32
+query P
+SELECT to_timestamp_millis(arrow_cast(86400000, 'UInt32'));
+----
+1970-01-02T00:00:00
+
+# UInt64
+query P
+SELECT to_timestamp_millis(arrow_cast(86400000, 'UInt64'));
+----
+1970-01-02T00:00:00
+
+# Float32
+query P
+SELECT to_timestamp_millis(arrow_cast(1000.9, 'Float32'));
+----
+1970-01-01T00:00:01
+
+# Float64
+query P
+SELECT to_timestamp_millis(arrow_cast(1000.9, 'Float64'));
+----
+1970-01-01T00:00:01
+
+# Test to_timestamp_micros with all integer types
+# Int8
+query P
+SELECT to_timestamp_micros(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_micros(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:00:00.000100
+
+# Int16
+query P
+SELECT to_timestamp_micros(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:00:00.001
+
+# UInt8
+query P
+SELECT to_timestamp_micros(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:00:00.000100
+
+# UInt16
+query P
+SELECT to_timestamp_micros(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:00:00.001
+
+# UInt32
+query P
+SELECT to_timestamp_micros(arrow_cast(1000000, 'UInt32'));
+----
+1970-01-01T00:00:01
+
+# UInt64
+query P
+SELECT to_timestamp_micros(arrow_cast(1000000, 'UInt64'));
+----
+1970-01-01T00:00:01
+
+# Float32
+query P
+SELECT to_timestamp_micros(arrow_cast(1000000.9, 'Float32'));
+----
+1970-01-01T00:00:01
+
+# Float64
+query P
+SELECT to_timestamp_micros(arrow_cast(1000000.9, 'Float64'));
+----
+1970-01-01T00:00:01
+
+# Test to_timestamp_nanos with all integer types
+# Int8
+query P
+SELECT to_timestamp_nanos(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_nanos(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:00:00.000000100
+
+# Int16
+query P
+SELECT to_timestamp_nanos(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:00:00.000001
+
+# UInt8
+query P
+SELECT to_timestamp_nanos(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:00:00.000000100
+
+# UInt16
+query P
+SELECT to_timestamp_nanos(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:00:00.000001
+
+# UInt32
+query P
+SELECT to_timestamp_nanos(arrow_cast(1000000000, 'UInt32'));
+----
+1970-01-01T00:00:01
+
+# UInt64
+query P
+SELECT to_timestamp_nanos(arrow_cast(1000000000, 'UInt64'));
+----
+1970-01-01T00:00:01
+
+# Float32
+query P
+SELECT to_timestamp_nanos(arrow_cast(1000000000.9, 'Float32'));
+----
+1970-01-01T00:00:01
+
+# Float64
+query P
+SELECT to_timestamp_nanos(arrow_cast(1000000000.9, 'Float64'));
+----
+1970-01-01T00:00:01
+
+# Verify arrow_typeof for all to_timestamp functions with various input types
+query T
+SELECT arrow_typeof(to_timestamp(arrow_cast(0, 'Int8')));
+----
+Timestamp(ns)
+
+query T
+SELECT arrow_typeof(to_timestamp(arrow_cast(0, 'UInt64')));
+----
+Timestamp(ns)
+
+query T
+SELECT arrow_typeof(to_timestamp(arrow_cast(0.0, 'Float32')));
+----
+Timestamp(ns)
+
+query T
+SELECT arrow_typeof(to_timestamp_seconds(arrow_cast(0, 'Int8')));
+----
+Timestamp(s)
+
+query T
+SELECT arrow_typeof(to_timestamp_seconds(arrow_cast(0, 'UInt64')));
+----
+Timestamp(s)
+
+query T
+SELECT arrow_typeof(to_timestamp_seconds(arrow_cast(0.0, 'Float32')));
+----
+Timestamp(s)
+
+query T
+SELECT arrow_typeof(to_timestamp_millis(arrow_cast(0, 'Int8')));
+----
+Timestamp(ms)
+
+query T
+SELECT arrow_typeof(to_timestamp_millis(arrow_cast(0, 'UInt64')));
+----
+Timestamp(ms)
+
+query T
+SELECT arrow_typeof(to_timestamp_millis(arrow_cast(0.0, 'Float32')));
+----
+Timestamp(ms)
+
+query T
+SELECT arrow_typeof(to_timestamp_micros(arrow_cast(0, 'Int8')));
+----
+Timestamp(µs)
+
+query T
+SELECT arrow_typeof(to_timestamp_micros(arrow_cast(0, 'UInt64')));
+----
+Timestamp(µs)
+
+query T
+SELECT arrow_typeof(to_timestamp_micros(arrow_cast(0.0, 'Float32')));
+----
+Timestamp(µs)
+
+query T
+SELECT arrow_typeof(to_timestamp_nanos(arrow_cast(0, 'Int8')));
+----
+Timestamp(ns)
+
+query T
+SELECT arrow_typeof(to_timestamp_nanos(arrow_cast(0, 'UInt64')));
+----
+Timestamp(ns)
+
+query T
+SELECT arrow_typeof(to_timestamp_nanos(arrow_cast(0.0, 'Float32')));
+----
+Timestamp(ns)
+
+# Test Decimal128 support for all to_timestamp functions
+# to_timestamp with Decimal128
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Decimal128(10,1)'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Decimal128(10,1)'));
+----
+1970-01-01T00:00:01.500
+
+query P
+SELECT to_timestamp(arrow_cast(86400.123, 'Decimal128(10,3)'));
+----
+1970-01-02T00:00:00.123
+
+# to_timestamp_seconds with Decimal128
+query P
+SELECT to_timestamp_seconds(arrow_cast(0, 'Decimal128(10,0)'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_seconds(arrow_cast(86400, 'Decimal128(10,0)'));
+----
+1970-01-02T00:00:00
+
+# to_timestamp_millis with Decimal128
+query P
+SELECT to_timestamp_millis(arrow_cast(0, 'Decimal128(10,0)'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_millis(arrow_cast(1000, 'Decimal128(10,0)'));
+----
+1970-01-01T00:00:01
+
+query P
+SELECT to_timestamp_millis(arrow_cast(86400000, 'Decimal128(15,0)'));
+----
+1970-01-02T00:00:00
+
+# to_timestamp_micros with Decimal128
+query P
+SELECT to_timestamp_micros(arrow_cast(0, 'Decimal128(10,0)'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_micros(arrow_cast(1000000, 'Decimal128(10,0)'));
+----
+1970-01-01T00:00:01
+
+query P
+SELECT to_timestamp_micros(arrow_cast(86400000000, 'Decimal128(15,0)'));
+----
+1970-01-02T00:00:00
+
+# to_timestamp_nanos with Decimal128
+query P
+SELECT to_timestamp_nanos(arrow_cast(0, 'Decimal128(10,0)'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_nanos(arrow_cast(1000000000, 'Decimal128(15,0)'));
+----
+1970-01-01T00:00:01
+
+query P
+SELECT to_timestamp_nanos(arrow_cast(86400000000000, 'Decimal128(20,0)'));
+----
+1970-01-02T00:00:00
+
+# Verify arrow_typeof for Decimal128 inputs
+query T
+SELECT arrow_typeof(to_timestamp(arrow_cast(0, 'Decimal128(10,0)')));
+----
+Timestamp(ns)
+
+query T
+SELECT arrow_typeof(to_timestamp_seconds(arrow_cast(0, 'Decimal128(10,0)')));
+----
+Timestamp(s)
+
+query T
+SELECT arrow_typeof(to_timestamp_millis(arrow_cast(0, 'Decimal128(10,0)')));
+----
+Timestamp(ms)
+
+query T
+SELECT arrow_typeof(to_timestamp_micros(arrow_cast(0, 'Decimal128(10,0)')));
+----
+Timestamp(µs)
+
+query T
+SELECT arrow_typeof(to_timestamp_nanos(arrow_cast(0, 'Decimal128(10,0)')));
+----
+Timestamp(ns)
+
+# Test negative values
+# to_timestamp with negative seconds
+query P
+SELECT to_timestamp(arrow_cast(-86400, 'Int32'));
+----
+1969-12-31T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(-1, 'Int64'));
+----
+1969-12-31T23:59:59
+
+query P
+SELECT to_timestamp(arrow_cast(-0.5, 'Float64'));
+----
+1969-12-31T23:59:59.500
+
+# to_timestamp_seconds with negative values
+query P
+SELECT to_timestamp_seconds(arrow_cast(-86400, 'Int32'));
+----
+1969-12-31T00:00:00
+
+query P
+SELECT to_timestamp_seconds(arrow_cast(-1, 'Int64'));
+----
+1969-12-31T23:59:59
+
+# to_timestamp_millis with negative values
+query P
+SELECT to_timestamp_millis(arrow_cast(-1000, 'Int32'));
+----
+1969-12-31T23:59:59
+
+query P
+SELECT to_timestamp_millis(arrow_cast(-1, 'Int64'));
+----
+1969-12-31T23:59:59.999
+
+# to_timestamp_micros with negative values
+query P
+SELECT to_timestamp_micros(arrow_cast(-1000000, 'Int32'));
+----
+1969-12-31T23:59:59
+
+query P
+SELECT to_timestamp_micros(arrow_cast(-1, 'Int64'));
+----
+1969-12-31T23:59:59.999999
+
+# to_timestamp_nanos with negative values
+query P
+SELECT to_timestamp_nanos(arrow_cast(-1000000000, 'Int64'));
+----
+1969-12-31T23:59:59
+
+query P
+SELECT to_timestamp_nanos(arrow_cast(-1, 'Int64'));
+----
+1969-12-31T23:59:59.999999999
+
+# Test large unsigned values
+query P
+SELECT to_timestamp_seconds(arrow_cast(4294967295, 'UInt64'));
+----
+2106-02-07T06:28:15
+
+# Large UInt64 value for milliseconds
+query P
+SELECT to_timestamp_millis(arrow_cast(4294967295000, 'UInt64'));
Review Comment:
Please add a test with a number that is bigger than i64::MAX and smaller than u64::MAX
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "gokselk (via GitHub)" <gi...@apache.org>.
gokselk commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2672999726
##########
datafusion/sqllogictest/test_files/datetime/timestamps.slt:
##########
@@ -4422,6 +4422,714 @@ FROM (VALUES
1970-01-01T00:00:00.000000005Z
+##########
+## to_timestamp functions with all numeric types
+##########
+
+# Test to_timestamp with all integer types
+# Int8
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:01:40
+
+# Int16
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:16:40
+
+# Int32
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'Int32'));
+----
+1970-01-02T00:00:00
+
+# Int64
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'Int64'));
+----
+1970-01-02T00:00:00
+
+# UInt8
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:01:40
+
+# UInt16
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:16:40
+
+# UInt32
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'UInt32'));
+----
+1970-01-02T00:00:00
+
+# UInt64
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'UInt64'));
+----
+1970-01-02T00:00:00
+
+# Float16
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float16'));
+----
+1970-01-01T00:00:01.500
+
+# Float32
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float32'));
+----
+1970-01-01T00:00:01.500
+
+# Float64
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float64'));
+----
+1970-01-01T00:00:01.500
+
+# Test to_timestamp_seconds with all integer types
+# Int8
+query P
+SELECT to_timestamp_seconds(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_seconds(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:01:40
+
+# Int16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:16:40
+
+# UInt8
+query P
+SELECT to_timestamp_seconds(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:01:40
+
+# UInt16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:16:40
+
+# UInt32
+query P
+SELECT to_timestamp_seconds(arrow_cast(86400, 'UInt32'));
+----
+1970-01-02T00:00:00
+
+# UInt64
+query P
+SELECT to_timestamp_seconds(arrow_cast(86400, 'UInt64'));
+----
+1970-01-02T00:00:00
+
+# Float16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float16'));
+----
+1970-01-01T00:00:01
+
+# Float32
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float32'));
+----
+1970-01-01T00:00:01
+
+# Float64
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float64'));
+----
+1970-01-01T00:00:01
+
+# Test to_timestamp_millis with all integer types
Review Comment:
My bad, should be addressed on b194930 and 6fb4435.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#issuecomment-3719343201
Thanks for submitting this PR @gokselk, much appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "gokselk (via GitHub)" <gi...@apache.org>.
gokselk closed pull request #19663: Add support for additional numeric types in to_timestamp functions
URL: https://github.com/apache/datafusion/pull/19663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "gokselk (via GitHub)" <gi...@apache.org>.
gokselk commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2675656169
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -325,6 +327,75 @@ impl_to_timestamp_constructors!(ToTimestampMillisFunc);
impl_to_timestamp_constructors!(ToTimestampMicrosFunc);
impl_to_timestamp_constructors!(ToTimestampNanosFunc);
+fn decimal_to_nanoseconds(value: i128, scale: i8) -> i64 {
+ let nanos_exponent = 9_i16 - scale as i16;
+ let timestamp_nanos = if nanos_exponent >= 0 {
+ value * 10_i128.pow(nanos_exponent as u32)
+ } else {
+ value / 10_i128.pow(nanos_exponent.unsigned_abs() as u32)
+ };
+ timestamp_nanos as i64
+}
+
+fn decimal128_to_timestamp_nanos(
Review Comment:
Good idea, I did some changes about that in 376bd9d.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2675715923
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -380,13 +451,14 @@ impl ScalarUDFImpl for ToTimestampFunc {
let tz = self.timezone.clone();
match args[0].data_type() {
- Int32 | Int64 => args[0]
+ Int8 | Int16 | Int32 | Int64 | UInt8 | UInt16 | UInt32 | UInt64 => args[0]
.cast_to(&Timestamp(Second, None), None)?
.cast_to(&Timestamp(Nanosecond, tz), None),
Null | Timestamp(_, _) => args[0].cast_to(&Timestamp(Nanosecond, tz), None),
- Float64 => {
+ Float16 | Float32 | Float64 => {
+ let arg = args[0].cast_to(&Float64, None)?;
Review Comment:
Yep 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "gokselk (via GitHub)" <gi...@apache.org>.
gokselk commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2675140996
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -325,6 +327,75 @@ impl_to_timestamp_constructors!(ToTimestampMillisFunc);
impl_to_timestamp_constructors!(ToTimestampMicrosFunc);
impl_to_timestamp_constructors!(ToTimestampNanosFunc);
+fn decimal_to_nanoseconds(value: i128, scale: i8) -> i64 {
+ let nanos_exponent = 9_i16 - scale as i16;
+ let timestamp_nanos = if nanos_exponent >= 0 {
+ value * 10_i128.pow(nanos_exponent as u32)
+ } else {
+ value / 10_i128.pow(nanos_exponent.unsigned_abs() as u32)
+ };
+ timestamp_nanos as i64
+}
+
+fn decimal128_to_timestamp_nanos(
Review Comment:
Yes, even i64 nanos caps at around 292 years from epoch. But `as_i128()` silently truncates, I can add bounds checking for an error instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#issuecomment-3733929016
Thanks @gokselk, @Omega359 & @martin-g
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2669934845
##########
datafusion/sqllogictest/test_files/datetime/timestamps.slt:
##########
@@ -4422,6 +4422,714 @@ FROM (VALUES
1970-01-01T00:00:00.000000005Z
+##########
+## to_timestamp functions with all numeric types
+##########
+
+# Test to_timestamp with all integer types
+# Int8
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:01:40
+
+# Int16
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:16:40
+
+# Int32
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'Int32'));
+----
+1970-01-02T00:00:00
+
+# Int64
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'Int64'));
+----
+1970-01-02T00:00:00
+
+# UInt8
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:01:40
+
+# UInt16
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:16:40
+
+# UInt32
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'UInt32'));
+----
+1970-01-02T00:00:00
+
+# UInt64
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'UInt64'));
+----
+1970-01-02T00:00:00
+
+# Float16
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float16'));
+----
+1970-01-01T00:00:01.500
+
+# Float32
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float32'));
+----
+1970-01-01T00:00:01.500
+
+# Float64
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float64'));
+----
+1970-01-01T00:00:01.500
+
+# Test to_timestamp_seconds with all integer types
+# Int8
+query P
+SELECT to_timestamp_seconds(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_seconds(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:01:40
+
+# Int16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:16:40
+
+# UInt8
+query P
+SELECT to_timestamp_seconds(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:01:40
+
+# UInt16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:16:40
+
+# UInt32
+query P
+SELECT to_timestamp_seconds(arrow_cast(86400, 'UInt32'));
+----
+1970-01-02T00:00:00
+
+# UInt64
+query P
+SELECT to_timestamp_seconds(arrow_cast(86400, 'UInt64'));
+----
+1970-01-02T00:00:00
+
+# Float16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float16'));
+----
+1970-01-01T00:00:01
+
+# Float32
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float32'));
+----
+1970-01-01T00:00:01
+
+# Float64
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float64'));
+----
+1970-01-01T00:00:01
+
+# Test to_timestamp_millis with all integer types
Review Comment:
I don't seem to see a test with positive i32 && i64. i32 is likely handled somewhere already since it's default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "gokselk (via GitHub)" <gi...@apache.org>.
gokselk commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2671605069
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -325,6 +327,75 @@ impl_to_timestamp_constructors!(ToTimestampMillisFunc);
impl_to_timestamp_constructors!(ToTimestampMicrosFunc);
impl_to_timestamp_constructors!(ToTimestampNanosFunc);
+fn decimal_to_nanoseconds(value: i128, scale: i8) -> i64 {
Review Comment:
rescale_decimal preserves logical value, doesn't multiply. I think we'd still need to compute 10^(9-scale).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "gokselk (via GitHub)" <gi...@apache.org>.
gokselk commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2671636906
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -325,6 +327,75 @@ impl_to_timestamp_constructors!(ToTimestampMillisFunc);
impl_to_timestamp_constructors!(ToTimestampMicrosFunc);
impl_to_timestamp_constructors!(ToTimestampNanosFunc);
+fn decimal_to_nanoseconds(value: i128, scale: i8) -> i64 {
+ let nanos_exponent = 9_i16 - scale as i16;
+ let timestamp_nanos = if nanos_exponent >= 0 {
+ value * 10_i128.pow(nanos_exponent as u32)
+ } else {
+ value / 10_i128.pow(nanos_exponent.unsigned_abs() as u32)
+ };
+ timestamp_nanos as i64
+}
+
+fn decimal128_to_timestamp_nanos(
Review Comment:
Looked into this. Decimal32/64 to Decimal128 cast is lossless and cheap, so generic would add complexity without much gain. Happy to change if you see a case where it matters though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2670617288
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -325,6 +327,75 @@ impl_to_timestamp_constructors!(ToTimestampMillisFunc);
impl_to_timestamp_constructors!(ToTimestampMicrosFunc);
impl_to_timestamp_constructors!(ToTimestampNanosFunc);
+fn decimal_to_nanoseconds(value: i128, scale: i8) -> i64 {
Review Comment:
Is using rescale_decimal kernel here applicable?
- https://docs.rs/arrow/latest/arrow/compute/fn.rescale_decimal.html
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -380,13 +451,14 @@ impl ScalarUDFImpl for ToTimestampFunc {
let tz = self.timezone.clone();
match args[0].data_type() {
- Int32 | Int64 => args[0]
+ Int8 | Int16 | Int32 | Int64 | UInt8 | UInt16 | UInt32 | UInt64 => args[0]
.cast_to(&Timestamp(Second, None), None)?
.cast_to(&Timestamp(Nanosecond, tz), None),
Null | Timestamp(_, _) => args[0].cast_to(&Timestamp(Nanosecond, tz), None),
- Float64 => {
+ Float16 | Float32 | Float64 => {
+ let arg = args[0].cast_to(&Float64, None)?;
Review Comment:
I feel each of these can be done natively instead of casting to float64
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -325,6 +327,75 @@ impl_to_timestamp_constructors!(ToTimestampMillisFunc);
impl_to_timestamp_constructors!(ToTimestampMicrosFunc);
impl_to_timestamp_constructors!(ToTimestampNanosFunc);
+fn decimal_to_nanoseconds(value: i128, scale: i8) -> i64 {
+ let nanos_exponent = 9_i16 - scale as i16;
+ let timestamp_nanos = if nanos_exponent >= 0 {
+ value * 10_i128.pow(nanos_exponent as u32)
+ } else {
+ value / 10_i128.pow(nanos_exponent.unsigned_abs() as u32)
+ };
+ timestamp_nanos as i64
+}
+
+fn decimal128_to_timestamp_nanos(
Review Comment:
It would be nice if we could make this generic over the decimal types to not force the codepaths all through decimal128 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2667408877
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -325,6 +327,74 @@ impl_to_timestamp_constructors!(ToTimestampMillisFunc);
impl_to_timestamp_constructors!(ToTimestampMicrosFunc);
impl_to_timestamp_constructors!(ToTimestampNanosFunc);
+fn decimal_to_nanoseconds(value: i128, scale: i8) -> i64 {
+ let scale_factor = 10_i128.pow(scale as u32);
Review Comment:
Some other PRs started adding support for negative scale:
* https://github.com/apache/datafusion/pull/19409
* https://github.com/apache/datafusion/pull/19584#pullrequestreview-3621905861
The casting to u32 here would prevent this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "gokselk (via GitHub)" <gi...@apache.org>.
gokselk commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2667595877
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -325,6 +327,74 @@ impl_to_timestamp_constructors!(ToTimestampMillisFunc);
impl_to_timestamp_constructors!(ToTimestampMicrosFunc);
impl_to_timestamp_constructors!(ToTimestampNanosFunc);
+fn decimal_to_nanoseconds(value: i128, scale: i8) -> i64 {
+ let scale_factor = 10_i128.pow(scale as u32);
Review Comment:
Should be resolved in 03c1937.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "gokselk (via GitHub)" <gi...@apache.org>.
gokselk commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2667394121
##########
datafusion/sqllogictest/test_files/datetime/timestamps.slt:
##########
@@ -4422,6 +4422,616 @@ FROM (VALUES
1970-01-01T00:00:00.000000005Z
+##########
+## to_timestamp functions with all numeric types
+##########
+
+# Test to_timestamp with all integer types
+# Int8
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:01:40
+
+# Int16
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:16:40
+
+# Int32
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'Int32'));
+----
+1970-01-02T00:00:00
+
+# Int64
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'Int64'));
+----
+1970-01-02T00:00:00
+
+# UInt8
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:01:40
+
+# UInt16
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:16:40
+
+# UInt32
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'UInt32'));
+----
+1970-01-02T00:00:00
+
+# UInt64
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'UInt64'));
+----
+1970-01-02T00:00:00
+
+# Float32
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float32'));
+----
+1970-01-01T00:00:01.500
+
+# Float64
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float64'));
+----
+1970-01-01T00:00:01.500
+
+# Test to_timestamp_seconds with all integer types
+# Int8
+query P
+SELECT to_timestamp_seconds(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_seconds(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:01:40
+
+# Int16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:16:40
+
+# UInt8
+query P
+SELECT to_timestamp_seconds(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:01:40
+
+# UInt16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:16:40
+
+# UInt32
+query P
+SELECT to_timestamp_seconds(arrow_cast(86400, 'UInt32'));
+----
+1970-01-02T00:00:00
+
+# UInt64
+query P
+SELECT to_timestamp_seconds(arrow_cast(86400, 'UInt64'));
+----
+1970-01-02T00:00:00
+
+# Float32
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float32'));
+----
+1970-01-01T00:00:01
+
+# Float64
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float64'));
+----
+1970-01-01T00:00:01
+
+# Test to_timestamp_millis with all integer types
+# Int8
+query P
+SELECT to_timestamp_millis(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_millis(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:00:00.100
+
+# Int16
+query P
+SELECT to_timestamp_millis(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:00:01
+
+# UInt8
+query P
+SELECT to_timestamp_millis(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:00:00.100
+
+# UInt16
+query P
+SELECT to_timestamp_millis(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:00:01
+
+# UInt32
+query P
+SELECT to_timestamp_millis(arrow_cast(86400000, 'UInt32'));
+----
+1970-01-02T00:00:00
+
+# UInt64
+query P
+SELECT to_timestamp_millis(arrow_cast(86400000, 'UInt64'));
+----
+1970-01-02T00:00:00
+
+# Float32
+query P
+SELECT to_timestamp_millis(arrow_cast(1000.9, 'Float32'));
+----
+1970-01-01T00:00:01
+
+# Float64
+query P
+SELECT to_timestamp_millis(arrow_cast(1000.9, 'Float64'));
+----
+1970-01-01T00:00:01
+
+# Test to_timestamp_micros with all integer types
+# Int8
+query P
+SELECT to_timestamp_micros(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_micros(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:00:00.000100
+
+# Int16
+query P
+SELECT to_timestamp_micros(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:00:00.001
+
+# UInt8
+query P
+SELECT to_timestamp_micros(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:00:00.000100
+
+# UInt16
+query P
+SELECT to_timestamp_micros(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:00:00.001
+
+# UInt32
+query P
+SELECT to_timestamp_micros(arrow_cast(1000000, 'UInt32'));
+----
+1970-01-01T00:00:01
+
+# UInt64
+query P
+SELECT to_timestamp_micros(arrow_cast(1000000, 'UInt64'));
+----
+1970-01-01T00:00:01
+
+# Float32
+query P
+SELECT to_timestamp_micros(arrow_cast(1000000.9, 'Float32'));
+----
+1970-01-01T00:00:01
+
+# Float64
+query P
+SELECT to_timestamp_micros(arrow_cast(1000000.9, 'Float64'));
+----
+1970-01-01T00:00:01
+
+# Test to_timestamp_nanos with all integer types
+# Int8
+query P
+SELECT to_timestamp_nanos(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_nanos(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:00:00.000000100
+
+# Int16
+query P
+SELECT to_timestamp_nanos(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:00:00.000001
+
+# UInt8
+query P
+SELECT to_timestamp_nanos(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:00:00.000000100
+
+# UInt16
+query P
+SELECT to_timestamp_nanos(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:00:00.000001
+
+# UInt32
+query P
+SELECT to_timestamp_nanos(arrow_cast(1000000000, 'UInt32'));
+----
+1970-01-01T00:00:01
+
+# UInt64
+query P
+SELECT to_timestamp_nanos(arrow_cast(1000000000, 'UInt64'));
+----
+1970-01-01T00:00:01
+
+# Float32
+query P
+SELECT to_timestamp_nanos(arrow_cast(1000000000.9, 'Float32'));
+----
+1970-01-01T00:00:01
+
+# Float64
+query P
+SELECT to_timestamp_nanos(arrow_cast(1000000000.9, 'Float64'));
+----
+1970-01-01T00:00:01
+
+# Verify arrow_typeof for all to_timestamp functions with various input types
+query T
+SELECT arrow_typeof(to_timestamp(arrow_cast(0, 'Int8')));
+----
+Timestamp(ns)
+
+query T
+SELECT arrow_typeof(to_timestamp(arrow_cast(0, 'UInt64')));
+----
+Timestamp(ns)
+
+query T
+SELECT arrow_typeof(to_timestamp(arrow_cast(0.0, 'Float32')));
+----
+Timestamp(ns)
+
+query T
+SELECT arrow_typeof(to_timestamp_seconds(arrow_cast(0, 'Int8')));
+----
+Timestamp(s)
+
+query T
+SELECT arrow_typeof(to_timestamp_seconds(arrow_cast(0, 'UInt64')));
+----
+Timestamp(s)
+
+query T
+SELECT arrow_typeof(to_timestamp_seconds(arrow_cast(0.0, 'Float32')));
+----
+Timestamp(s)
+
+query T
+SELECT arrow_typeof(to_timestamp_millis(arrow_cast(0, 'Int8')));
+----
+Timestamp(ms)
+
+query T
+SELECT arrow_typeof(to_timestamp_millis(arrow_cast(0, 'UInt64')));
+----
+Timestamp(ms)
+
+query T
+SELECT arrow_typeof(to_timestamp_millis(arrow_cast(0.0, 'Float32')));
+----
+Timestamp(ms)
+
+query T
+SELECT arrow_typeof(to_timestamp_micros(arrow_cast(0, 'Int8')));
+----
+Timestamp(µs)
+
+query T
+SELECT arrow_typeof(to_timestamp_micros(arrow_cast(0, 'UInt64')));
+----
+Timestamp(µs)
+
+query T
+SELECT arrow_typeof(to_timestamp_micros(arrow_cast(0.0, 'Float32')));
+----
+Timestamp(µs)
+
+query T
+SELECT arrow_typeof(to_timestamp_nanos(arrow_cast(0, 'Int8')));
+----
+Timestamp(ns)
+
+query T
+SELECT arrow_typeof(to_timestamp_nanos(arrow_cast(0, 'UInt64')));
+----
+Timestamp(ns)
+
+query T
+SELECT arrow_typeof(to_timestamp_nanos(arrow_cast(0.0, 'Float32')));
+----
+Timestamp(ns)
+
+# Test Decimal128 support for all to_timestamp functions
+# to_timestamp with Decimal128
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Decimal128(10,1)'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Decimal128(10,1)'));
+----
+1970-01-01T00:00:01.500
+
+query P
+SELECT to_timestamp(arrow_cast(86400.123, 'Decimal128(10,3)'));
+----
+1970-01-02T00:00:00.123
+
+# to_timestamp_seconds with Decimal128
+query P
+SELECT to_timestamp_seconds(arrow_cast(0, 'Decimal128(10,0)'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_seconds(arrow_cast(86400, 'Decimal128(10,0)'));
+----
+1970-01-02T00:00:00
+
+# to_timestamp_millis with Decimal128
+query P
+SELECT to_timestamp_millis(arrow_cast(0, 'Decimal128(10,0)'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_millis(arrow_cast(1000, 'Decimal128(10,0)'));
+----
+1970-01-01T00:00:01
+
+query P
+SELECT to_timestamp_millis(arrow_cast(86400000, 'Decimal128(15,0)'));
+----
+1970-01-02T00:00:00
+
+# to_timestamp_micros with Decimal128
+query P
+SELECT to_timestamp_micros(arrow_cast(0, 'Decimal128(10,0)'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_micros(arrow_cast(1000000, 'Decimal128(10,0)'));
+----
+1970-01-01T00:00:01
+
+query P
+SELECT to_timestamp_micros(arrow_cast(86400000000, 'Decimal128(15,0)'));
+----
+1970-01-02T00:00:00
+
+# to_timestamp_nanos with Decimal128
+query P
+SELECT to_timestamp_nanos(arrow_cast(0, 'Decimal128(10,0)'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_nanos(arrow_cast(1000000000, 'Decimal128(15,0)'));
+----
+1970-01-01T00:00:01
+
+query P
+SELECT to_timestamp_nanos(arrow_cast(86400000000000, 'Decimal128(20,0)'));
+----
+1970-01-02T00:00:00
+
+# Verify arrow_typeof for Decimal128 inputs
+query T
+SELECT arrow_typeof(to_timestamp(arrow_cast(0, 'Decimal128(10,0)')));
+----
+Timestamp(ns)
+
+query T
+SELECT arrow_typeof(to_timestamp_seconds(arrow_cast(0, 'Decimal128(10,0)')));
+----
+Timestamp(s)
+
+query T
+SELECT arrow_typeof(to_timestamp_millis(arrow_cast(0, 'Decimal128(10,0)')));
+----
+Timestamp(ms)
+
+query T
+SELECT arrow_typeof(to_timestamp_micros(arrow_cast(0, 'Decimal128(10,0)')));
+----
+Timestamp(µs)
+
+query T
+SELECT arrow_typeof(to_timestamp_nanos(arrow_cast(0, 'Decimal128(10,0)')));
+----
+Timestamp(ns)
+
+# Test negative values
+# to_timestamp with negative seconds
+query P
+SELECT to_timestamp(arrow_cast(-86400, 'Int32'));
+----
+1969-12-31T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(-1, 'Int64'));
+----
+1969-12-31T23:59:59
+
+query P
+SELECT to_timestamp(arrow_cast(-0.5, 'Float64'));
+----
+1969-12-31T23:59:59.500
+
+# to_timestamp_seconds with negative values
+query P
+SELECT to_timestamp_seconds(arrow_cast(-86400, 'Int32'));
+----
+1969-12-31T00:00:00
+
+query P
+SELECT to_timestamp_seconds(arrow_cast(-1, 'Int64'));
+----
+1969-12-31T23:59:59
+
+# to_timestamp_millis with negative values
+query P
+SELECT to_timestamp_millis(arrow_cast(-1000, 'Int32'));
+----
+1969-12-31T23:59:59
+
+query P
+SELECT to_timestamp_millis(arrow_cast(-1, 'Int64'));
+----
+1969-12-31T23:59:59.999
+
+# to_timestamp_micros with negative values
+query P
+SELECT to_timestamp_micros(arrow_cast(-1000000, 'Int32'));
+----
+1969-12-31T23:59:59
+
+query P
+SELECT to_timestamp_micros(arrow_cast(-1, 'Int64'));
+----
+1969-12-31T23:59:59.999999
+
+# to_timestamp_nanos with negative values
+query P
+SELECT to_timestamp_nanos(arrow_cast(-1000000000, 'Int64'));
+----
+1969-12-31T23:59:59
+
+query P
+SELECT to_timestamp_nanos(arrow_cast(-1, 'Int64'));
+----
+1969-12-31T23:59:59.999999999
+
+# Test large unsigned values
+query P
+SELECT to_timestamp_seconds(arrow_cast(4294967295, 'UInt64'));
+----
+2106-02-07T06:28:15
+
+# Large UInt64 value for milliseconds
+query P
+SELECT to_timestamp_millis(arrow_cast(4294967295000, 'UInt64'));
Review Comment:
Resolved in 17ff223.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "gokselk (via GitHub)" <gi...@apache.org>.
gokselk commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2671581256
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -380,13 +451,14 @@ impl ScalarUDFImpl for ToTimestampFunc {
let tz = self.timezone.clone();
match args[0].data_type() {
- Int32 | Int64 => args[0]
+ Int8 | Int16 | Int32 | Int64 | UInt8 | UInt16 | UInt32 | UInt64 => args[0]
.cast_to(&Timestamp(Second, None), None)?
.cast_to(&Timestamp(Nanosecond, tz), None),
Null | Timestamp(_, _) => args[0].cast_to(&Timestamp(Nanosecond, tz), None),
- Float64 => {
+ Float16 | Float32 | Float64 => {
+ let arg = args[0].cast_to(&Float64, None)?;
Review Comment:
Considered native handling, but Float32 to Float64 cast is lossless and Float16 still needs to be cast to Float64. Single code path is cleaner IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "gokselk (via GitHub)" <gi...@apache.org>.
gokselk commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2671667070
##########
datafusion/sqllogictest/test_files/datetime/timestamps.slt:
##########
@@ -4422,6 +4422,714 @@ FROM (VALUES
1970-01-01T00:00:00.000000005Z
+##########
+## to_timestamp functions with all numeric types
+##########
+
+# Test to_timestamp with all integer types
+# Int8
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:01:40
+
+# Int16
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:16:40
+
+# Int32
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'Int32'));
+----
+1970-01-02T00:00:00
+
+# Int64
+query P
+SELECT to_timestamp(arrow_cast(0, 'Int64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'Int64'));
+----
+1970-01-02T00:00:00
+
+# UInt8
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:01:40
+
+# UInt16
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:16:40
+
+# UInt32
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'UInt32'));
+----
+1970-01-02T00:00:00
+
+# UInt64
+query P
+SELECT to_timestamp(arrow_cast(0, 'UInt64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(86400, 'UInt64'));
+----
+1970-01-02T00:00:00
+
+# Float16
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float16'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float16'));
+----
+1970-01-01T00:00:01.500
+
+# Float32
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float32'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float32'));
+----
+1970-01-01T00:00:01.500
+
+# Float64
+query P
+SELECT to_timestamp(arrow_cast(0.0, 'Float64'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp(arrow_cast(1.5, 'Float64'));
+----
+1970-01-01T00:00:01.500
+
+# Test to_timestamp_seconds with all integer types
+# Int8
+query P
+SELECT to_timestamp_seconds(arrow_cast(0, 'Int8'));
+----
+1970-01-01T00:00:00
+
+query P
+SELECT to_timestamp_seconds(arrow_cast(100, 'Int8'));
+----
+1970-01-01T00:01:40
+
+# Int16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1000, 'Int16'));
+----
+1970-01-01T00:16:40
+
+# UInt8
+query P
+SELECT to_timestamp_seconds(arrow_cast(100, 'UInt8'));
+----
+1970-01-01T00:01:40
+
+# UInt16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1000, 'UInt16'));
+----
+1970-01-01T00:16:40
+
+# UInt32
+query P
+SELECT to_timestamp_seconds(arrow_cast(86400, 'UInt32'));
+----
+1970-01-02T00:00:00
+
+# UInt64
+query P
+SELECT to_timestamp_seconds(arrow_cast(86400, 'UInt64'));
+----
+1970-01-02T00:00:00
+
+# Float16
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float16'));
+----
+1970-01-01T00:00:01
+
+# Float32
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float32'));
+----
+1970-01-01T00:00:01
+
+# Float64
+query P
+SELECT to_timestamp_seconds(arrow_cast(1.9, 'Float64'));
+----
+1970-01-01T00:00:01
+
+# Test to_timestamp_millis with all integer types
Review Comment:
Resolved in e8587cf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for additional numeric types in to_timestamp functions [datafusion]
Posted by "gokselk (via GitHub)" <gi...@apache.org>.
gokselk commented on code in PR #19663:
URL: https://github.com/apache/datafusion/pull/19663#discussion_r2675662607
##########
datafusion/functions/src/datetime/to_timestamp.rs:
##########
@@ -380,13 +451,14 @@ impl ScalarUDFImpl for ToTimestampFunc {
let tz = self.timezone.clone();
match args[0].data_type() {
- Int32 | Int64 => args[0]
+ Int8 | Int16 | Int32 | Int64 | UInt8 | UInt16 | UInt32 | UInt64 => args[0]
.cast_to(&Timestamp(Second, None), None)?
.cast_to(&Timestamp(Nanosecond, tz), None),
Null | Timestamp(_, _) => args[0].cast_to(&Timestamp(Nanosecond, tz), None),
- Float64 => {
+ Float16 | Float32 | Float64 => {
+ let arg = args[0].cast_to(&Float64, None)?;
Review Comment:
Done in 07258356a. Is this what you had in mind?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org