You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jonathanc-n (via GitHub)" <gi...@apache.org> on 2025/03/31 22:38:28 UTC
[PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]
jonathanc-n opened a new pull request, #1150:
URL: https://github.com/apache/iceberg-rust/pull/1150
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #1110 .
-->
- Closes #timestamp.
## What changes are included in this PR?
Add bucket transforms for `TimestampNs` and TimestampTzNs`
<!--
Provide a summary of the modifications in this PR. List the main changes such as new features, bug fixes, refactoring, or any other updates.
-->
## Are these changes tested?
Unit tests
<!--
Specify what test covers (unit test, integration test, etc.).
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on code in PR #1150:
URL: https://github.com/apache/iceberg-rust/pull/1150#discussion_r2023707366
##########
crates/iceberg/src/transform/bucket.rs:
##########
@@ -888,4 +904,66 @@ mod test {
Datum::int(32)
);
}
+
+ #[test]
+ fn test_timestamptz_literal() {
+ let bucket = Bucket::new(100);
+ assert_eq!(
+ bucket
+ .transform_literal(&Datum::timestamptz_micros(1510871468000000))
+ .unwrap()
+ .unwrap(),
+ Datum::int(7)
+ );
+ }
+
+ #[test]
+ fn test_timestamp_ns_literal() {
+ let bucket = Bucket::new(100);
+ let ns_value = 1510871468000000i64 * 1000;
+ assert_eq!(
+ bucket
+ .transform_literal(&Datum::timestamp_nanos(ns_value))
+ .unwrap()
+ .unwrap(),
+ Datum::int(79)
Review Comment:
Yes I applied the conversion to the `transform_literal` as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 merged PR #1150:
URL: https://github.com/apache/iceberg-rust/pull/1150
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on code in PR #1150:
URL: https://github.com/apache/iceberg-rust/pull/1150#discussion_r2023638265
##########
crates/iceberg/src/transform/bucket.rs:
##########
@@ -167,6 +167,16 @@ impl TransformFunction for Bucket {
.downcast_ref::<arrow_array::TimestampMicrosecondArray>()
.unwrap()
.unary(|v| self.bucket_timestamp(v)),
+ DataType::Time64(TimeUnit::Nanosecond) => input
Review Comment:
Good catch, It wasn't performing this conversion. The test is added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1150:
URL: https://github.com/apache/iceberg-rust/pull/1150#discussion_r2023158926
##########
crates/iceberg/src/transform/bucket.rs:
##########
@@ -167,6 +167,16 @@ impl TransformFunction for Bucket {
.downcast_ref::<arrow_array::TimestampMicrosecondArray>()
.unwrap()
.unary(|v| self.bucket_timestamp(v)),
+ DataType::Time64(TimeUnit::Nanosecond) => input
Review Comment:
in pyiceberg, `TimestampNanoType` in converted to micros precision before bucketing. This is to ensure that `TimestampType` and `TimestampNanoType` are transformed to the same value
https://github.com/apache/iceberg-python/blob/4d4714a46241d0d89519a2a605dbce27b713a60e/pyiceberg/transforms.py#L322-L340
I think we should add a test to ensure this here too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1150:
URL: https://github.com/apache/iceberg-rust/pull/1150#discussion_r2023649923
##########
crates/iceberg/src/transform/bucket.rs:
##########
@@ -888,4 +904,66 @@ mod test {
Datum::int(32)
);
}
+
+ #[test]
+ fn test_timestamptz_literal() {
+ let bucket = Bucket::new(100);
+ assert_eq!(
+ bucket
+ .transform_literal(&Datum::timestamptz_micros(1510871468000000))
+ .unwrap()
+ .unwrap(),
+ Datum::int(7)
+ );
+ }
+
+ #[test]
+ fn test_timestamp_ns_literal() {
+ let bucket = Bucket::new(100);
+ let ns_value = 1510871468000000i64 * 1000;
+ assert_eq!(
+ bucket
+ .transform_literal(&Datum::timestamp_nanos(ns_value))
+ .unwrap()
+ .unwrap(),
+ Datum::int(79)
Review Comment:
hm, since we're using the same time (`1510871468000000`) here
should the bucket value for `bucket(100)` transform be the same for this test and the test above (`test_timestamptz_literal`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1150:
URL: https://github.com/apache/iceberg-rust/pull/1150#issuecomment-2769870847
@Fokko FYI new transform function we can use for https://github.com/apache/iceberg-python/pull/1833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1150:
URL: https://github.com/apache/iceberg-rust/pull/1150#discussion_r2023646275
##########
crates/iceberg/src/transform/bucket.rs:
##########
@@ -167,6 +167,16 @@ impl TransformFunction for Bucket {
.downcast_ref::<arrow_array::TimestampMicrosecondArray>()
.unwrap()
.unary(|v| self.bucket_timestamp(v)),
+ DataType::Time64(TimeUnit::Nanosecond) => input
+ .as_any()
+ .downcast_ref::<arrow_array::Time64NanosecondArray>()
+ .unwrap()
+ .unary(|v| self.bucket_time(v / 1000)),
Review Comment:
there was an issue with rounding when performing division for the hour transform (https://github.com/apache/iceberg-rust/pull/1146)
but since we're dividing by `1000`, i dont think the same issue applies here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org