You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2025/03/28 08:18:02 UTC

[PR] Fix hour transform [iceberg-rust]

Fokko opened a new pull request, #1146:
URL: https://github.com/apache/iceberg-rust/pull/1146

   ## Which issue does this PR close?
   
   Seems like we're rounding towards zero by default by Rust. While we want to `floor` the value.
   
   Checked against the reference implementation:
   
   ![image](https://github.com/user-attachments/assets/48f8ac2e-c124-4b53-9888-40e3fa7e6cd2)
   
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   - Closes #.
   
   ## What changes are included in this PR?
   
   <!--
   Provide a summary of the modifications in this PR. List the main changes such as new features, bug fixes, refactoring, or any other updates.
   -->
   
   ## Are these changes tested?
   
   <!--
   Specify what test covers (unit test, integration test, etc.).
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Fix hour transform [iceberg-rust]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1146:
URL: https://github.com/apache/iceberg-rust/pull/1146#discussion_r2020240652


##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -337,12 +337,12 @@ pub struct Hour;
 impl Hour {
     #[inline]
     fn hour_timestamp_micro(v: i64) -> i32 {
-        (v / MICROSECONDS_PER_HOUR) as i32
+        (v as f64 / MICROSECONDS_PER_HOUR).floor() as i32

Review Comment:
   Out of curiosity, what's the risk of using the unstable API? We have limited test cases in place to verify the behavior. Shouldn't you use unstable API's at all?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Fix hour transform [iceberg-rust]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1146:
URL: https://github.com/apache/iceberg-rust/pull/1146#discussion_r2020238880


##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -337,12 +337,12 @@ pub struct Hour;
 impl Hour {
     #[inline]
     fn hour_timestamp_micro(v: i64) -> i32 {
-        (v / MICROSECONDS_PER_HOUR) as i32
+        (v as f64 / MICROSECONDS_PER_HOUR).floor() as i32

Review Comment:
   Love it, thanks for the suggestion! 🙌 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Fix hour transform [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1146:
URL: https://github.com/apache/iceberg-rust/pull/1146#discussion_r2018976984


##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -2760,14 +2738,23 @@ mod test {
     fn test_transform_hours_literal() {
         let hour = Box::new(super::Hour) as BoxedTransformFunction;
 
-        // Test TimestampMicrosecond
-        test_timestamp_and_tz_transform("2017-12-01 18:00:00.00", &hour, Datum::int(420042));
-        test_timestamp_and_tz_transform("1969-12-31 23:00:00.00", &hour, Datum::int(-1));
-        test_timestamp_and_tz_transform("0022-05-01 22:01:01.00", &hour, Datum::int(-17072905));
+        test_timestamp_and_tz_transform("2017-12-01T18:00:00.000000", &hour, Datum::int(420042));
+        test_timestamp_and_tz_transform("1970-01-01T22:01:01.000000", &hour, Datum::int(22));
+        test_timestamp_and_tz_transform("1969-12-31T23:00:00.000000", &hour, Datum::int(-1));
+        test_timestamp_and_tz_transform("1969-12-31T22:01:01.000000", &hour, Datum::int(-2));
+        test_timestamp_and_tz_transform("0022-05-01T22:01:01.000000", &hour, Datum::int(-17072906));

Review Comment:
   `-17_072_905.98` but rounded up 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Fix hour transform [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1146:
URL: https://github.com/apache/iceberg-rust/pull/1146#discussion_r2020345739


##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -337,12 +337,12 @@ pub struct Hour;
 impl Hour {
     #[inline]
     fn hour_timestamp_micro(v: i64) -> i32 {
-        (v / MICROSECONDS_PER_HOUR) as i32
+        (v as f64 / MICROSECONDS_PER_HOUR).floor() as i32

Review Comment:
   Using unstable api means we need to use unstable rust sdk, what's worse it requires all of our downstream users to use unstable rust sdk, which is anti pattern for a rust library.
   Unstable rust sdk contains experimental apis, but there is no guarantee that those apis will be still there when a stable version is released.
   See [doc about rust channels](https://rust-lang.github.io/rustup/concepts/channels.html)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Fix hour transform [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1146:
URL: https://github.com/apache/iceberg-rust/pull/1146#discussion_r2018832174


##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -337,12 +337,12 @@ pub struct Hour;
 impl Hour {
     #[inline]
     fn hour_timestamp_micro(v: i64) -> i32 {
-        (v / MICROSECONDS_PER_HOUR) as i32
+        (v as f64 / MICROSECONDS_PER_HOUR).floor() as i32
     }
 
     #[inline]
     fn hour_timestamp_nano(v: i64) -> i32 {
-        (v / NANOSECONDS_PER_HOUR) as i32
+        (v as f64 / NANOSECONDS_PER_HOUR).floor() as i32

Review Comment:
   Ditto.



##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -337,12 +337,12 @@ pub struct Hour;
 impl Hour {
     #[inline]
     fn hour_timestamp_micro(v: i64) -> i32 {
-        (v / MICROSECONDS_PER_HOUR) as i32
+        (v as f64 / MICROSECONDS_PER_HOUR).floor() as i32

Review Comment:
   ```suggestion
          (v.div_euclid(MICROSECONDS_PER_HOUR)) as i32
   ```
   There is a [`div_euclid`](https://doc.rust-lang.org/std/primitive.i64.html#method.div_euclid) which could help use to do this. The [`div_floor`](https://doc.rust-lang.org/std/primitive.i64.html#method.div_floor) method seems better, but it's unstable. Give `MICROSECONDS_PER_HOUR` is always positive, div_euclid is good enough here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Fix hour transform [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 merged PR #1146:
URL: https://github.com/apache/iceberg-rust/pull/1146


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org