You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/05/27 03:50:05 UTC

[GitHub] [incubator-iceberg] shardulm94 opened a new pull request #1069: ORC: Disable predicate pushdown for timestamp type

shardulm94 opened a new pull request #1069:
URL: https://github.com/apache/incubator-iceberg/pull/1069


   In https://github.com/apache/incubator-iceberg/pull/983#issuecomment-633808288 we discovered issues with ORC predicate pushdown for timestamp types, where timestamps less than epoch were not handled correctly.
   https://github.com/apache/incubator-iceberg/commit/ed91355d5559858cd665560f92c6d2dfdc4a6885 fixes this and updates the test case to test for both pre epoch and post epoch values
   
   After fixing this, I also found that ORC column stats for timestamp are only stored to millisecond precisions (with round down) and hence predicates on timestamp types with sub-millisecond literals can potentially be evaluated incorrectly. This is being fixed in https://issues.apache.org/jira/browse/ORC-611. So for the time being, I have disabled pushdown for timestamp types in  https://github.com/apache/incubator-iceberg/commit/63bc62b0707f9fea220759b12a8a544aa63eb59e


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [incubator-iceberg] rdblue merged pull request #1069: ORC: Disable predicate pushdown for timestamp type

Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #1069:
URL: https://github.com/apache/incubator-iceberg/pull/1069


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [incubator-iceberg] shardulm94 commented on pull request #1069: ORC: Disable predicate pushdown for timestamp type

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on pull request #1069:
URL: https://github.com/apache/incubator-iceberg/pull/1069#issuecomment-634327409


   cc @rdblue @chenjunjiedada 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #1069: ORC: Disable predicate pushdown for timestamp type

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1069:
URL: https://github.com/apache/incubator-iceberg/pull/1069#discussion_r430764489



##########
File path: orc/src/main/java/org/apache/iceberg/orc/ExpressionToSearchArgument.java
##########
@@ -254,8 +255,10 @@ public Action or(Action leftChild, Action rightChild) {
         return Date.valueOf(LocalDate.ofEpochDay((Integer) icebergLiteral));
       case TIMESTAMP:
         long microsFromEpoch = (Long) icebergLiteral;
-        return Timestamp.from(Instant.ofEpochSecond(Math.floorDiv(microsFromEpoch, 1_000_000),
-            (microsFromEpoch % 1_000_000) * 1_000));
+        return Timestamp.from(Instant.ofEpochSecond(
+            Math.floorDiv(microsFromEpoch, 1_000_000),
+            Math.floorMod(microsFromEpoch, 1_000_000) * 1_000

Review comment:
       Why did this change?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [incubator-iceberg] shardulm94 commented on a change in pull request #1069: ORC: Disable predicate pushdown for timestamp type

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1069:
URL: https://github.com/apache/incubator-iceberg/pull/1069#discussion_r430767382



##########
File path: orc/src/main/java/org/apache/iceberg/orc/ExpressionToSearchArgument.java
##########
@@ -254,8 +255,10 @@ public Action or(Action leftChild, Action rightChild) {
         return Date.valueOf(LocalDate.ofEpochDay((Integer) icebergLiteral));
       case TIMESTAMP:
         long microsFromEpoch = (Long) icebergLiteral;
-        return Timestamp.from(Instant.ofEpochSecond(Math.floorDiv(microsFromEpoch, 1_000_000),
-            (microsFromEpoch % 1_000_000) * 1_000));
+        return Timestamp.from(Instant.ofEpochSecond(
+            Math.floorDiv(microsFromEpoch, 1_000_000),
+            Math.floorMod(microsFromEpoch, 1_000_000) * 1_000

Review comment:
       The previous logic did not correctly handle pre-epoch timestamps and resulted in off by one second error which was the primary cause of failures in #983 (`%` returns negative values if sign(a)!=sign(b) so it interfered with floorDiv which also has handling for the same). After I fixed this issue, I ran into ORC-611,




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org