[GitHub] [iceberg] yyanyy commented on a change in pull request #1981: Fix date and timestamp transforms

[GitBox <gi...@apache.org>]

yyanyy commented on a change in pull request #1981:
URL: https://github.com/apache/iceberg/pull/1981#discussion_r548345460



##########
File path: api/src/main/java/org/apache/iceberg/transforms/Timestamps.java
##########
@@ -52,12 +52,19 @@ public Integer apply(Long timestampMicros) {
       return null;
     }
 
-    // discards fractional seconds, not needed for calculation
-    OffsetDateTime timestamp = Instant
-        .ofEpochSecond(timestampMicros / 1_000_000)
-        .atOffset(ZoneOffset.UTC);
-
-    return (int) granularity.between(EPOCH, timestamp);
+    if (timestampMicros >= 0) {
+      OffsetDateTime timestamp = Instant
+          .ofEpochSecond(Math.floorDiv(timestampMicros, 1_000_000), Math.floorMod(timestampMicros, 1_000_000))
+          .atOffset(ZoneOffset.UTC);
+      return (int) granularity.between(EPOCH, timestamp);
+    } else {
+      // add 1 micro to the value to account for the case where there is exactly 1 unit between the timestamp and epoch
+      // because the result will always be decremented.
+      OffsetDateTime timestamp = Instant
+          .ofEpochSecond(Math.floorDiv(timestampMicros, 1_000_000), Math.floorMod(timestampMicros + 1, 1_000_000))

Review comment:
       I think the second argument in `Instant.ofEpochSecond` is nanosecond instead of microsecond? Although this doesn't change the outcome of this method since we don't accept nanosecond in the first place, and inputing microsecond for nanosecond field just makes the field smaller than it should be, which is already enough to address the case when `Math.floorMod()` == 0

##########
File path: api/src/test/java/org/apache/iceberg/transforms/TestDatesProjection.java
##########
@@ -142,6 +161,24 @@ public void testMonthStrictUpperBound() {
     assertProjectionStrictValue(spec, in("date", anotherDate, date), Expression.Operation.FALSE);
   }
 
+  @Test
+  public void testNegativeMonthStrictUpperBound() {
+    Integer date = (Integer) Literal.of("1969-12-31").to(TYPE).value();
+    PartitionSpec spec = PartitionSpec.builderFor(SCHEMA).month("date").build();
+
+    assertProjectionStrictValue(spec, lessThan("date", date), Expression.Operation.FALSE);
+    assertProjectionStrictValue(spec, lessThanOrEqual("date", date), Expression.Operation.FALSE);
+    assertProjectionStrict(spec, greaterThan("date", date), Expression.Operation.GT, "1969-12");
+    assertProjectionStrict(spec, greaterThanOrEqual("date", date), Expression.Operation.GT, "1969-12");
+    assertProjectionStrict(spec, notEqual("date", date), Expression.Operation.NOT_IN, "[1969-12, 1970-01]");
+    assertProjectionStrictValue(spec, equal("date", date), Expression.Operation.FALSE);
+
+    Integer anotherDate = (Integer) Literal.of("1970-01-01").to(TYPE).value();
+    assertProjectionStrict(spec, notIn("date", date, anotherDate),
+        Expression.Operation.NOT_IN, "[1969-12, 1970-01]");

Review comment:
       I think this is testing the same thing for `NOT_IN` in `testNegativeMonthStrictLowerBound`, do we want to change 1970-01-01 to an earlier time? 

##########
File path: api/src/main/java/org/apache/iceberg/transforms/ProjectionUtil.java
##########
@@ -254,4 +257,124 @@ private ProjectionUtil() {
     return predicate(predicate.op(), fieldName,
         Iterables.transform(predicate.asSetPredicate().literalSet(), transform::apply));
   }
+
+  static UnboundPredicate<Integer> fixInclusiveTimeProjection(UnboundPredicate<Integer> projected) {
+    if (projected == null) {
+      return projected;
+    }
+
+    // adjust the predicate for values that were 1 larger than the correct transformed value

Review comment:
       I think the mention of "correct" and "incorrect" in these comments might not be obvious to understand outside the context of this PR, as the reader has no prior knowledge of the issue we are fixing here from looking at the code base itself. Do we want to add some more explanation/link to the issue? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org