You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/06/16 04:09:28 UTC
[GitHub] [iceberg] chenjunjiedada opened a new issue #1116: Master build failed on local
chenjunjiedada opened a new issue #1116:
URL: https://github.com/apache/iceberg/issues/1116
The current master failed to build on my local machine while it passes the CI. I found it breaks after commit 260bfa0a33aebc25d07e9a3add46fb7ba0eeca43 gets in.
>
Task :iceberg-data:test org.apache.iceberg.data.TestLocalScan > testFilterWithDateAndTimestamp[1] FAILED java.lang.AssertionError at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.iceberg.data.TestLocalScan.testFilterWithDateAndTimestamp(TestLocalScan.java:486)
org.apache.iceberg.orc.TestOrcMetrics > testMetricsForTopLevelFields FAILED java.lang.AssertionError: expected:<1500> but was:<1499> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:834) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at org.apache.iceberg.TestMetrics.assertBounds(TestMetrics.java:452) at org.apache.iceberg.orc.TestOrcMetrics.assertBounds(TestOrcMetrics.java:107)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] chenjunjiedada commented on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645802976
@edgarRd, so would you add it to the unsupported list? Or do we have some way to fix this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] chenjunjiedada commented on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645132876
This should be duplicated to #1113. @shardulm94 @edgarRd FYI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] shardulm94 commented on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
shardulm94 commented on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645870356
@edgarRd Yes, that should be okay. I am not sure though if it is reasonable to expect ORC to function with multiple timezones within the same thread. So wondering if we should file an issue against ORC.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
edgarRd commented on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645851936
@shardulm94 right, I did notice that the tests pass individually. I think this seems to point to an ORC bug failing to update a TimeZone.
To make the tests pass locally from master for folks in other time zones I can add the DATE to unsupported for now and push my PR if folks are okay with those fixes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
edgarRd commented on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645643279
Thanks @chenjunjiedada, I was able to reproduce.
I've been looking into this and I think https://github.com/apache/iceberg/commit/bca6380c56469c01f4eb02cb91f905dd57ad03d3 seems to fix both test cases as they are.
However, I added a test case https://github.com/apache/iceberg/commit/e61cc6187c6a13b7913f66f548f4be9d05d2a717 for multiple timezones on `TestLocalScan testFilterWithDateAndTimestamp` and there seems to be something wrong going on:
1. Test runs fine for `America/Los_Angeles` passes
2. Test runs fine for `Europe/London` fails
```
org.apache.iceberg.data.TestLocalScan > testFilterWithDateAndTimestamp[1] FAILED
java.lang.AssertionError: TZ: Europe/London
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.apache.iceberg.data.TestLocalScan.testFilterWithDateAndTimestamp(TestLocalScan.java:498)
```
Values:
```
-- 1. America/Los_Angeles run
Date min: -136246
Date max: 134321
-- 1. Europe/London run (fail)
Date min: -136246
Date max: 134321
Record(1986-01-30T08:11:32.189299Z, 1995-05-15T23:02:42.319212, 1596-12-21, 00:29:18.577901)
```
If I switch the order of the tests, the second TZ test keeps failing (different timezone as the failed test before):
```
org.apache.iceberg.data.TestLocalScan > testFilterWithDateAndTimestamp[1] FAILED
java.lang.AssertionError: TZ: America/Los_Angeles
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.apache.iceberg.data.TestLocalScan.testFilterWithDateAndTimestamp(TestLocalScan.java:493)
```
Values:
```
1. Europe/London run
Date min: -136246
Date max: 134321
2. America/Los_Angeles run (fail)
Date min: -136247
Date max: 134320
Record(1936-02-12T14:24:11.970343Z, 1962-08-02T07:28:37.107299, 2337-10-05, 00:15:29.965042)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
edgarRd commented on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-646280830
@shardulm94 I think the behavior should be the same regardless of the timezone and thread state running - like parquet and avro behave in the tests - hence I do think this should probably be a bug as it leads to unexpected results and would pretty much prevent us from having tests on DATE for multiple timezones.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd edited a comment on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
edgarRd edited a comment on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645643279
Thanks @chenjunjiedada, I was able to reproduce.
I've been looking into this and I think https://github.com/apache/iceberg/commit/bca6380c56469c01f4eb02cb91f905dd57ad03d3 seems to fix both test cases as they are.
However, I added a test case https://github.com/apache/iceberg/commit/e61cc6187c6a13b7913f66f548f4be9d05d2a717 for multiple timezones on `TestLocalScan testFilterWithDateAndTimestamp` and there seems to be something wrong going on:
1. Test runs fine for `America/Los_Angeles` passes
2. Test runs fine for `Europe/London` fails
```
org.apache.iceberg.data.TestLocalScan > testFilterWithDateAndTimestamp[1] FAILED
java.lang.AssertionError: TZ: Europe/London
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.apache.iceberg.data.TestLocalScan.testFilterWithDateAndTimestamp(TestLocalScan.java:498)
```
Values:
```
-- 1. America/Los_Angeles run
Date min: -136246
Date max: 134321
-- 2. Europe/London run (fail)
Date min: -136246
Date max: 134321
Record(1986-01-30T08:11:32.189299Z, 1995-05-15T23:02:42.319212, 1596-12-21, 00:29:18.577901)
```
If I switch the order of the tests, the second TZ test keeps failing (different timezone as the failed test before):
```
org.apache.iceberg.data.TestLocalScan > testFilterWithDateAndTimestamp[1] FAILED
java.lang.AssertionError: TZ: America/Los_Angeles
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.apache.iceberg.data.TestLocalScan.testFilterWithDateAndTimestamp(TestLocalScan.java:493)
```
Values:
```
1. Europe/London run
Date min: -136246
Date max: 134321
2. America/Los_Angeles run (fail)
Date min: -136247
Date max: 134320
Record(1936-02-12T14:24:11.970343Z, 1962-08-02T07:28:37.107299, 2337-10-05, 00:15:29.965042)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
edgarRd commented on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645681102
I think the failures in my test above are coming from the residual filters, after adding `TypeID.DATE` in https://github.com/apache/iceberg/blob/master/orc/src/main/java/org/apache/iceberg/orc/ExpressionToSearchArgument.java#L62 the tests pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
edgarRd commented on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-652817094
Can we close this issue? Fix has been merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] chenjunjiedada closed issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
chenjunjiedada closed issue #1116:
URL: https://github.com/apache/iceberg/issues/1116
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd edited a comment on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
edgarRd edited a comment on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645208019
I'm able to reproduce the failure on `org.apache.iceberg.data.TestLocalScan testFilterWithDateAndTimestamp[1]` - not the one in `org.apache.iceberg.orc.TestOrcMetrics > testMetricsForTopLevelFields` but I assume the latter is result a change decreasing min by `1` in date lower bounds, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] shardulm94 commented on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
shardulm94 commented on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645818922
One interesting thing to note is that in the tests added by @edgarRd, if we run the test for each timezone individually, they work just fine. Only when the tests are run one after the other does it fail. I haven't debugged this deeply, but my best guess is that ORC uses ThreadLocals to store the local timezone (I have seen these). And these ThreadLocals are not getting reset across tests which causes subsequent tests to reuse the timezones of previous tests. I tried to run the test for each timezone in a different thread within the for loop and the tests pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] shardulm94 edited a comment on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
shardulm94 edited a comment on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645818922
One interesting thing to note is that in the tests added by @edgarRd, if we run the test for each timezone individually, they work just fine. Only when the tests are run one after the other does it fail. I haven't debugged this deeply, but my best guess is that ORC uses ThreadLocals to store the local timezone (https://github.com/apache/hive/blob/204d8525bf48bf85d15feb427e6f5a31ec192bac/storage-api/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java#L48). And these ThreadLocals are not getting reset across tests which causes subsequent tests to reuse the timezones of previous tests. I tried to run the test for each timezone in a different thread within the for loop and the tests pass. So I don't think there is a bug in the conversion logic from Expression to ORC SearchArgument, but rather in how ORC chooses to maintain the current timezone as a thread local.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] chenjunjiedada commented on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645460307
@edgarRd , I guess you could reproduce this by setting time zone to `Asia/shanghai`, in ubuntu you could use the following command:
`sudo timedatectl set-timezone Asia/Shanghai`
The following statement return 1499.
```java
min = Optional.ofNullable(((DateColumnStatistics) columnStats).getMinimum())
.map(minStats -> DateTimeUtil.daysFromDate(
DateTimeUtil.EPOCH.plus(minStats.getTime(), ChronoUnit.MILLIS).toLocalDate()))
```
The workaround for ORC seems depends on the time zone.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] shardulm94 edited a comment on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
shardulm94 edited a comment on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645818922
One interesting thing to note is that in the tests added by @edgarRd, if we run the test for each timezone individually, they work just fine. Only when the tests are run one after the other does it fail. I haven't debugged this deeply, but my best guess is that ORC uses ThreadLocals to store the local timezone (I have seen these). And these ThreadLocals are not getting reset across tests which causes subsequent tests to reuse the timezones of previous tests. I tried to run the test for each timezone in a different thread within the for loop and the tests pass. So I don't think there is a bug in the conversion logic from Expression to ORC SearchArgument, but rather in how ORC chooses to maintain the current timezone as a thread local.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd edited a comment on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
edgarRd edited a comment on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645208019
I'm able to reproduce the failure on `org.apache.iceberg.data.TestLocalScan testFilterWithDateAndTimestamp[1]` - not the one in `org.apache.iceberg.orc.TestOrcMetrics > testMetricsForTopLevelFields` but I assume the latter is result a change decreasing lowerBound by `1` in date lower bounds, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd edited a comment on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
edgarRd edited a comment on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645681102
I think the failures in my test above are coming from the residual filters on pushdown predicates, after adding `TypeID.DATE` in https://github.com/apache/iceberg/blob/master/orc/src/main/java/org/apache/iceberg/orc/ExpressionToSearchArgument.java#L62 the tests pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on issue #1116: Master build failed on local
Posted by GitBox <gi...@apache.org>.
edgarRd commented on issue #1116:
URL: https://github.com/apache/iceberg/issues/1116#issuecomment-645208019
I'm able to reproduce the failure on `org.apache.iceberg.data.TestLocalScan testFilterWithDateAndTimestamp[1]` - not the one in `org.apache.iceberg.orc.TestOrcMetrics > testMetricsForTopLevelFields` but I assume the latter is result a change decreasing min by `-1` in date lower bounds, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org