You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "islamismailov (via GitHub)" <gi...@apache.org> on 2023/04/09 04:42:48 UTC
[GitHub] [iceberg] islamismailov opened a new issue, #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
islamismailov opened a new issue, #7303:
URL: https://github.com/apache/iceberg/issues/7303
### Apache Iceberg version
1.2.0 (latest release)
### Query engine
None
### Please describe the bug 🐞
I see this test fail on some machines we have in production. I wasn't able to narrow down why but I ssh'ed into that host and was able to git bisect to this commit which essentially upgrades Apache ORC library version: `9b373b1541e7ec5f187fb775477dba2e3b8d11bc`. Might be related to timestamp column and timezone settings on a particular host. Perhaps @williamhyun might have an idea. `1.8.1` passes the test, and both `1.8.2` and `1.8.3` are broken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dongjoon-hyun commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1502051086
Could you share some detail info for that machine, @islamismailov ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1502034064
cc @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1857049772
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dongjoon-hyun commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1502455716
Thank you for your confirmation, @islamismailov .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dongjoon-hyun commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1502041686
Thank you for pinging me, @Fokko .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dongjoon-hyun commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1503830869
I cannot reproduce it locally. I used Java 8 on Apple Silicon Mac.
```
$ sw_vers
ProductName: macOS
ProductVersion: 12.6.4
BuildVersion: 21G526
$ git log -n1
commit fc6bd1e7ab92086611bab87037b9edd17f99ed40 (HEAD -> master, origin/master, origin/HEAD)
Author: Ajantha Bhat <aj...@gmail.com>
Date: Tue Apr 11 21:49:52 2023 +0530
Nessie: Fix testcase failures (#7320)
$ ./gradlew :iceberg-data:test --tests "org.apache.iceberg.data.TestMetricsRowGroupFilterTypes"
BUILD SUCCESSFUL in 2s
23 actionable tasks: 1 executed, 22 up-to-date
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] islamismailov commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "islamismailov (via GitHub)" <gi...@apache.org>.
islamismailov commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1546520928
I have not dug deeper on this. I just reverted ORC library version to `1.8.0` version and left it at that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dongjoon-hyun commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1502171487
May I rephrase your comments?
1. In the PR description, you wrote
> 1.8.1 passes the test, and both 1.8.2 and 1.8.3 are broken.
2. In the previous comment, you wrote
> can you actually check 1.8.0 - I know that version consistently passes. I still see problems with 1.8.1
Which version do you want to report your issues, @islamismailov ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dongjoon-hyun commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1537355230
Do you still see the issues, @islamismailov ? For a record, Apache ORC 1.8.3 is used in Apache Spark, Hive, Arrow community with various environments. Since there is no this kind of report, I'm highly interested in what is going on your environment.
- Apache Spark: https://github.com/apache/spark/blob/master/pom.xml#L145
- Apache Hive: https://github.com/apache/hive/blob/master/pom.xml#L182
- Apache Arrow: https://github.com/apache/arrow/blob/main/java/adapter/orc/pom.xml#L40
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1828847304
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] islamismailov commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "islamismailov (via GitHub)" <gi...@apache.org>.
islamismailov commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1502647738
This is the command to run the test `./gradlew :iceberg-data:test --tests "org.apache.iceberg.data.TestMetricsRowGroupFilterTypes"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] islamismailov commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "islamismailov (via GitHub)" <gi...@apache.org>.
islamismailov commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1546521179
It fails on certain machines some of the time, so it's hard to catch. I have a feeling that it's related to timestamp column handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
URL: https://github.com/apache/iceberg/issues/7303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dongjoon-hyun commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1549934546
Got it. Thank you for the updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] islamismailov commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "islamismailov (via GitHub)" <gi...@apache.org>.
islamismailov commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1502160403
@dongjoon-hyun can you actually check `1.8.0` - I know that version consistently passes. I still see problems with `1.8.1`
I do think it's somehow related to timestamp since issue seems to disappear and re-appear. I found a similar problem that has happened before:
https://github.com/apache/iceberg/issues/1113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dongjoon-hyun commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1502518447
According to Apache ORC 1.8.1 release note, there exists three bug fixes.
https://github.com/apache/orc/releases/tag/v1.8.1
Although I took a look,
1. [ORC-1283](https://issues.apache.org/jira/browse/ORC-1283) ENABLE_INDEXES does not take effect
- This is recovered back at v1.8.2
2. [ORC-1288](https://issues.apache.org/jira/browse/ORC-1288) Invalid memory freeing with ZLIB compression
- This is C++ change which is irrelevant to Java world.
3. [ORC-1291](https://issues.apache.org/jira/browse/ORC-1291) NullPointerException in TypeDescription
- This changes only the hashCode of types.
[As I asked before](https://github.com/apache/iceberg/issues/7303#issuecomment-1502051086), we need more details about your some machines (`I see this test fail on some machines we have in production.`) if you want to investigate more, @islamismailov .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dongjoon-hyun commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1502050628
According to Apache ORC 1.8.2 release note, there is no change about no timestamp column and timezone.
- https://github.com/apache/orc/releases/tag/v1.8.2
The following three are the technical code changes.
- [ORC-1332](https://issues.apache.org/jira/browse/ORC-1332) Avoid NegativeArraySizeException when using searchArgument
- [ORC-1343](https://issues.apache.org/jira/browse/ORC-1343) Disable ENABLE_INDEXES
- [ORC-1338](https://issues.apache.org/jira/browse/ORC-1338) Set bloom filter fpp to 1%
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1503884389
This also does not reproduce on any of our CI's, could you please give more information about the environment in which this test is being run? Is it failing on the same machine every time, or is it variable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] islamismailov commented on issue #7303: TestMetricsRowGroupFilterTypes fails on certain machines, bisects to ORC lib upgrade
Posted by "islamismailov (via GitHub)" <gi...@apache.org>.
islamismailov commented on issue #7303:
URL: https://github.com/apache/iceberg/issues/7303#issuecomment-1502452461
My initial assessment was wrong. The problem still reproduces on `1.8.1`. `1.8.0` works fine though @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org