You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/09/04 00:19:30 UTC
[GitHub] [spark] williamhyun opened a new pull request, #37787: [SPARK-40323][BUILD] Update ORC to 1.8.0
williamhyun opened a new pull request, #37787:
URL: https://github.com/apache/spark/pull/37787
### What changes were proposed in this pull request?
This PR aims to update ORC to 1.8.0.
### Why are the changes needed?
This will bring the latest changes and bug fixes.
- https://github.com/apache/orc/releases/tag/v1.8.0
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #37787: [SPARK-40323][BUILD] Update ORC to 1.8.0
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #37787:
URL: https://github.com/apache/spark/pull/37787#issuecomment-1424173443
I checked the file inside `threeten-extra-1.7.1.jar`, and only `module-info.class` has the issue, other classes and loaded by JDK8 correctly. Thus IMO it's not a big problem.
```
➜ tmp javap module-info.class
Error: error while reading constant pool for module-info.class: unexpected tag at #5: 19
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #37787: [SPARK-40323][BUILD] Update ORC to 1.8.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #37787:
URL: https://github.com/apache/spark/pull/37787#issuecomment-1255850265
After this upgrade, do we still need protobuf 2.5.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #37787: [SPARK-40323][BUILD] Update ORC to 1.8.0
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #37787:
URL: https://github.com/apache/spark/pull/37787#issuecomment-1256104538
> After this upgrade, do we still need protobuf 2.5.0?
May I ask who is `we`? For native ORC reader, it's not required in Apache Spark. And, this is a part of Apache Spark community effort which is trying to reduce the dependency of protobuf 2.5.0. However, removing protobuf 2.5.0 from Apache Spark distribution is a different issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #37787: [SPARK-40323][BUILD] Update ORC to 1.8.0
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #37787:
URL: https://github.com/apache/spark/pull/37787#issuecomment-1424550534
Thank you for checking, @pan3793 . You are welcome to make an Apache ORC JIRA or PR to upgrade `threeten-extra` if needed. I can merge it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #37787: [SPARK-40323][BUILD] Update ORC to 1.8.0
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #37787:
URL: https://github.com/apache/spark/pull/37787#issuecomment-1423882497
This upgrade brings `threeten-extra:1.7.1`, which has a known issue with JDK 8 bytecode compatibility, see https://github.com/ThreeTen/threeten-extra/issues/226
As a downstream project of Spark, Kyuubi found the same issue https://github.com/apache/kyuubi/actions/runs/4130823342/jobs/7137880315, but I'm not sure if it's a real issue or just does not matter.
I suppose it's not a big deal since all Spark ORC tests passed w/ JDK8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #37787: [SPARK-40323][BUILD] Update ORC to 1.8.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #37787:
URL: https://github.com/apache/spark/pull/37787#issuecomment-1256123496
> May I ask who is we?
I mean the Spark project/community.
> And, this is a part of Apache Spark community effort which is trying to reduce the dependency of protobuf 2.5.0.
That's why I ask it in the Spark community.
> However, removing protobuf 2.5.0 from Apache Spark distribution is a different issue.
I agree, but it's related to this issue, because I see "[ORC-1212](https://issues.apache.org/jira/browse/ORC-1212) Upgrade protobuf-java to 3.17.3" in the ORC 1.8.0 release note. As far as I know, Spark requires protobuf-java 2.5.0 for Hadoop2 and ORC 1.7 and earlier versions, as Spark already switched to Hadoop Shaded client for Hadoop3, I think maybe Spark can remove protobuf 2.5.0 from the binary release(only for hadoop3 profile).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #37787: [SPARK-40323][BUILD] Update ORC to 1.8.0
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #37787:
URL: https://github.com/apache/spark/pull/37787#issuecomment-1256306854
Yes, of course, it's technically feasible to try in Hadoop 3 distribution and we will eventually at some releases, @pan3793 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #37787: [SPARK-40323][BUILD] Update ORC to 1.8.0
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #37787: [SPARK-40323][BUILD] Update ORC to 1.8.0
URL: https://github.com/apache/spark/pull/37787
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org