You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/30 19:02:41 UTC
[GitHub] [spark] sunchao opened a new pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
sunchao opened a new pull request #30556:
URL: https://github.com/apache/spark/pull/30556
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This creates a `hadoop-aws-shaded` jar within `hadoop-cloud` module, which shades Guava and relocate them to the `org.apache.hadoop.shaded` namespace used by the Hadoop side.
A large portion of the change involves moving `hadoop-aws`, `hadoop-azure` and `hadoop-openstack` under the new `hadoop-2.7` profile. This is to make sure they are not included when the default `hadoop-3.2` profile is active, since otherwise we'd have both `hadoop-aws` and `hadoop-aws-shaded` after build.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Due to [HADOOP-15387](https://issues.apache.org/jira/browse/HADOOP-15387), `hadoop-aws` module currently doesn't work with shaded Hadoop clients, namely `hadoop-client-api` and `hadoop-client-runtime`, because the former references some private API from `hadoop-common` which uses Guava and is shaded in `hadoop-client-api`. Consequently, when talking to S3 Spark users may encounter the following error:
```
: java.lang.NoSuchMethodError: 'void org.apache.hadoop.util.SemaphoredDelegatingExecutor.<init>(com.google.common.util.concurrent.ListeningExecutorService, int, boolean)'
at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:772)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1118)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1098)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:987)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:975)
at org.apache.spark.deploy.SparkHadoopUtil$.createFile(SparkHadoopUtil.scala:510)
```
This PR mitigate the issue by shading `hadoop-aws` within Spark itself, and replace the existing `hadoop-aws` jar with `hadoop-aws-shaded` jar. This, however, shall be treated as a temporary fix and should eventually be replaced by the shaded jar from Hadoop side, once [HADOOP-15387](https://issues.apache.org/jira/browse/HADOOP-15387) is resolved.
**Also note that, with this PR, `hadoop-aws-shaded` jar becomes a hard dependency, and shall be included even if `hadoop-provided` is specified by Spark users.**
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes, the newly introduced `hadoop-aws-shaded` jar becomes a hard dependency, and shall be included even if `hadoop-provided` is specified by Spark users.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
I manually checked the bytecode of the generated `hadoop-aws-shaded` jar and verified that all the Guava references are now relocated .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao closed pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
sunchao closed pull request #30556:
URL: https://github.com/apache/spark/pull/30556
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-736189690
> @AngersZhuuuu yes that should work. On the other hand, if you are building Spark with `hadoop-provided` option on, you can also build your own shaded `hadoop-aws` jar and put it in the classpath.
Thanks for your suggestion. Is there any new developments in Hive 2.3 support for Hadoop-3.3.0? I saw this pr https://github.com/apache/hive/pull/1356 but I am not sure it will make hive run well with hadoop-3.3.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-736852351
Hi, All. It turns out that `master` branch's normal distribution (without `hadoop-cloud`) is also affected.
```
$ bin/spark-shell --conf spark.hadoop.fs.s3a.access.key=$AWS_ACCESS_KEY_ID --conf spark.hadoop.fs.s3a.secret.key=$AWS_SECRET_ACCESS_KEY -c spark.eventLog.enabled=true -c spark.eventLog.dir=s3a://dongjoon/ --packages org.apache.hadoop:hadoop-aws:3.2.0,org.apache.hadoop:hadoop-common:3.2.0
...
java.lang.NoSuchMethodError: 'org.apache.hadoop.conf.Configuration org.apache.hadoop.security.ProviderUtils.excludeIncompatibleCredentialProviders(org.apache.hadoop.conf.Configuration, java.lang.Class)'
at org.apache.hadoop.fs.s3a.S3AUtils.getAWSAccessKeys(S3AUtils.java:740)
at org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider.<init>(SimpleAWSCredentialsProvider.java:58)
at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:600)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:257)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1853)
at org.apache.spark.deploy.history.EventLogFileWriter.<init>(EventLogFileWriters.scala:60)
at org.apache.spark.deploy.history.SingleEventLogFileWriter.<init>(EventLogFileWriters.scala:213)
at org.apache.spark.deploy.history.EventLogFileWriter$.apply(EventLogFileWriters.scala:181)
at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:64)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:588)
```
Up to 3.0.1, the following was enough.
```
$ bin/spark-shell --conf spark.hadoop.fs.s3a.access.key=$AWS_ACCESS_KEY_ID --conf spark.hadoop.fs.s3a.secret.key=$AWS_SECRET_ACCESS_KEY -c spark.eventLog.enabled=true -c spark.eventLog.dir=s3a://dongjoon/spark-events/ --packages org.apache.hadoop:hadoop-aws:3.2.0
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-736547560
I am perfectly happy to have the -shaded stuff in hadoop-common, as that stops the ASF getting upset about the spark project publishing o.a.hadoop artifacts (see : org.apache.hive artifacts). Other people would be happy too. Get that in to hadoop branches 3.2 and 3.3 and you can pick it up there.
we would also want a hadoop-cloud-storage-shaded which pulled in the relevant artifacts from the other modules
Now, one troublespot will be hadoop-common dependencies, especially the fs implementation stuff. I've been trying with all recent work to put them in .impl packages so that they can be isolated for java 9 modules; there's inevitably stuff in org.apache.hadoop.fs; the committers need hadoop-mapreduce-client. Suggestions?
Anyway, I'd rather a move to java 9 modules over shading, which is a workaround designed to hide CVEs in larger JARs. We just have to make do with that now, but it doesn't mean we need to stay with it.
so: serious question. What would it take to make hadoop-* modular? As well as the module-info files, we'd need to understand what is public and what isn't...and how to deal with
* mixed public/private packages
* stuff we thought was private but turns out gets needed by apps
* stuff needed by extension points
thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-737089192
Sure @dongjoon-hyun . This sounds good to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-736816385
@steveloughran thanks for your comments. What is your opinion on this PR though? this allows us to move forward in parallel while waiting for a proper fix from Hadoop side (which I can also help with). It seems that Spark doesn't officially publish any jars from `hadoop-cloud` module though.
Personally I feel the Java 9 modules feature is promising but on the other hand it seems like a radical change, given a lot of emphasis have been around shading in Hadoop. Further, I haven't seen wide adoption of this yet within the "eco-system" and not sure if it will impose chain effects on downstream apps depending on Hadoop.
> so: serious question. What would it take to make hadoop-* modular?
I think this, together with moving to modules, should be properly discussed in a Hadoop JIRA or dev list. I still have lots of homework to do regarding the modules approach, but IMO we'd have to take a close look at downstream projects (at least those within close proximity of Hadoop), and figuring out the current state. From my experience Spark is relatively loosely decoupled with Hadoop but projects such as Hive may prove to be quite different.
Within Hadoop, we should examine what is public, what is "project-private" and what is "module-private", and tighten the APIs so that even different modules within Hadoop can play more nicely with each other.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-736188374
@AngersZhuuuu yes that should work. On the other hand, if you are building Spark with `hadoop-provided` option on, you can also build your own shaded `hadoop-aws` jar and put it in the classpath.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-737029894
Thanks @dongjoon-hyun . This is bad news and it means we'd have to abandon the approach in this PR. The only solution seems have to be on the Hadoop side. I've opened a [Hadoop PR](https://github.com/apache/hadoop/pull/2510) and tested it successfully with the code snippet you pasted above. @steveloughran could you take a look there? thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-736210125
All right, there is still a lot of work to be done to integrate these engine with hadoop.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-736196419
Ah ok. That part, as far as I know, is stuck because Hive has dependency on a old version of Spark which blocks it from upgrading Guava. Hopefully that should be unblocked after [HADOOP-17288](https://issues.apache.org/jira/browse/HADOOP-17288) is shipped in the upcoming release (whether that be 3.3.1 or 3.4.0). I'm not sure whether the change will go to Hive 2.3 branch though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-736161647
For users of hadoop S3A using a custom version with custom code (such as me), they can change hadoop build info and deploy to their own maven repo and change spark dependencies to their own maven repo.
In this way can solve problem of user who will use their own jar, right? But it seems like a lot of work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-736852351
Hi, All. It turns out that `master` branch's normal distribution is also affected.
```
$ bin/spark-shell --conf spark.hadoop.fs.s3a.access.key=$AWS_ACCESS_KEY_ID --conf spark.hadoop.fs.s3a.secret.key=$AWS_SECRET_ACCESS_KEY -c spark.eventLog.enabled=true -c spark.eventLog.dir=s3a://dongjoon/ --packages org.apache.hadoop:hadoop-aws:3.2.0,org.apache.hadoop:hadoop-common:3.2.0
...
java.lang.NoSuchMethodError: 'org.apache.hadoop.conf.Configuration org.apache.hadoop.security.ProviderUtils.excludeIncompatibleCredentialProviders(org.apache.hadoop.conf.Configuration, java.lang.Class)'
at org.apache.hadoop.fs.s3a.S3AUtils.getAWSAccessKeys(S3AUtils.java:740)
at org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider.<init>(SimpleAWSCredentialsProvider.java:58)
at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:600)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:257)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1853)
at org.apache.spark.deploy.history.EventLogFileWriter.<init>(EventLogFileWriters.scala:60)
at org.apache.spark.deploy.history.SingleEventLogFileWriter.<init>(EventLogFileWriters.scala:213)
at org.apache.spark.deploy.history.EventLogFileWriter$.apply(EventLogFileWriters.scala:181)
at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:64)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:588)
```
Up to 3.0.1, the following was enough.
```
$ bin/spark-shell --conf spark.hadoop.fs.s3a.access.key=$AWS_ACCESS_KEY_ID --conf spark.hadoop.fs.s3a.secret.key=$AWS_SECRET_ACCESS_KEY -c spark.eventLog.enabled=true -c spark.eventLog.dir=s3a://dongjoon/spark-events/ --packages org.apache.hadoop:hadoop-aws:3.2.0
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-737040013
Ya. I'll proceed https://github.com/apache/spark/pull/30508 first since Apache Spark 3.1 branch cut is this Friday.
We can revisit this with later during QA period.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-736194591
> @AngersZhuuuu you mean for Spark to work with Hive and Hadoop 3.3.0, right? the major issue is around resolving potential Guava conflicts between these components. Hadoop 3.2.1+/3.3.0+ has moved to Guava 27 while Hive/Spark are still on Guava 14. One of the motivations to move to the shaded client in Spark is to isolate the Guava dependencies on the Hadoop side. Similarly, @viirya is working on the above PR to shade Guava from Hive side.
Not only spark, we need hive can support running with hadoop-3.3.0 too. Seems doesn't look finished yet.
https://issues.apache.org/jira/browse/HIVE-21569
https://issues.apache.org/jira/browse/HIVE-22916
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-736059464
cc @srowen and @steveloughran
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-736193307
@AngersZhuuuu you mean for Spark to work with Hive and Hadoop 3.3.0, right? the major issue is around resolving potential Guava conflicts between these components. Hadoop 3.2.1+/3.3.0+ has moved to Guava 27 while Hive/Spark are still on Guava 14. One of the motivations to move to the shaded client in Spark is to isolate the Guava dependencies on the Hadoop side. Similarly, @viirya is working on the above PR to shade Guava from Hive side.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30556: [WIP][SPARK-33212][BUILD] Provide hadoop-aws-shaded jar in hadoop-cloud module
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30556:
URL: https://github.com/apache/spark/pull/30556#issuecomment-736058470
Thank you, @sunchao .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org