You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/06 23:12:58 UTC
[GitHub] [spark] sunchao opened a new pull request #30284: Remove shares Hadoop classes
sunchao opened a new pull request #30284:
URL: https://github.com/apache/spark/pull/30284
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This removes the `sharesHadoopClasses` flag from `IsolatedClientLoader` in Hive module.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Currently, when initializing `IsolatedClientLoader`, users can set the `sharesHadoopClasses` flag to decide whether the `HiveClient` created should share Hadoop classes with Spark itself or not. In the latter case, the client will only load Hadoop classes from the Hive dependencies.
There are two reasons to remove this:
1. this feature is currently used in two cases: 1) unit tests, 2) when the Hadoop version defined in Maven can not be found when `spark.sql.hive.metastore.jars` is equal to "maven", which could be very rare.
2. when `sharesHadoopClasses` is false, Spark doesn't really only use Hadoop classes from Hive jars: we also download `hadoop-client` jar and put all the sub-module jars (e.g., `hadoop-common`, `hadoop-hdfs`) together with the Hive jars, and the Hadoop version used by `hadoop-client` is the same version used by Spark itself. As result, we're mixing two versions of Hadoop jars in the classpath, which could potentially cause issues, especially considering that the default Hadoop version is already 3.2.0 while most Hive versions supported by the `IsolatedClientLoader` is still using Hadoop 2.x or even lower.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
This affects Spark users in one scenario: when `spark.sql.hive.metastore.jars` is set to `maven` AND the Hadoop version specified in pom file cannot be downloaded, currently the behavior is to switch to _not_ share Hadoop classes, but with the PR it will share Hadoop classes with Spark.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Existing UTs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724548238
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724673051
**[Test build #130866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130866/testReport)** for PR 30284 at commit [`0b035b2`](https://github.com/apache/spark/commit/0b035b2a9201a17d80582ff40b2b2b9cfff9b412).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724573024
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30284: [SPARK-33376][HIVE] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-723356143
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724420060
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35428/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724759717
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724573039
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130851/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724602505
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35459/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #30284: [SPARK-33376][HIVE] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-723482402
cc @cloud-fan @gatorsmile @srowen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724461518
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130819/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724534804
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130837/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724843637
So we want to change from "fallback to Hadoop 2.7 which may or may not work" to "don't fallback just fail". It's risky as it may break existing workloads, but if it's mainly used in tests, then failing is better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724534422
**[Test build #130837 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130837/testReport)** for PR 30284 at commit [`0b035b2`](https://github.com/apache/spark/commit/0b035b2a9201a17d80582ff40b2b2b9cfff9b412).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30284: [SPARK-33376][HIVE] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-723366281
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724614723
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #30284:
URL: https://github.com/apache/spark/pull/30284#discussion_r520008320
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -75,7 +72,6 @@ private[hive] object IsolatedClientLoader extends Logging {
"again. Hadoop classes will not be shared between Spark and Hive metastore client. " +
Review comment:
Forgot about this place - will fix. Yes agree that an exception might be more appropriate. It seems a bit arbitrary to fall back to 2.7.4 here since the original Hadoop version could vary based on the Hadoop profile (either 2.7.4 or 3.2.0 at the moment).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][HIVE] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-723341437
**[Test build #130740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130740/testReport)** for PR 30284 at commit [`b735947`](https://github.com/apache/spark/commit/b735947d68f18109a09fcd1d1eee048f823651be).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724670474
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724551780
**[Test build #130851 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130851/testReport)** for PR 30284 at commit [`0b035b2`](https://github.com/apache/spark/commit/0b035b2a9201a17d80582ff40b2b2b9cfff9b412).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724461046
**[Test build #130819 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130819/testReport)** for PR 30284 at commit [`0b035b2`](https://github.com/apache/spark/commit/0b035b2a9201a17d80582ff40b2b2b9cfff9b412).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724426644
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724426636
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35428/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724614697
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35459/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724759717
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724506917
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724490715
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35446/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724506899
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35446/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30284: [SPARK-33376][HIVE] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-723356143
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724614723
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724186158
> For example, we can place the the downloaded jars ahead to have a higher precedence in the classpath.
@HyukjinKwon I'm not sure if this can solve the problem. The fundamental issue is we are trying to mix a single Hadoop version (i.e., the one used by Spark, for `hadoop-client` jar) with various others from the different Hive versions supported, so it is bound to to load some classes from the former while some others from the latter.
To truly solve the issue and make `sharesHadoopClasses` stick to its meaning, IMO we'd have to pick a matching Hadoop version that is used by the specific Hive version. For instance, if the Hive version is 2.3.7, we should use Hadoop 2.7.2, and if Hive version is 2.2.0, we should pick Hadoop 2.6.0 instead. However I'm not sure if this is something we should do given that some of the Hadoop version used by Hive is really old, and also this is a rarely used feature like I mentioned in the PR description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724811709
I'm concerned about the comment
```
// If the error message contains hadoop, it is probably because the hadoop
// version cannot be resolved.
```
If we can't precisely know if the requested Hadoop version could not be download, throwing exception seems risky.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724816475
While I do agree that string-matching on `"hadoop"` is a bit sketchy, I don't think I agree with your conclusion about risk -- currently we only catch the `RuntimeException` if it contains `"hadoop"`, else we allow the exception to bubble up. Adding an exception throw at line 74 instead of calling `downloadVersion` again just aligns the behavior of "missing Hadoop JARs" with the behavior of a non-Hadoop exception. The current logic is more risky in my opinion, bringing unpredictable behavior. Given that this feature is mainly used in unit tests, it brings an increased degree of non-determinism to the unit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][HIVE] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-723351380
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35349/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724506917
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724708844
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35474/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30284:
URL: https://github.com/apache/spark/pull/30284#discussion_r519663083
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -75,7 +72,6 @@ private[hive] object IsolatedClientLoader extends Logging {
"again. Hadoop classes will not be shared between Spark and Hive metastore client. " +
Review comment:
Shall we update this message since we now always share? Actually we should think about throwing an exception that says all related metastore jars have to be specified instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724721763
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][HIVE] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-723365955
**[Test build #130740 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130740/testReport)** for PR 30284 at commit [`b735947`](https://github.com/apache/spark/commit/b735947d68f18109a09fcd1d1eee048f823651be).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724462619
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724673051
**[Test build #130866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130866/testReport)** for PR 30284 at commit [`0b035b2`](https://github.com/apache/spark/commit/0b035b2a9201a17d80582ff40b2b2b9cfff9b412).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-723919359
@sunchao, I think the PR description explains why the current `sharesHadoopClasses=false` works in a corner case, and why it might not currently work but I think that doesn't mean that we should drop this. For example, we can place the the downloaded jars ahead to have a higher precedence in the classpath.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724721763
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724476909
**[Test build #130837 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130837/testReport)** for PR 30284 at commit [`0b035b2`](https://github.com/apache/spark/commit/0b035b2a9201a17d80582ff40b2b2b9cfff9b412).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724721744
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35474/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724461510
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724404232
**[Test build #130819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130819/testReport)** for PR 30284 at commit [`0b035b2`](https://github.com/apache/spark/commit/0b035b2a9201a17d80582ff40b2b2b9cfff9b412).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724758233
**[Test build #130866 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130866/testReport)** for PR 30284 at commit [`0b035b2`](https://github.com/apache/spark/commit/0b035b2a9201a17d80582ff40b2b2b9cfff9b412).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724402866
Thanks, addressed your comment @HyukjinKwon . I agree that we should think about not using the fallback version and instead throw exception if the specified Hadoop version cannot be downloaded. @cloud-fan let us know what you think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724476909
**[Test build #130837 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130837/testReport)** for PR 30284 at commit [`0b035b2`](https://github.com/apache/spark/commit/0b035b2a9201a17d80582ff40b2b2b9cfff9b412).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724461510
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][HIVE] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-723356135
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35349/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724784046
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724572704
**[Test build #130851 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130851/testReport)** for PR 30284 at commit [`0b035b2`](https://github.com/apache/spark/commit/0b035b2a9201a17d80582ff40b2b2b9cfff9b412).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #30284:
URL: https://github.com/apache/spark/pull/30284
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724551780
**[Test build #130851 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130851/testReport)** for PR 30284 at commit [`0b035b2`](https://github.com/apache/spark/commit/0b035b2a9201a17d80582ff40b2b2b9cfff9b412).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30284: [SPARK-33376][HIVE] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-723366281
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30284: [SPARK-33376][HIVE] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-723341437
**[Test build #130740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130740/testReport)** for PR 30284 at commit [`b735947`](https://github.com/apache/spark/commit/b735947d68f18109a09fcd1d1eee048f823651be).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724867746
I tend to agree with @xkrogen that fail-fast is better in this case, and of course we'd have to document this in migration guide so Spark users will be informed properly.
This is not only used by UT: Spark users can set `spark.sql.hive.metastore.jars` to `maven` and if they happen to rely on a custom Hadoop version that is not downloadable, this code path will be triggered. I think this is a pretty rare case though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724807786
I see this got merged but I think the last discussion indicated that we would throw an exception if the requested Hadoop version could not be download, instead of falling back to an arbitrarily decided 2.7.4 version. @sunchao should we address this in a follow-on?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724404232
**[Test build #130819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130819/testReport)** for PR 30284 at commit [`0b035b2`](https://github.com/apache/spark/commit/0b035b2a9201a17d80582ff40b2b2b9cfff9b412).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724573024
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724426644
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724534799
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30284: [SPARK-33376][SQL] Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30284:
URL: https://github.com/apache/spark/pull/30284#issuecomment-724534799
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org