You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/04/27 23:00:38 UTC
[GitHub] [spark] dbtsai opened a new pull request #28376: [SPARK-31582] [Yarn] Able to not populate Hadoop classpath
dbtsai opened a new pull request #28376:
URL: https://github.com/apache/spark/pull/28376
### What changes were proposed in this pull request?
We are adding a new Yarn configuration to not populate hadoop classpath from `yarn.application.classpath` and ``mapreduce.application.classpath`.
### Why are the changes needed?
Spark Yarn client will populate hadoop classpath from `yarn.application.classpath` and `mapreduce.application.classpath`.
However, for Spark with embedded hadoop build, it can cause jar conflicts because spark distribution can contain different version of hadoop jars.
Typically situation is when an user uses an Apache Spark distribution with its-own embedded hadoop, and submits a job to Cloudera or Hortonworks Yarn clusters, because of two different incompatible hadoop jars in the classpath, it can run into errors.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
It's hard to add UTs for this configuration since this requires using different incompatible versions of hadoop. We manually tested this PR, and we are able to submit a Spark job using Spark distribution built with Apache Hadoop 2.10 to CDH 5.6 without populating CDH classpath.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620920044
@dbtsai @adamkennedy Thanks for clarifying that. Making sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621469989
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya edited a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
viirya edited a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620717201
@tgravescs I see. Thanks. That makes sense. So users still need to pay attention on dependencies issue if any when running multiple versions there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621459337
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621462704
**[Test build #122091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122091/testReport)** for PR 28376 at commit [`65f2504`](https://github.com/apache/spark/commit/65f2504ae96dc2b1c6c70eb4dcd3b3b06d7642a6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620288648
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620918572
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621446516
**[Test build #122089 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122089/testReport)** for PR 28376 at commit [`46c14ff`](https://github.com/apache/spark/commit/46c14ff36b9c648a9d3bf1b3a41dcf6e3672d28f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620281603
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620281151
**[Test build #121929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121929/testReport)** for PR 28376 at commit [`6d54251`](https://github.com/apache/spark/commit/6d54251dd950c4f6b36a2aa88165789aefeb8750).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621475997
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620284493
Thank you for pinging me, @dbtsai .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621458663
**[Test build #122090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122090/testReport)** for PR 28376 at commit [`762fd39`](https://github.com/apache/spark/commit/762fd39b22324b91db57242ecd9867227c1ab206).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621469989
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621446516
**[Test build #122089 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122089/testReport)** for PR 28376 at commit [`46c14ff`](https://github.com/apache/spark/commit/46c14ff36b9c648a9d3bf1b3a41dcf6e3672d28f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621475997
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai edited a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
dbtsai edited a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620738022
@tgravescs the standard "with hadoop" Spark built with Yarn is sufficient to work in our CDH env. Thanks for the review. I'll update the doc and config.md.
@viirya the idea of this PR is isolate the classpath in cluster from the Spark distribution, so when Spark runs in the Yarn clusters, the deps from Yarn will not be added. Thus, users don't need to pay attention on dep issue since Spark distribution is the only source of the classpath.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620288648
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620296694
The other way we can do is changing `yarn.application.classpath` and `mapreduce.application.classpath` to an empty folder, but this is more hackey unfortunately.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620913658
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28376: [SPARK-31582] [Yarn] Able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620281151
**[Test build #121929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121929/testReport)** for PR 28376 at commit [`6d54251`](https://github.com/apache/spark/commit/6d54251dd950c4f6b36a2aa88165789aefeb8750).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620288495
**[Test build #121929 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121929/testReport)** for PR 28376 at commit [`6d54251`](https://github.com/apache/spark/commit/6d54251dd950c4f6b36a2aa88165789aefeb8750).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28376:
URL: https://github.com/apache/spark/pull/28376#discussion_r416733376
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala
##########
@@ -70,6 +70,13 @@ package object config {
.booleanConf
.createWithDefault(false)
+ private[spark] val POPULATE_HADOOP_CLASSPATH = ConfigBuilder("spark.yarn.populateHadoopClasspath")
+ .doc("Whether to populate Hadoop classpath from `yarn.application.classpath` and " +
Review comment:
+1 for the comment and the doc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621459337
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621462704
**[Test build #122091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122091/testReport)** for PR 28376 at commit [`65f2504`](https://github.com/apache/spark/commit/65f2504ae96dc2b1c6c70eb4dcd3b3b06d7642a6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621469824
**[Test build #122090 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122090/testReport)** for PR 28376 at commit [`762fd39`](https://github.com/apache/spark/commit/762fd39b22324b91db57242ecd9867227c1ab206).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] adamkennedy commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
adamkennedy commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620914526
@viirya The embedded Hadoop could easily be incompatible with the cluster. But then the Spark version could be incompatible with the cluster's Hadoop as well. If the Spark chooses to bundle Hadoop libraries, then at least we know that works, and then the onus is on the Spark distribution to provide a different-but-compatible version, and the failure is only a failure for that one job. But at least the Spark provider gets an opportunity to provide something working, as opposed to the situation where different parts of the two different distributions end up cross-loading each other unexpectedly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621457718
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620918497
**[Test build #122019 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122019/testReport)** for PR 28376 at commit [`0ed4617`](https://github.com/apache/spark/commit/0ed46177ce77b2213cfb37bc9d67a0aabac660be).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620913347
**[Test build #122019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122019/testReport)** for PR 28376 at commit [`0ed4617`](https://github.com/apache/spark/commit/0ed46177ce77b2213cfb37bc9d67a0aabac660be).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621443246
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621450856
@tgravescs I plan to bring this to 2.4.6 since it's default to true and there is no risk.
Many Spark users who use vendor's hadoop such as CDH or HDP are stuck on very old versions of Spark. When they try to upgrade the Spark version, vendors ask them to upgrade the whole stack which is often not realistic. With this feature, those users will be able to migrate to latest OSS Apache Spark without depending on vendors.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621451320
Thanks all for reviews. I added a new UT, and after test passes, I'll merge it into master, branch-3.0, branch-2.4.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620918572
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620913347
**[Test build #122019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122019/testReport)** for PR 28376 at commit [`0ed4617`](https://github.com/apache/spark/commit/0ed46177ce77b2213cfb37bc9d67a0aabac660be).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620992909
cc @holdenk since this seems to be targeting 2.4.6.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #28376: [SPARK-31582] [Yarn] Able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620280527
cc @vanzin @srowen @dongjoon-hyun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621457718
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621475803
**[Test build #122091 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122091/testReport)** for PR 28376 at commit [`65f2504`](https://github.com/apache/spark/commit/65f2504ae96dc2b1c6c70eb4dcd3b3b06d7642a6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620625239
overall I think the change is fine. You could end up having issues if the user tried the external shuffle service and things weren't compatible but that is problem with anyone trying to run multiple versions. obviously requires them to have all the dependencies they need.
@viirya its ok to have multiple versions of spark on a cluster and many times there are just dependency issues with the different Hadoop versions. It will be rpc compatible but you run into different versions of dependencies (guava, jetty, etc).
The other way to possible solve this is to make sure these versions are on the class path first, but that can get tricky.
Is the standard spark package that includes Hadoop sufficient to run for you then @dbtsai ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621458663
**[Test build #122090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122090/testReport)** for PR 28376 at commit [`762fd39`](https://github.com/apache/spark/commit/762fd39b22324b91db57242ecd9867227c1ab206).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621443246
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621457556
**[Test build #122089 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122089/testReport)** for PR 28376 at commit [`46c14ff`](https://github.com/apache/spark/commit/46c14ff36b9c648a9d3bf1b3a41dcf6e3672d28f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620913658
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620717201
@tgravescs I see. That makes sense. So users still need to pay attention on dependencies issue if any when running multiple versions there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620291609
I generally don't like yet another config option, but I don't object to this.
Is there any simpler way to solve it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620738022
@tgravescs the standard "with hadoop" Spark built with Yarn is sufficient to work in our CDH env. Thanks for the review. I'll update the doc and config.md.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620404585
Ping @jerryshao @tgravescs @ulysses-you who work on this area for more eyes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620437448
One question, for the case mentioned in the description, "One case we have is when a user uses an Apache Spark distribution with its-own embedded hadoop, and submits a job to Cloudera or Hortonworks Yarn clusters", since the embedded hadoop is incompatible to the cluster, is it generally okay to submit the app there? Even you don't populate the classpath, will RPC or protocol be a problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28376: [SPARK-31582] [Yarn] Able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620281603
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-642866772
https://github.com/apache/spark/pull/28788 is a PR that truns `spark.yarn.populateHadoopClasspath` to `false` by default for `with-hadoop` build.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org