You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/10/08 17:22:17 UTC

[GitHub] [spark] sunchao opened a new pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

sunchao opened a new pull request #29843:
URL: https://github.com/apache/spark/pull/29843


   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   This upgrades Spark's Hadoop version from 3.2.0 to 3.2.1. As Guava was upgraded to 27.0 in Hadoop 3.2.1, this also switches to use shaded Hadoop client jars (i.e., hadoop-client-api and hadoop-client-runtime) to avoid the Guava conflicts.
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   Latest Hadoop versions have upgraded to use Guava 27+ and in order to adopt the latest Hadoop versions in Spark, we'll need to resolve the Guava conflict issue. This takes the approach by switching to shaded client jars provided by Hadoop.
   
   **TODO: handle Hadoop 2.7, perhaps through Maven profile.**
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   Default Hadoop version is bumped to 3.2.1
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   
   Relying on existing tests.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704629847


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704629847






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r494467897



##########
File path: external/kafka-0-10-sql/pom.xml
##########
@@ -79,6 +79,10 @@
       <artifactId>kafka-clients</artifactId>
       <version>${kafka.version}</version>
     </dependency>
+    <dependency>
+      <groupId>com.google.code.findbugs</groupId>

Review comment:
       Thanks. Yes will do after making all tests pass.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698158264


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129064/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-734002840


   Currently, we are searching all options.
   - The Spark-side workaround
   - New Hadoop release.
   - Reverting to the old dependency


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701061067


   **[Test build #129258 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129258/testReport)** for PR 29843 at commit [`58c461b`](https://github.com/apache/spark/commit/58c461b82138309f1d23b6a66eb57fb6d8b5a7a0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao edited a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
sunchao edited a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-705712615


   BTW all tests in github actions are passing now.  It'd be great if this can get another round of high-level review (still need to fix some small issues here and there) to see if the approach is worthwhile. 
   
   Edit: oops closed this PR by mistake.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r509716889



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -118,19 +118,32 @@ private[hive] object IsolatedClientLoader extends Logging {
       hadoopVersion: String,
       ivyPath: Option[String],
       remoteRepos: String): Seq[URL] = {
+    val hadoopJarNames = if (hadoopVersion.startsWith("3")) {
+      Seq(s"org.apache.hadoop:hadoop-client-api:$hadoopVersion",
+        s"org.apache.hadoop:hadoop-client-runtime:$hadoopVersion")
+    } else {
+      Seq(s"org.apache.hadoop:hadoop-client:$hadoopVersion")
+    }
     val hiveArtifacts = version.extraDeps ++
       Seq("hive-metastore", "hive-exec", "hive-common", "hive-serde")
         .map(a => s"org.apache.hive:$a:${version.fullVersion}") ++
-      Seq("com.google.guava:guava:14.0.1",
-        s"org.apache.hadoop:hadoop-client:$hadoopVersion")
+      Seq("com.google.guava:guava:14.0.1") ++ hadoopJarNames
+
+    val extraExclusions = if (hadoopVersion.startsWith("3")) {
+      // this introduced from lower version of Hive could conflict with jars in Hadoop 3.2+, so
+      // exclude here in favor of the ones in Hadoop 3.2+
+      Seq("org.apache.hadoop:hadoop-auth")

Review comment:
       No not same artifacts with different versions, but different artifacts with different versions (e.g., `hadoop-common` 2.7 but `hadoop-auth` 3.2).
   
   Yes we could do a cleanup and remove all other dependencies here, although somewhat related I'm wondering whether we do need to support not-sharing classes at all. It is currently used only in tests and in a rare case where it can't find [the specified hadoop version](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala#L78) (I don't fully understand why it switch to not using shared classes in the latter). 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-705246474


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129516/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-712466499


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34626/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-703741389


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-707390559






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704799523


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34103/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704313101


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129436/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706760756


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34248/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709682442






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-702466463


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33936/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706844653


   **[Test build #129648 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129648/testReport)** for PR 29843 at commit [`d8af755`](https://github.com/apache/spark/commit/d8af755bc0148541311f6829c78af74d44e9e7c2).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-703704196


   **[Test build #129416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129416/testReport)** for PR 29843 at commit [`08c03fb`](https://github.com/apache/spark/commit/08c03fb6cfbd2518938f86e695307b93d62a4e02).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710562172






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710585079


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129911/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-712559058






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706823721


   **[Test build #129644 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129644/testReport)** for PR 29843 at commit [`d8af755`](https://github.com/apache/spark/commit/d8af755bc0148541311f6829c78af74d44e9e7c2).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709710578


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129865/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao edited a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
sunchao edited a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-705712615


   BTW all tests in github actions are passing now.  It'd be great if this can get another round of high-level review (still need to fix some small issues here and there) to see if the overall approach is worthwhile. 
   
   Edit: oops closed this PR by mistake.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701148273






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709608018


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34463/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-708978108






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on a change in pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r493886155



##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -308,7 +307,7 @@ private[spark] class ApplicationMaster(
       // The client-mode AM doesn't listen for incoming connections, so report an invalid port.
       registerAM(Utils.localHostName, -1, sparkConf,
         sparkConf.getOption("spark.driver.appUIAddress"), appAttemptId)
-      addAmIpFilter(Some(driverRef), ProxyUriUtils.getPath(appAttemptId.getApplicationId))
+      addAmIpFilter(Some(driverRef), s"/proxy/$appAttemptId")

Review comment:
       BTW, do we need uri encode as `ProxyUriUtils.getPath` does internally?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-696968506






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-714097937


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34720/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704607572






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709452091


   **[Test build #129847 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129847/testReport)** for PR 29843 at commit [`a537e41`](https://github.com/apache/spark/commit/a537e41176b620cdd6d73e3c92a83993c9b720ac).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704071681


   Let me push the change and see if it passes in Jenkins too.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r509720001



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -118,19 +118,32 @@ private[hive] object IsolatedClientLoader extends Logging {
       hadoopVersion: String,
       ivyPath: Option[String],
       remoteRepos: String): Seq[URL] = {
+    val hadoopJarNames = if (hadoopVersion.startsWith("3")) {
+      Seq(s"org.apache.hadoop:hadoop-client-api:$hadoopVersion",
+        s"org.apache.hadoop:hadoop-client-runtime:$hadoopVersion")
+    } else {
+      Seq(s"org.apache.hadoop:hadoop-client:$hadoopVersion")
+    }
     val hiveArtifacts = version.extraDeps ++
       Seq("hive-metastore", "hive-exec", "hive-common", "hive-serde")
         .map(a => s"org.apache.hive:$a:${version.fullVersion}") ++
-      Seq("com.google.guava:guava:14.0.1",
-        s"org.apache.hadoop:hadoop-client:$hadoopVersion")
+      Seq("com.google.guava:guava:14.0.1") ++ hadoopJarNames
+
+    val extraExclusions = if (hadoopVersion.startsWith("3")) {
+      // this introduced from lower version of Hive could conflict with jars in Hadoop 3.2+, so
+      // exclude here in favor of the ones in Hadoop 3.2+
+      Seq("org.apache.hadoop:hadoop-auth")

Review comment:
       It also related to the definition of "not sharing Hadoop classes": does it mean Hadoop classes from Hive dependency take precedence over those from Spark itself? if that's true then we probably shouldn't exclude those.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29843: [SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706739672


   Gentle ping, @sunchao .


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-708860464


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34386/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dbtsai commented on a change in pull request #29843: [SPARK-29250][BUILD] Move to shaded clients for Hadoop 3.2

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r509671705



##########
File path: external/kafka-0-10-sql/pom.xml
##########
@@ -79,6 +79,10 @@
       <artifactId>kafka-clients</artifactId>
       <version>${kafka.version}</version>
     </dependency>
+    <dependency>
+      <groupId>com.google.code.findbugs</groupId>

Review comment:
       Should we add the license before we merge it?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-697025413


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710730499


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34533/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710584861


   **[Test build #129911 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129911/testReport)** for PR 29843 at commit [`5d27163`](https://github.com/apache/spark/commit/5d271633a55c1ae025086621ce82ecdf6dbb00e0).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709692035


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34473/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706845099


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704635457


   **[Test build #129482 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129482/testReport)** for PR 29843 at commit [`e7d0f11`](https://github.com/apache/spark/commit/e7d0f115fb27bb597360b4048e2ba4d413c38905).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-733984555


   Since we need a working `hadoop-aws` feature for Spark QA, I inevitably made a PR as one of the alternative to see the feasibility on reverting.
   - https://github.com/apache/spark/pull/30508


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dbtsai commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704753694


   Test this again 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704756667


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709800953






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709631978


   **[Test build #129860 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129860/testReport)** for PR 29843 at commit [`727a6f8`](https://github.com/apache/spark/commit/727a6f86b3ea8055eb9c65f6b4577a500b7dc509).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698626897


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33709/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-700993793






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709711069






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-697025413






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-697025418


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128992/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-696971847


   Great! Thanks, @sunchao .


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-703903632


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701024835


   **[Test build #129256 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129256/testReport)** for PR 29843 at commit [`9c5e225`](https://github.com/apache/spark/commit/9c5e2250646cecc4f616566a6ea6d0a9507bf1c9).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701698585


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33906/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701745390


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33910/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710714498


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706817475


   Sorry but I guess you don't get me. I meant simply sticking to Hadoop 3.2.x sounds better than keeping us to deal with both Hadoop 3.x and 2.x, if Hadoop community "officially" guarantees the compatibility. Bumping up the 2.x version is less ideal and should be taken only when Hadoop 3.x isn't guaranteed to communicate with Hadoop 2.x. So my intention is simply opposite.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-712528184


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34634/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-705043482






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon edited a comment on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-697066294


   @sunchao, I think it's because Jenkins uses Python 2 for `python` executable which we dropped (whereas GitHub Actions uses Python 3 properly for `python` executable`. That module has been renamed to socketserver in Python 3.
   
   Can you explicitly change the test to use `python3` instead of `python` in the tests at, for example, here:
   https://github.com/apache/spark/blob/master/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala#L177-L180
   
   This fails in this PR specifically because the tests failed here have `ExtendedYarnTest` tag which will only be run when some changes are made into Yarn.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-705020439






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698626880


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33709/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710356604


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34511/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698737858


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129088/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-700931113


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33865/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-703903613


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34028/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706823337


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34252/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-702483734


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33939/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dbtsai commented on pull request #29843: [SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706035486


   +1 on considering to deprecate Hadoop 2.7
   
   As @HeartSaVioR mentioned, if we are able to upgrade to Hadoop 3.2.1, we can remove workaround via SPARK-28025. However, without deprecating Hadoop 2.7, we'll have to maintain two code paths and we choose different implements depending on which Hadoop client is used.
   
   New Hadoop 3.2 client works really well with older Hadoop clusters such as 2.7. In fact, we deploy Spark with bundled Hadoop 3.2 on Hadoop 2.7 Yarn clusters in production with great success.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704022584






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701127202






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709645284


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34466/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-702520516


   Just one last test failure:
   
   "Exception: Python in worker has different version 3.6 than that in driver 3.8, PySpark cannot run with different minor versions. Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set."
   
   @HyukjinKwon do you happen to know the reason for this? looking at the CI script it seems it should install either Python 3.6 or 3.8, but not both.
   
   I'm also not sure which part of this PR could affect the Yarn/Python tests. I tried on my own Spark fork with some dummy change in `YarnClusterSuite` (just to trigger tests on `ExtendedYarnTest`) and the tests there all passed, so it seems the failure is indeed related to this PR.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-700980371


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33867/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-707533856


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129713/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706807152


   **[Test build #129647 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129647/testReport)** for PR 29843 at commit [`d8af755`](https://github.com/apache/spark/commit/d8af755bc0148541311f6829c78af74d44e9e7c2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-703903639


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34028/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706755735


   **[Test build #129644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129644/testReport)** for PR 29843 at commit [`d8af755`](https://github.com/apache/spark/commit/d8af755bc0148541311f6829c78af74d44e9e7c2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-702008427






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704427362


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34062/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-703903632






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698618920


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33709/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704809090






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-702483812






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #29843: [SPARK-29250][BUILD] Move to shaded clients for Hadoop 3.2

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-713194961


   > Should we create a new JIRA for moving to shaded client?
   
   I'm fine with a new JIRA - was going to use SPARK-29250 (which is titled "Upgrade to Hadoop 3.2.1 and move to shaded client") for both PRs.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-707369074


   **[Test build #129710 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129710/testReport)** for PR 29843 at commit [`0312f0a`](https://github.com/apache/spark/commit/0312f0a8536d368e8d469fb387d0fcd0dda75df1).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao edited a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
sunchao edited a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-712586049


   > @sunchao, yes I think we can do that but would you mind creating a separate PR to fix the test first though? Using python3 with my workaround fix should be good enough.
   
   @HyukjinKwon Currently the github action tests pass without the `python3` change, and the jenkins tests fail either w/ or w/o it: in the latter case it fails with error such as:
   ```
   20/10/16 19:20:36 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0) (amp-jenkins-worker-03.amp executor 1): org.apache.spark.SparkException: 
   Error from python worker:
     Traceback (most recent call last):
       File "/usr/lib64/python2.6/runpy.py", line 104, in _run_module_as_main
         loader, code, fname = _get_module_details(mod_name)
       File "/usr/lib64/python2.6/runpy.py", line 79, in _get_module_details
         loader = get_loader(mod_name)
       File "/usr/lib64/python2.6/pkgutil.py", line 456, in get_loader
         return find_loader(fullname)
       File "/usr/lib64/python2.6/pkgutil.py", line 466, in find_loader
         for importer in iter_importers(fullname):
       File "/usr/lib64/python2.6/pkgutil.py", line 422, in iter_importers
         __import__(pkg)
       File "/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/__init__.py", line 53, in <module>
         from pyspark.rdd import RDD, RDDBarrier
       File "/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/rdd.py", line 34, in <module>
         from pyspark.java_gateway import local_connect_and_auth
       File "/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/java_gateway.py", line 29, in <module>
         from py4j.java_gateway import java_import, JavaGateway, JavaObject, GatewayParameters
       File "/home/jenkins/workspace/SparkPullRequestBuilder@2/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 60
         PY4J_TRUE = {"yes", "y", "t", "true"}
                           ^
     SyntaxError: invalid syntax
   ```
   
   So do we still need a separate PR for that right now?
   
   > Also, seems like we're going to split PR (?). The first one (this) is for preparation, and second one is actually bumping up to Hadoop version to 3.2.1 (?). Would you mind clarifying the plan and what this PR proposes in the description/title?
   
   Yes that's right. The plan is to have a separate PR bumping Hadoop version to 3.2.2 when that comes out (probably will be soon). There is a [bug](https://issues.apache.org/jira/browse/HDFS-15191) in 3.2.1 which affects wire compatibility between 3.2 clients and 2.x server. 
   
   I'll update the PR description soon. Thanks.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-735999866


   Thank you, @sunchao !


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701039690


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33873/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-708978108






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-712450150


   **[Test build #130019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130019/testReport)** for PR 29843 at commit [`d19b36b`](https://github.com/apache/spark/commit/d19b36b7de12f2a9418a9bc7ddca927a7c18317e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-697105452


   Ah, thanks @HyukjinKwon ! Yeah on my local laptop I set those two environment variables and tests are passing. Will make the change in the test suite.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709666125


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129860/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706809001


   @HeartSaVioR . Why don't you send out your opinion to the community?
   For this, your question (or request) is far beyond the scope of this PR. Although I removed `hive-1.2` to unblock the removal of `Hadoop 2.7`, I disagree with you that `bumping Hadoop 2.x version is a better option than this`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709790822






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-735284760


   Thanks @steveloughran , yeah agree that the java 9 modules feature looks promising (it was discussed some years back in [HADOOP-11656](https://issues.apache.org/jira/browse/HADOOP-11656) but now the timing should be more right). I can try to spend sometime looking at this.
   
   This will take a while though. In the meanwhile, I'm wondering what we can do to ship this in the soon-coming Spark 3.1 release. One possible solution, maybe, is to still use non-shaded client when the `hadoop-cloud` profile is picked up:
   
   ```diff
   diff --git a/pom.xml b/pom.xml
   index 3ae2e7420e..12c36af557 100644
   --- a/pom.xml
   +++ b/pom.xml
   @@ -3238,6 +3238,11 @@
   
        <profile>
          <id>hadoop-cloud</id>
   +      <properties>
   +        <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
   +        <hadoop-client-runtime.artifact>hadoop-client</hadoop-client-runtime.artifact>
   +        <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
   +      </properties>
          <modules>
            <module>hadoop-cloud</module>
          </modules>
   ```
   
   Or maybe we could try to shade the `hadoop-aws` jar in the `spark-hadoop-cloud_2.12` module itself so that it invokes the shaded API from `hadoop-common` side. This won't work if Spark users decide to use their own Hadoop jars (via `hadoop-provided`) so we may have to make the `hadoop-aws` a compile scope dependency.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] xkrogen commented on a change in pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r509722738



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -118,19 +118,32 @@ private[hive] object IsolatedClientLoader extends Logging {
       hadoopVersion: String,
       ivyPath: Option[String],
       remoteRepos: String): Seq[URL] = {
+    val hadoopJarNames = if (hadoopVersion.startsWith("3")) {
+      Seq(s"org.apache.hadoop:hadoop-client-api:$hadoopVersion",
+        s"org.apache.hadoop:hadoop-client-runtime:$hadoopVersion")
+    } else {
+      Seq(s"org.apache.hadoop:hadoop-client:$hadoopVersion")
+    }
     val hiveArtifacts = version.extraDeps ++
       Seq("hive-metastore", "hive-exec", "hive-common", "hive-serde")
         .map(a => s"org.apache.hive:$a:${version.fullVersion}") ++
-      Seq("com.google.guava:guava:14.0.1",
-        s"org.apache.hadoop:hadoop-client:$hadoopVersion")
+      Seq("com.google.guava:guava:14.0.1") ++ hadoopJarNames
+
+    val extraExclusions = if (hadoopVersion.startsWith("3")) {
+      // this introduced from lower version of Hive could conflict with jars in Hadoop 3.2+, so
+      // exclude here in favor of the ones in Hadoop 3.2+
+      Seq("org.apache.hadoop:hadoop-auth")

Review comment:
       True... I've recently been debugging some test issues in our environment that are caused by this behavior, and am also confused by its existence.
   
   Given that we explicitly specify a Hadoop version to `downloadVersion` (which can be different from the one the rest of Spark is using), it seems that should take precedence over the ones Hive pulls in.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710315519


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34511/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dbtsai commented on a change in pull request #29843: [SPARK-29250][BUILD] Move to shaded clients for Hadoop 3.2

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r509671252



##########
File path: dev/deps/spark-deps-hadoop-2.7-hive-2.3
##########
@@ -126,7 +126,7 @@ javax.inject/1//javax.inject-1.jar
 javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar
 javax.servlet-api/3.1.0//javax.servlet-api-3.1.0.jar
 javolution/5.5.1//javolution-5.5.1.jar
-jaxb-api/2.2.2//jaxb-api-2.2.2.jar
+jaxb-api/2.2.11//jaxb-api-2.2.11.jar

Review comment:
       As we don't touch hadoop 2.7 deps, why do we need to change this file?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709692015


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34473/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-708863383






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon edited a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-712540024


   > BTW I'm still not sure why my PR will trigger the YARN/Python test failures - seems it shouldn't be related.
   
   This is because the regular test cases do not trigger the Yarn test cases. I am sure it was already broken before (in Jenkins). The relevant YARN test cases are triggered when the PR has some changes _only in YARN side_. See also https://github.com/apache/spark/blob/31a16fbb405a19dc3eb732347e0e1f873b16971d/dev/sparktestsupport/modules.py#L615
   
   See also https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130016/testReport/ at https://github.com/apache/spark/pull/29906


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698626889


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698624838


   **[Test build #129092 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129092/testReport)** for PR 29843 at commit [`c9534c7`](https://github.com/apache/spark/commit/c9534c709ee8e7fbd87fe5d261e8a8f317e7660d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-697025108


   **[Test build #128992 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128992/testReport)** for PR 29843 at commit [`09ea1e3`](https://github.com/apache/spark/commit/09ea1e3621448c13ff931ad1f23ec2f387a83f57).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710356575






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704733871






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709197897


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34432/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710462150


   **[Test build #129911 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129911/testReport)** for PR 29843 at commit [`5d27163`](https://github.com/apache/spark/commit/5d271633a55c1ae025086621ce82ecdf6dbb00e0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709692028


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-703875211


   **[Test build #129421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129421/testReport)** for PR 29843 at commit [`a4c6908`](https://github.com/apache/spark/commit/a4c69084b1d56d61191ce6975cd82e370a2a36cb).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-702466475






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701685269


   **[Test build #129290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129290/testReport)** for PR 29843 at commit [`765b958`](https://github.com/apache/spark/commit/765b9587dd81f877a5e06ad6d50d9fb021f7ee9e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709619400


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34464/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710562102


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34517/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698251842


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709790822


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-708925060






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-708784872






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-733983586


   Thank, please try to build a distribution with `hadoop-cloud` and access `S3`.
   ```
   NO_MANUAL=1 ./dev/make-distribution.sh --pip --tgz -Pkubernetes,hadoop-3.2,hadoop-cloud
   ```
   
   In the above example, I used `S3` as a Spark event log directory.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698183820


   **[Test build #129071 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129071/testReport)** for PR 29843 at commit [`7bca429`](https://github.com/apache/spark/commit/7bca429dd72835d7d6fa54f2420d39192624fd18).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-707376554


   Retest this please.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709682442






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-697049956


   The Python/YARN tests are failing with:
   ```
   /usr/bin/python: No module named socketserver
   ```
   which is kind of interesting. I also was not able to reproduce the error for "run Python application in yarn-client mode"


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698462510


   BTW the other SparkQA also failed with:
   ```
   java.io.IOException: Cannot run program "python3": error=2, No such file or directory
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698184534






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710533518


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34517/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709630557


   Retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-703741389






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-703894836


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34028/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-705043439


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34121/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-705020450


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129495/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] tgravescs commented on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-714488365


   sorry I said that backwards, by default if we are including a shaded version of Hadoop, that means users won't pick up all the dependencies Hadoop has by default. This is kind of the point for guava for spark. But it also means if users were relying on this behavior its now broken. Ideally users wouldn't rely on this but many times they don't know or think about it. I guess my point is I think its worth documenting this as its a change in the behavior. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704586991


   **[Test build #129474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129474/testReport)** for PR 29843 at commit [`df849b5`](https://github.com/apache/spark/commit/df849b55fe3fc355077e8a457267f9979b3df0c5).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-714008486


   **[Test build #130111 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130111/testReport)** for PR 29843 at commit [`0359047`](https://github.com/apache/spark/commit/0359047cd02ed7051ade3002c84ab5f3255a67cb).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710746708






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710562172






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701875155


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29843: [SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r502329417



##########
File path: sql/hive/pom.xml
##########
@@ -191,6 +191,16 @@
     </dependency>
   </dependencies>
   <profiles>
+    <profile>
+    <id>hadoop-3.2</id>

Review comment:
       indentation?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-697066294


   @sunchao, I think it's because Jenkins uses Python 2. That module has been renamed to socketserver in Python 3. Can you explicitly change the test to use `python3` instead of `python` in the tests?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701726322


   **[Test build #129294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129294/testReport)** for PR 29843 at commit [`586bf14`](https://github.com/apache/spark/commit/586bf1459ed2f027d8fe8346baab07af18251a86).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-733984555


   Since we need a working `hadoop-aws` feature for Spark QA, I inevitably made a PR as one of the alternative to see the feasibility on reverting. I'll keep it open until next Monday before `branch-cut` or until we find a workaround.
   - https://github.com/apache/spark/pull/30508


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710385652


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34513/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-702008437


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129294/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698158257






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-708867813


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34386/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698120109






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-712997968






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710585049


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-712372841


   The github action tests are all passing now while SparkQA is failing because of Python version issue (please check the [this run](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129907/testReport/)):
   
   ```
   20/10/16 10:52:54 INFO ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: User class threw exception: java.io.IOException: Cannot run program "python3": error=2, No such file or directory
   	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
   	at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:97)
   	at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:732)
   Caused by: java.io.IOException: error=2, No such file or directory
   	at java.lang.UNIXProcess.forkAndExec(Native Method)
   	at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
   	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
   	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
   	... 7 more
   ```
   
   @HyukjinKwon you mentioned that the jenkins only has Python 2 installed, and we've dropped support for Python 2? should we move jenkins to Python 3 as well?
   
   BTW I'm still not sure why my PR will trigger the YARN/Python test failures - seems it shouldn't be related. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710512983






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709141228


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34432/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701875155






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r494467897



##########
File path: external/kafka-0-10-sql/pom.xml
##########
@@ -79,6 +79,10 @@
       <artifactId>kafka-clients</artifactId>
       <version>${kafka.version}</version>
     </dependency>
+    <dependency>
+      <groupId>com.google.code.findbugs</groupId>

Review comment:
       Thanks. Yes will do after making all tests pass.

##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -118,11 +118,15 @@ private[hive] object IsolatedClientLoader extends Logging {
       hadoopVersion: String,
       ivyPath: Option[String],
       remoteRepos: String): Seq[URL] = {
+    val hadoopJarName = if (hadoopVersion.startsWith("3")) {

Review comment:
       Yes I think so. These modules should be available in any production Hadoop 3.x releases I think. See https://issues.apache.org/jira/browse/HADOOP-11804, it is fixed in 3.0.0-alpha2.

##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -118,11 +118,15 @@ private[hive] object IsolatedClientLoader extends Logging {
       hadoopVersion: String,
       ivyPath: Option[String],
       remoteRepos: String): Seq[URL] = {
+    val hadoopJarName = if (hadoopVersion.startsWith("3")) {

Review comment:
       Yes I believe so. These modules should be available in any production Hadoop 3.x releases I think. See https://issues.apache.org/jira/browse/HADOOP-11804, it is fixed in 3.0.0-alpha2.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #29843: [SPARK-29250][BUILD] Move to shaded clients for Hadoop 3.2

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-713193113


   The test failure is recorded in https://issues.apache.org/jira/browse/SPARK-33189


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-712498720


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] xkrogen commented on a change in pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r509710072



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -118,19 +118,32 @@ private[hive] object IsolatedClientLoader extends Logging {
       hadoopVersion: String,
       ivyPath: Option[String],
       remoteRepos: String): Seq[URL] = {
+    val hadoopJarNames = if (hadoopVersion.startsWith("3")) {
+      Seq(s"org.apache.hadoop:hadoop-client-api:$hadoopVersion",
+        s"org.apache.hadoop:hadoop-client-runtime:$hadoopVersion")
+    } else {
+      Seq(s"org.apache.hadoop:hadoop-client:$hadoopVersion")
+    }
     val hiveArtifacts = version.extraDeps ++
       Seq("hive-metastore", "hive-exec", "hive-common", "hive-serde")
         .map(a => s"org.apache.hive:$a:${version.fullVersion}") ++
-      Seq("com.google.guava:guava:14.0.1",
-        s"org.apache.hadoop:hadoop-client:$hadoopVersion")
+      Seq("com.google.guava:guava:14.0.1") ++ hadoopJarNames
+
+    val extraExclusions = if (hadoopVersion.startsWith("3")) {
+      // this introduced from lower version of Hive could conflict with jars in Hadoop 3.2+, so
+      // exclude here in favor of the ones in Hadoop 3.2+
+      Seq("org.apache.hadoop:hadoop-auth")

Review comment:
       I thought that Ivy would do dependency conflict resolution to resolve such issues? Do you actually get two versions of the same module (e.g. `hadoop-common` 2.7 and 3.2) or mixed Hadoop modules (like `hadoop-common` 2.7 but `hadoop-auth` 3.2)? I was under the impression that the former should be resolved automatically.
   
   In any case -- this seems like a good opportunity to clean it up. If we exclude everything _except_ `hadoop-client-api` and `hadoop-client-runtime`, that should cover excluding all of Hive's JARs, right?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-714183270


   **[Test build #130111 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130111/testReport)** for PR 29843 at commit [`0359047`](https://github.com/apache/spark/commit/0359047cd02ed7051ade3002c84ab5f3255a67cb).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710696280


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34526/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701703225






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dbtsai commented on a change in pull request #29843: [SPARK-29250][BUILD] Move to shaded clients for Hadoop 3.2

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r509673430



##########
File path: resource-managers/yarn/pom.xml
##########
@@ -69,25 +96,21 @@
     </dependency>
     <dependency>
       <groupId>org.apache.hadoop</groupId>
-      <artifactId>hadoop-yarn-api</artifactId>
-    </dependency>
-    <dependency>
-      <groupId>org.apache.hadoop</groupId>
-      <artifactId>hadoop-yarn-common</artifactId>
-    </dependency>
-    <dependency>
-      <groupId>org.apache.hadoop</groupId>
-      <artifactId>hadoop-yarn-server-web-proxy</artifactId>
+      <artifactId>${hadoop-client-api.artifact}</artifactId>
+      <version>${hadoop.version}</version>
     </dependency>
     <dependency>
       <groupId>org.apache.hadoop</groupId>
-      <artifactId>hadoop-yarn-client</artifactId>
+      <artifactId>${hadoop-client-runtime.artifact}</artifactId>
+      <version>${hadoop.version}</version>
+      <scope>${hadoop.deps.scope}</scope>
     </dependency>
     <dependency>
       <groupId>org.apache.hadoop</groupId>
-      <artifactId>hadoop-client</artifactId>
+      <artifactId>${hadoop-client-minicluster.artifact}</artifactId>
+      <version>${hadoop.version}</version>
+      <scope>test</scope>
     </dependency>
-

Review comment:
       nit, revert this unneeded change




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704106150






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704400747


   **[Test build #129455 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129455/testReport)** for PR 29843 at commit [`1ec7dac`](https://github.com/apache/spark/commit/1ec7dacbe82082a564241c589555945b62284978).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-703944668






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698625663


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129092/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698251854


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129071/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709748276


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129868/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-703741399


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34023/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-728282002


   Thanks @steveloughran for sharing the info! Totally agree. Yes my current plan is to wait until 3.2.2 comes out and then upgrade Spark to use that, and hope community will agree with that. People who want to use 3.3.x can still use the existing `-Dhadoop.version` flag to enable it.
   
   > 3.3.3 has a lot of s3 and abfs changes, but there's enough changes elsewhere to make it more traumatic
   
   you mean 3.3 here right? does 3.3.3 exist yet?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-699068276


   Thanks @srowen , I'm not sure how that will be useful though. The root issue here is not CI but making Hadoop 3.2 as the default Maven profile. Previously, this is realized by specifying `hadoop.version` to be 3.2.0, and it works fine since both Hadoop 2.7 and 3.2 share the same set of dependencies.
   
   Now with this PR, we'd have to use different sets of dependencies for 2.7 and 3.2 separately. With a property-based approach like the following:
   
   ```xml
       <profile>
         <id>hadoop-2.7</id>
         <dependencies>
           ..
         </dependencies>
       </profile>
       <profile>
         <id>hadoop-3.2</id>
         <!-- Default hadoop profile. Uses global properties. -->
         <activation>
           <property><name>!hadoop-2.7</name></property>
         </activation>
         <dependencies>
            ..
         </dependencies>
       </profile>
   ```
   we'll be able to make 3.2 as default profile (i.e., it will be activated when ppl are not using the `-Phadoop-3.2` flag). However to compile against 2.7 ppl have to change the Maven command to be:
   ```shell
   build/mvn -Phadoop-2.7 -Dhadoop-2.7 ...
   ```
   instead of today's 
   ```shell
   build/mvn -Phadoop-2.7 ...
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701643372






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-696967953


   **[Test build #128992 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128992/testReport)** for PR 29843 at commit [`09ea1e3`](https://github.com/apache/spark/commit/09ea1e3621448c13ff931ad1f23ec2f387a83f57).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao edited a comment on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
sunchao edited a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-697049956


   The Python/YARN tests are failing with:
   ```
   /usr/bin/python: No module named socketserver
   ```
   which is weird. I also was not able to reproduce all the Python related test failures locally.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-700946573


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33865/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-712528191






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dbtsai commented on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-714196111


   Merged into master. Thanks all for reviewing!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710239987


   **[Test build #129905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129905/testReport)** for PR 29843 at commit [`788f295`](https://github.com/apache/spark/commit/788f295510533920091dc7a72a5fc72186199a01).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on a change in pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r493851542



##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -308,7 +307,7 @@ private[spark] class ApplicationMaster(
       // The client-mode AM doesn't listen for incoming connections, so report an invalid port.
       registerAM(Utils.localHostName, -1, sparkConf,
         sparkConf.getOption("spark.driver.appUIAddress"), appAttemptId)
-      addAmIpFilter(Some(driverRef), ProxyUriUtils.getPath(appAttemptId.getApplicationId))
+      addAmIpFilter(Some(driverRef), s"/proxy/$appAttemptId")

Review comment:
       Why we are not using `getApplicationId` as before?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701193911






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-705709693






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r493780533



##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -308,7 +307,7 @@ private[spark] class ApplicationMaster(
       // The client-mode AM doesn't listen for incoming connections, so report an invalid port.
       registerAM(Utils.localHostName, -1, sparkConf,
         sparkConf.getOption("spark.driver.appUIAddress"), appAttemptId)
-      addAmIpFilter(Some(driverRef), ProxyUriUtils.getPath(appAttemptId.getApplicationId))
+      addAmIpFilter(Some(driverRef), s"/proxy/$appAttemptId")

Review comment:
       no - this is to remove dependency on Hadoop server side (`ProxyUriUtils`, which also calls private API and is not good).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-700946601


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33865/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r509796677



##########
File path: external/kafka-0-10-sql/pom.xml
##########
@@ -79,6 +79,10 @@
       <artifactId>kafka-clients</artifactId>
       <version>${kafka.version}</version>
     </dependency>
+    <dependency>
+      <groupId>com.google.code.findbugs</groupId>

Review comment:
       @srowen @dbtsai actually this was [already added](https://github.com/apache/spark/blob/master/LICENSE-binary#L322) - it makes sense because we're already using this from the old hadoop transitive dependencies, what's new is that we're making it explicit since hadoop-client-api excludes it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709671297


   **[Test build #129868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129868/testReport)** for PR 29843 at commit [`83afe58`](https://github.com/apache/spark/commit/83afe586ab322a1922bc30c17f74b4ea148c85fa).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-714184062






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704437472


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34062/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-707379187


   **[Test build #129713 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129713/testReport)** for PR 29843 at commit [`0312f0a`](https://github.com/apache/spark/commit/0312f0a8536d368e8d469fb387d0fcd0dda75df1).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-707420794


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129710/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710681201


   **[Test build #129920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129920/testReport)** for PR 29843 at commit [`d6c0a8d`](https://github.com/apache/spark/commit/d6c0a8dfea6cb47fcdfe3c9fd8ecaee5e9481645).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704650593






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-705060495


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710268992


   **[Test build #129907 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129907/testReport)** for PR 29843 at commit [`5d27163`](https://github.com/apache/spark/commit/5d271633a55c1ae025086621ce82ecdf6dbb00e0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709800958


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129858/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709671297


   **[Test build #129868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129868/testReport)** for PR 29843 at commit [`83afe58`](https://github.com/apache/spark/commit/83afe586ab322a1922bc30c17f74b4ea148c85fa).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710691127


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34526/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701148268






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709614972






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-700993802


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33867/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-699102799


   Why does hadoop-3.2 have to become the default - I'm missing that. (It may be a fine idea.)
   I see, so some dependencies can _only_ be activated in hadoop-2.7 and not hadoop-3.2, or it fails? OK that's harder, because you can't un-set them in the hadoop-3.2 profile. We can simply declare that now you must set one of the two profiles to build - that is also valid IMHO.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-728312121


   If that's the Hadoop community's recommendation, +1 for 3.2.2.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710681201


   **[Test build #129920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129920/testReport)** for PR 29843 at commit [`d6c0a8d`](https://github.com/apache/spark/commit/d6c0a8dfea6cb47fcdfe3c9fd8ecaee5e9481645).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698251231


   **[Test build #129071 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129071/testReport)** for PR 29843 at commit [`7bca429`](https://github.com/apache/spark/commit/7bca429dd72835d7d6fa54f2420d39192624fd18).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-698603349


   **[Test build #129088 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129088/testReport)** for PR 29843 at commit [`f6f19b5`](https://github.com/apache/spark/commit/f6f19b5485e2d420c21d80f1d0fad7556ffe93c9).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710356466


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34511/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710559764


   Retest this please.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-696968506






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-705019662


   **[Test build #129495 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129495/testReport)** for PR 29843 at commit [`e7d0f11`](https://github.com/apache/spark/commit/e7d0f115fb27bb597360b4048e2ba4d413c38905).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-708071828


   > Among the following, `HivePartitionFilteringSuite.2.2: create client with sharesHadoopClasses = false` seems to fail at Github Action, too. Is this a new after the last dependency change commit?
   
   Yes this is new after I rebase. I also couldn't reproduce the error locally. Still trying to find the root cause.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-708947069






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704106111


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34043/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-703704196


   **[Test build #129416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129416/testReport)** for PR 29843 at commit [`08c03fb`](https://github.com/apache/spark/commit/08c03fb6cfbd2518938f86e695307b93d62a4e02).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709709960


   **[Test build #129865 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129865/testReport)** for PR 29843 at commit [`ecb7136`](https://github.com/apache/spark/commit/ecb71364fec86e4fea1a6453e2ee1dfb4c4dd05c).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709651344






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709666583


   **[Test build #129865 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129865/testReport)** for PR 29843 at commit [`ecb7136`](https://github.com/apache/spark/commit/ecb71364fec86e4fea1a6453e2ee1dfb4c4dd05c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r509720001



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -118,19 +118,32 @@ private[hive] object IsolatedClientLoader extends Logging {
       hadoopVersion: String,
       ivyPath: Option[String],
       remoteRepos: String): Seq[URL] = {
+    val hadoopJarNames = if (hadoopVersion.startsWith("3")) {
+      Seq(s"org.apache.hadoop:hadoop-client-api:$hadoopVersion",
+        s"org.apache.hadoop:hadoop-client-runtime:$hadoopVersion")
+    } else {
+      Seq(s"org.apache.hadoop:hadoop-client:$hadoopVersion")
+    }
     val hiveArtifacts = version.extraDeps ++
       Seq("hive-metastore", "hive-exec", "hive-common", "hive-serde")
         .map(a => s"org.apache.hive:$a:${version.fullVersion}") ++
-      Seq("com.google.guava:guava:14.0.1",
-        s"org.apache.hadoop:hadoop-client:$hadoopVersion")
+      Seq("com.google.guava:guava:14.0.1") ++ hadoopJarNames
+
+    val extraExclusions = if (hadoopVersion.startsWith("3")) {
+      // this introduced from lower version of Hive could conflict with jars in Hadoop 3.2+, so
+      // exclude here in favor of the ones in Hadoop 3.2+
+      Seq("org.apache.hadoop:hadoop-auth")

Review comment:
       It's also related to the definition of "not sharing Hadoop classes": does it mean Hadoop classes from Hive dependency take precedence over those from Spark itself? if that's true then we probably shouldn't exclude those.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706820766






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-713053523






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-714184062






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709669036


   **[Test build #129867 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129867/testReport)** for PR 29843 at commit [`c37a7f6`](https://github.com/apache/spark/commit/c37a7f608e269cc1da1f82963d685073ffcb5969).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-707533849


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704607572






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710746490


   **[Test build #129928 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129928/testReport)** for PR 29843 at commit [`507270b`](https://github.com/apache/spark/commit/507270bbd9d2ae24c6d4750e3931d1738d8b0141).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709800953


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706763282






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710585049






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701103287


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33875/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-704792822






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-703952905


   Will take a look for the test failure today


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709710511


   **[Test build #129867 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129867/testReport)** for PR 29843 at commit [`c37a7f6`](https://github.com/apache/spark/commit/c37a7f608e269cc1da1f82963d685073ffcb5969).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709670092






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706763279


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34248/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709610368


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34464/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709790846


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129857/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-708890947


   **[Test build #129795 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129795/testReport)** for PR 29843 at commit [`936efb8`](https://github.com/apache/spark/commit/936efb8895491ae90a0030a0fcb027acfb4ac5be).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon edited a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-712540024


   > BTW I'm still not sure why my PR will trigger the YARN/Python test failures - seems it shouldn't be related.
   
   This is because the regular test cases do not trigger the Yarn test cases. I am sure it was already broken before (in Jenkins). The relevant YARN test cases are triggered when the PR has some changes _only in YARN side_. See also https://github.com/apache/spark/blob/31a16fbb405a19dc3eb732347e0e1f873b16971d/dev/sparktestsupport/modules.py#L615
   
   See also https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130016/testReport/ at https://github.com/apache/spark/pull/29906
   
   cc @tgravescs FYI


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-696967953






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #29843: [SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r509686253



##########
File path: dev/deps/spark-deps-hadoop-2.7-hive-2.3
##########
@@ -126,7 +126,7 @@ javax.inject/1//javax.inject-1.jar
 javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar
 javax.servlet-api/3.1.0//javax.servlet-api-3.1.0.jar
 javolution/5.5.1//javolution-5.5.1.jar
-jaxb-api/2.2.2//jaxb-api-2.2.2.jar
+jaxb-api/2.2.11//jaxb-api-2.2.11.jar

Review comment:
       This is a side-effect from the default Hadoop 3.x profile: since the shaded client jars exclude 3rd party dependencies include this, we'd have to explicitly import it even for 2.x profile. With some Maven trick maybe we can avoid the change but I'm not sure if this is a big deal.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709042943


   **[Test build #129827 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129827/testReport)** for PR 29843 at commit [`4646dca`](https://github.com/apache/spark/commit/4646dcab16e01bb656e49f9697acbe5107808c51).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710734648






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-710510564


   **[Test build #129907 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129907/testReport)** for PR 29843 at commit [`5d27163`](https://github.com/apache/spark/commit/5d271633a55c1ae025086621ce82ecdf6dbb00e0).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-700903003


   **[Test build #129248 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129248/testReport)** for PR 29843 at commit [`07e2242`](https://github.com/apache/spark/commit/07e2242307f7bffae425a01ce4b225ab0b342991).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701103295






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-712559058






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-709665791


   **[Test build #129860 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129860/testReport)** for PR 29843 at commit [`727a6f8`](https://github.com/apache/spark/commit/727a6f86b3ea8055eb9c65f6b4577a500b7dc509).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-701193918


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129256/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29843: [WIP][SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-702484958






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org