You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/05/24 16:43:55 UTC

[GitHub] [spark] sunchao opened a new pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

sunchao opened a new pull request #30135:
URL: https://github.com/apache/spark/pull/30135


   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   This upgrade default Hadoop version from 3.2.1 to 3.3.0. The changes here are simply update the version number and dependency file.
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   [Hadoop 3.3.0](https://hadoop.apache.org/docs/r3.3.0/) was released a few months back, which comes with many client-side improvements such as for S3A/ABFS. These are important for users who want to use Spark in a cloud environment.
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   No
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   
   Existing tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853595736


   **[Test build #139275 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139275/testReport)** for PR 30135 at commit [`547ccb0`](https://github.com/apache/spark/commit/547ccb0fb4094982ba4dfb44f53b69775624b50d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] arghya18 commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
arghya18 commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-872701489


   @dongjoon-hyun I was able to test my use case with Hadoop 3.3.1 and posted the result [HADOOP-17755](https://issues.apache.org/jira/browse/HADOOP-17755?focusedCommentId=17373213&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17373213)
   To my surprise the read is slower(with same resource and same config) in Hadoop 3.3.1 than Hadoop 3.2.0 without the mentioned issue. It is possible I am missing something. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] github-actions[bot] closed pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #30135:
URL: https://github.com/apache/spark/pull/30135


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-864466090


   Hi, All. FYI,
   
   From Apache Hadoop 3.3.1, we reverted HADOOP-16878 as the last commit on `branch-3.3.1`.
   - https://github.com/apache/hadoop/commit/a3b9c37a397ad4188041dd80621bdeefc46885f2
   
   However, Apache Spark Jenkins seems to hit a flakiness issue.
   - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2/lastCompletedBuild/testReport/org.apache.spark.deploy.yarn/ClientSuite/distribute_jars_archive/history/
   
   ```
   org.apache.hadoop.fs.PathOperationException: 
   `Source (file:/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2/resource-managers/yarn/target/tmp/spark-703b8e99-63cc-4ba6-a9bc-25c7cae8f5f9/testJar9120517778809167117.jar) and destination (/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2/resource-managers/yarn/target/tmp/spark-703b8e99-63cc-4ba6-a9bc-25c7cae8f5f9/testJar9120517778809167117.jar)
   are equal in the copy command.': Operation not supported
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847486841


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138898/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] arghya18 commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
arghya18 commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-868187147


   What's the latest on this? Any issue to use Hadoop 3.3.1 with Spark?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] gengliangwang commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-894003073


   @sunchao @dongjoon-hyun @steveloughran Are there any other known issues of Hadoop 3.3.1? Should we have this upgrade in Spark 3.2?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-894007146


   I'm not aware of any issue at the moment - we've already been using this (although a slightly different internal Hadoop version) for a while now. @steveloughran can offer more inputs from Hadoop/S3 side.
   
   Besides that, I wanted to add https://github.com/apache/spark/pull/33160 so that users can build Spark with an older version of Hadoop which do not support shaded client. If people feel this is a good addition I can try to push it to the finish line. I also wanted to add some user documentation on compiling Spark against different versions of Hadoop (besides the default one).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847236579


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43403/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847220566


   Yay! Thank you, @sunchao .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853380133


   **[Test build #139242 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139242/testReport)** for PR 30135 at commit [`e825730`](https://github.com/apache/spark/commit/e82573061bdaa160569222d749abb93eae8628f6).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-854911770


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43864/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-771277246


   This is staled because Spark can't use Hadoop 3.3.0 as it is (see HADOOP-16080). The work may be resumed once Hadoop 3.3.1 is released.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r639393055



##########
File path: pom.xml
##########
@@ -197,6 +197,7 @@
     <maven-antrun.version>1.8</maven-antrun.version>
     <commons-crypto.version>1.1.0</commons-crypto.version>
     <commons-cli.version>1.2</commons-cli.version>
+    <bouncycastle.version>1.52</bouncycastle.version>

Review comment:
       You're right - will change to `1.68`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848362841


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43466/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853380133


   **[Test build #139242 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139242/testReport)** for PR 30135 at commit [`e825730`](https://github.com/apache/spark/commit/e82573061bdaa160569222d749abb93eae8628f6).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-857889163


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44111/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dbtsai commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847185787


   Reopening this RP.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847242862


   **[Test build #138883 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138883/testReport)** for PR 30135 at commit [`3d6a648`](https://github.com/apache/spark/commit/3d6a64848f55bea124c24d715281005b9f8c4596).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853709793


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139275/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r639390795



##########
File path: pom.xml
##########
@@ -197,6 +197,7 @@
     <maven-antrun.version>1.8</maven-antrun.version>
     <commons-crypto.version>1.1.0</commons-crypto.version>
     <commons-cli.version>1.2</commons-cli.version>
+    <bouncycastle.version>1.52</bouncycastle.version>

Review comment:
       Although this is a test case, this looks like too old because it was released on Mar, 2015. Do we need to align with to Hadoop 3.3.1? Can we use the latest one `1.68`?
   - https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847240776


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43403/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847310474






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-858031988


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139584/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848201300


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43459/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848336727


   **[Test build #138946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138946/testReport)** for PR 30135 at commit [`e775345`](https://github.com/apache/spark/commit/e77534538ef212193fa733ce59cb12cdc2fa3d93).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848110340


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43459/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848401399


   **[Test build #138946 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138946/testReport)** for PR 30135 at commit [`e775345`](https://github.com/apache/spark/commit/e77534538ef212193fa733ce59cb12cdc2fa3d93).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847421698


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138887/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847486387


   **[Test build #138898 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138898/testReport)** for PR 30135 at commit [`c6e75c8`](https://github.com/apache/spark/commit/c6e75c814755372e9bf5036e0ebe111c2b3f8b61).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-858021812


   **[Test build #139584 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139584/testReport)** for PR 30135 at commit [`e83e73f`](https://github.com/apache/spark/commit/e83e73f51a8da6d2776ab141aaad9c954c6c4b7d).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-858085281


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44116/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853709793


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139275/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714776776


   And all the tests in jenkins are passing. This is weird ...


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-771327635


   👍 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714776989






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847417613


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43409/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-864466090


   Hi, All. FYI,
   
   From Apache Hadoop 3.3.1, we reverted HADOOP-16878 as the last commit on `branch-3.3.1`.
   - https://github.com/apache/hadoop/commit/a3b9c37a397ad4188041dd80621bdeefc46885f2
   
   However, Apache Spark Jenkins seems to hit a flakiness issue.
   - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2/lastCompletedBuild/testReport/org.apache.spark.deploy.yarn/ClientSuite/distribute_jars_archive/history/
   
   ```
   org.apache.hadoop.fs.PathOperationException: 
   `Source (file:/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2/resource-managers/yarn/target/tmp/spark-703b8e99-63cc-4ba6-a9bc-25c7cae8f5f9/testJar9120517778809167117.jar) and destination (/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2/resource-managers/yarn/target/tmp/spark-703b8e99-63cc-4ba6-a9bc-25c7cae8f5f9/testJar9120517778809167117.jar)
   are equal in the copy command.': Operation not supported
   ```
   
   Please note that this is a flakiness. I'm investigating this.
   
   cc @gengliangwang since he is the release manager for Apache Spark 3.2.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847461394


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43420/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-850008994


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139032/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714719820






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao edited a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
sunchao edited a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714810305


   Yeah it should work with Hadoop 3.2+ clusters since [Hadoop maintains wire compatibility across minor releases](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html). +1 on renaming this to hadoop-3 (and probably rename `hadoop-2.7` to `hadoop-2` as well).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853457538


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139242/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714685922


   **[Test build #130175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130175/testReport)** for PR 30135 at commit [`1a4cdeb`](https://github.com/apache/spark/commit/1a4cdeb18139387a50d49b1e9508e910107715a4).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847210555


   **[Test build #138881 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138881/testReport)** for PR 30135 at commit [`1a4cdeb`](https://github.com/apache/spark/commit/1a4cdeb18139387a50d49b1e9508e910107715a4).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853668788


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43799/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853406443


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43765/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-858085281


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44116/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847388599


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43409/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-854843591


   **[Test build #139342 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139342/testReport)** for PR 30135 at commit [`c6f1802`](https://github.com/apache/spark/commit/c6f1802749b52aaac89301c23a083d4c21c562aa).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847244085


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138883/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848210093


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138938/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847210555


   **[Test build #138881 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138881/testReport)** for PR 30135 at commit [`1a4cdeb`](https://github.com/apache/spark/commit/1a4cdeb18139387a50d49b1e9508e910107715a4).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-857889163


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44111/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-858033990


   **[Test build #139591 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139591/testReport)** for PR 30135 at commit [`7549c36`](https://github.com/apache/spark/commit/7549c369b57480e767d27a952d85df3001a337ba).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847421698


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138887/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848408043


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138946/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847421158


   **[Test build #138887 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138887/testReport)** for PR 30135 at commit [`edcee70`](https://github.com/apache/spark/commit/edcee7021006b61e1470ad993fdf4d82a7f5470b).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-854201334


   @dongjoon-hyun interesting. This is related to https://issues.apache.org/jira/browse/HADOOP-16878, and I thought it's already reverted from RC3 so I reverted my earlier change of handling it in Spark code. Let me check later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] github-actions[bot] commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-770492906


   We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-854198489


   Interesting. The same test fails in both Jenkins and GitHub Action. Is something changed at this new RC?
   - `ClientSuite.distribute jars archive`
   ```
   org.apache.hadoop.fs.PathOperationException: `Source (file:/home/jenkins/workspace/SparkPullRequestBuilder/target/tmp/spark-8a380e11-5074-471e-b9c8-f3f764e2c4e6/testJar948737110654121459.jar) and destination (/home/jenkins/workspace/SparkPullRequestBuilder/target/tmp/spark-8a380e11-5074-471e-b9c8-f3f764e2c4e6/testJar948737110654121459.jar) are equal in the copy command.': Operation not supported
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714775891


   **[Test build #130175 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130175/testReport)** for PR 30135 at commit [`1a4cdeb`](https://github.com/apache/spark/commit/1a4cdeb18139387a50d49b1e9508e910107715a4).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r640115792



##########
File path: pom.xml
##########
@@ -197,6 +197,7 @@
     <maven-antrun.version>1.8</maven-antrun.version>
     <commons-crypto.version>1.1.0</commons-crypto.version>
     <commons-cli.version>1.2</commons-cli.version>
+    <bouncycastle.version>1.52</bouncycastle.version>

Review comment:
       Actually Hadoop is reverting the change in https://issues.apache.org/jira/browse/HADOOP-17563 so we'll need to wait a bit. cc @steveloughran .




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714719820






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] arghya18 commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
arghya18 commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-868289618


   @dongjoon-hyun Thanks for your response. Yes I understand magic committer is not related, I just wanted to understand if I build Spark 3.1.1 with Hadoop 3.3.0/3.3.1 will the existing issues (SPARK-35831, SPARK-35868, SPARK-35878) to be merged will impact my use case? I am using s3a to read and write orc files or I have to wait for Spark 3.2.0 ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r639390795



##########
File path: pom.xml
##########
@@ -197,6 +197,7 @@
     <maven-antrun.version>1.8</maven-antrun.version>
     <commons-crypto.version>1.1.0</commons-crypto.version>
     <commons-cli.version>1.2</commons-cli.version>
+    <bouncycastle.version>1.52</bouncycastle.version>

Review comment:
       Although this is a test dependency, this looks like too old because it was released on Mar, 2015. Do we need to align with to Hadoop 3.3.1? Can we use the latest one `1.68`?
   - https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] github-actions[bot] closed pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #30135:
URL: https://github.com/apache/spark/pull/30135


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853657558


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43799/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-849931685


   **[Test build #139032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139032/testReport)** for PR 30135 at commit [`c56cb14`](https://github.com/apache/spark/commit/c56cb142414562c6caec8d4604dc3f6e7a8688a3).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] github-actions[bot] closed pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #30135:
URL: https://github.com/apache/spark/pull/30135


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847162782


   Sine Hadoop 3.3.1 RC1 is out, I'm going to revive this PR and test it here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] steveloughran commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-859048545


   > reverted from RC3 so I reverted my earlier change of handling it in Spark code. Let me check later.
   
   maybe the build is picking up the old RC.
   
   FWIW I'm surprised about this regression...add as much as you can to the test, as in "what is the test trying to do?". Gabor's patch is trying to stop "hadoop fs" copy command from copying a file onto itself, which can be, well, lossy, depending on the OS and things.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-849957464


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43550/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847481567


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43420/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-849931542


   Thank you for updates, @sunchao !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] arghya18 commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
arghya18 commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-868195002


   @dongjoon-hyun Thanks for the info. Actually we are having some major issue related to s3a [HADOOP-17755](https://issues.apache.org/jira/browse/HADOOP-17755) with orc which seems to be solved in Hadoop 3.3.0 for parquet but seems its a common issue . [HADOOP-16109](https://issues.apache.org/jira/browse/HADOOP-16109) . Currently we are working around it using fs.s3a.readahead.range": "1G" which is not a good idea.
   
   I am using Spark on k8s and magic committer, so wanted to check if the above issues mentioned will impact me if I manually build Spark 3.1.1 with Hadoop 3.3.0/3.3.1?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] steveloughran commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r649517273



##########
File path: pom.xml
##########
@@ -197,6 +197,7 @@
     <maven-antrun.version>1.8</maven-antrun.version>
     <commons-crypto.version>1.1.0</commons-crypto.version>
     <commons-cli.version>1.2</commons-cli.version>
+    <bouncycastle.version>1.52</bouncycastle.version>

Review comment:
       we rolled back the latest bouncy castle update as some internal spark builds with some extra modules were failing as the asm JAR couldn't handle the changes. Tried to fiddle with the spark settings there to use an updated asm JAR and gave up




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853668788


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43799/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847481567


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43420/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847275034


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43405/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848181744


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43459/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853420021






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847348795


   **[Test build #138887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138887/testReport)** for PR 30135 at commit [`edcee70`](https://github.com/apache/spark/commit/edcee7021006b61e1470ad993fdf4d82a7f5470b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848210093


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138938/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-864469570


   Since Apache Hadoop trunk has this behavior, I made a PR in order to be more robust on the underlying behavior difference.
   - https://github.com/apache/spark/pull/32983


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848374328


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43466/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r639393055



##########
File path: pom.xml
##########
@@ -197,6 +197,7 @@
     <maven-antrun.version>1.8</maven-antrun.version>
     <commons-crypto.version>1.1.0</commons-crypto.version>
     <commons-cli.version>1.2</commons-cli.version>
+    <bouncycastle.version>1.52</bouncycastle.version>

Review comment:
       You're right, forgot to check the latest version used by Hadoop 3.3.1. Will change to `1.68`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714719794


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34781/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853420001


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43765/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] steveloughran commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-868530293


   the parquet EOF fix is also in hadoop-3.2.2, so you could try that. However, testing with 3.3.1 is better because
   
   1. we can do workarounds in spark before the release
   2. we can fix hadoop branch-3.3;
   3.  any bug you file against hadoop 3.2.x will have "try with 3.3.1" as the initial respose


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714769312


   Yes the test failures are new. It seems to be related to #29843 but I'm not sure why I've never seen it before in all the previous runs.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-858085251


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44116/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-858033990


   **[Test build #139591 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139591/testReport)** for PR 30135 at commit [`7549c36`](https://github.com/apache/spark/commit/7549c369b57480e767d27a952d85df3001a337ba).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-858142554


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139591/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848336727


   **[Test build #138946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138946/testReport)** for PR 30135 at commit [`e775345`](https://github.com/apache/spark/commit/e77534538ef212193fa733ce59cb12cdc2fa3d93).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848073474


   **[Test build #138938 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138938/testReport)** for PR 30135 at commit [`263570c`](https://github.com/apache/spark/commit/263570c4b448e062e9b8454a9300ea4f7b0a9506).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r638157976



##########
File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3
##########
@@ -58,9 +58,10 @@ flatbuffers-java/1.9.0//flatbuffers-java-1.9.0.jar
 generex/1.0.2//generex-1.0.2.jar
 gson/2.2.4//gson-2.2.4.jar
 guava/14.0.1//guava-14.0.1.jar
-hadoop-client-api/3.2.2//hadoop-client-api-3.2.2.jar
-hadoop-client-runtime/3.2.2//hadoop-client-runtime-3.2.2.jar
-hadoop-yarn-server-web-proxy/3.2.2//hadoop-yarn-server-web-proxy-3.2.2.jar
+hadoop-client-api/3.3.1//hadoop-client-api-3.3.1.jar
+hadoop-client-runtime/3.3.1//hadoop-client-runtime-3.3.1.jar
+hadoop-shaded-guava/1.1.0//hadoop-shaded-guava-1.1.0.jar

Review comment:
       nice!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-849931685


   **[Test build #139032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139032/testReport)** for PR 30135 at commit [`c56cb14`](https://github.com/apache/spark/commit/c56cb142414562c6caec8d4604dc3f6e7a8688a3).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-849957464


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43550/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-771327635


   👍 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-715572480






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-857850894


   **[Test build #139584 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139584/testReport)** for PR 30135 at commit [`e83e73f`](https://github.com/apache/spark/commit/e83e73f51a8da6d2776ab141aaad9c954c6c4b7d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-858067849


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44116/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-854872911


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43864/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853420021


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43765/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847290944


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43405/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848201300


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43459/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] gengliangwang commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-895330908


   @steveloughran Thanks for the info!
   @sunchao then let's finish #33160 and have it in 3.2. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] steveloughran commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-859463610


   > For the regression, don't know the full context behind the original change but seems like a good thing to do, although a boolean flag returned might be less disruptive IMO :). Either way it is a simple fix from Spark side.
   
   we didn't think it was going to break things. a big issue is that in come cases the data was being destroyed. I think partly its cleanup code which attempts to delete the dest file on any form of failure, so any kind of transient problem and the shell overreacts by destroying the source. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847244085


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138883/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r639392907



##########
File path: core/pom.xml
##########
@@ -413,6 +413,16 @@
       <artifactId>hadoop-minikdc</artifactId>
       <scope>test</scope>
     </dependency>
+    <dependency>
+      <groupId>org.bouncycastle</groupId>
+      <artifactId>bcprov-jdk15on</artifactId>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.bouncycastle</groupId>
+      <artifactId>bcpkix-jdk15on</artifactId>
+      <scope>test</scope>
+    </dependency>

Review comment:
       Oops forgot it. Yes I should put it under `hadoop-3.2`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847240912


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43403/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847399687


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43409/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853452980


   **[Test build #139242 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139242/testReport)** for PR 30135 at commit [`e825730`](https://github.com/apache/spark/commit/e82573061bdaa160569222d749abb93eae8628f6).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-771308098


   Thank you for keeping working on this area.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847288345


   **[Test build #138881 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138881/testReport)** for PR 30135 at commit [`1a4cdeb`](https://github.com/apache/spark/commit/1a4cdeb18139387a50d49b1e9508e910107715a4).
    * This patch **fails Spark unit tests**.
    * This patch **does not merge cleanly**.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714745096


   BTW, thank you so much, @sunchao !


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-715572480


   I retriggered GitHub Action once more.
   
   @srowen and @sunchao . For renaming, shall we do in another PR because it will touch many scripts, too.
   - `hadoop-3.2` -> `hadoop-3`
   - `hadoop-2.7` -> `hadoop-2`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847417613


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43409/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-858135865


   **[Test build #139591 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139591/testReport)** for PR 30135 at commit [`7549c36`](https://github.com/apache/spark/commit/7549c369b57480e767d27a952d85df3001a337ba).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714796700


   What does it imply for compatibility with clusters? it works with Hadoop 3.2, earlier? we may need to rename the profile eventually to "hadoop-3" or something. Implicitly it's "Hadoop 3.2+"


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847445708


   **[Test build #138898 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138898/testReport)** for PR 30135 at commit [`c6e75c8`](https://github.com/apache/spark/commit/c6e75c814755372e9bf5036e0ebe111c2b3f8b61).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848408043


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138946/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r639390795



##########
File path: pom.xml
##########
@@ -197,6 +197,7 @@
     <maven-antrun.version>1.8</maven-antrun.version>
     <commons-crypto.version>1.1.0</commons-crypto.version>
     <commons-cli.version>1.2</commons-cli.version>
+    <bouncycastle.version>1.52</bouncycastle.version>

Review comment:
       Although this is a test case, this looks like too old because it's released on Mar, 2015. Do we need to align with to Hadoop 3.3.1? Can we use the latest one `1.68`?
   - https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r640197781



##########
File path: pom.xml
##########
@@ -197,6 +197,7 @@
     <maven-antrun.version>1.8</maven-antrun.version>
     <commons-crypto.version>1.1.0</commons-crypto.version>
     <commons-cli.version>1.2</commons-cli.version>
+    <bouncycastle.version>1.52</bouncycastle.version>

Review comment:
       bouncycastle was unshaded in https://issues.apache.org/jira/browse/HADOOP-15832. Reason is: "Leave bouncycastle unshaded because it's signed with a special Oracle certificate so it can be a custom JCE security provider"




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-859061487


   > maybe the build is picking up the old RC
   
   Yes that's exactly what happened :)
   
   For the regression, don't know the full context behind the original change but seems like a good thing to do, although a boolean flag returned might be less disruptive IMO :). Either way it is a simple fix from Spark side.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-854843591


   **[Test build #139342 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139342/testReport)** for PR 30135 at commit [`c6f1802`](https://github.com/apache/spark/commit/c6f1802749b52aaac89301c23a083d4c21c562aa).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714685922


   **[Test build #130175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130175/testReport)** for PR 30135 at commit [`1a4cdeb`](https://github.com/apache/spark/commit/1a4cdeb18139387a50d49b1e9508e910107715a4).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] steveloughran commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-895209346


   known 3.3.1 regressions. Not AFAIK


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-854942922


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139342/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-849953278


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43550/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-854911770


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43864/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-854942922


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139342/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-857850894


   **[Test build #139584 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139584/testReport)** for PR 30135 at commit [`e83e73f`](https://github.com/apache/spark/commit/e83e73f51a8da6d2776ab141aaad9c954c6c4b7d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] arghya18 commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
arghya18 commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-868536148


   @steveloughran Thanks for the suggestion. 
   Can anyone help me with steps (or Dockerfile) to change hadoop version in prebuilt spark docker image with Hadoop  if available handy for me to quickly progress else I will google, newbie in docker :)
   
   Thanks a lot.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-854911752


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43864/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853420021






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847445708


   **[Test build #138898 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138898/testReport)** for PR 30135 at commit [`c6e75c8`](https://github.com/apache/spark/commit/c6e75c814755372e9bf5036e0ebe111c2b3f8b61).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714744870


   The failure seems to be relevant since they are YARN test suite.
   ```
   Error:  Error: Total 118, Failed 0, Errors 3, Passed 115, Canceled 1
   Error:  Error during tests:
   Error:  	org.apache.spark.deploy.yarn.YarnClusterSuite
   Error:  	org.apache.spark.deploy.yarn.YarnShuffleAuthSuite
   Error:  	org.apache.spark.deploy.yarn.YarnShuffleIntegrationSuite
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847442685


   Thanks @viirya . I was scratching my head on why this happened.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847244052


   **[Test build #138883 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138883/testReport)** for PR 30135 at commit [`3d6a648`](https://github.com/apache/spark/commit/3d6a64848f55bea124c24d715281005b9f8c4596).
    * This patch **fails to build**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] arghya18 edited a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
arghya18 edited a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-872701489


   @dongjoon-hyun @steveloughran I was able to test my use case with Hadoop 3.3.1 and posted the result [HADOOP-17755](https://issues.apache.org/jira/browse/HADOOP-17755?focusedCommentId=17373213&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17373213)
   To my surprise the read is slower(with same resource and same config) in Hadoop 3.3.1 than Hadoop 3.2.0 without the mentioned issue. It is possible I am missing something. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714776989






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714783980


   I retriggered GitHub Action. Let's see.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] gengliangwang commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-864585303


   @dongjoon-hyun Thanks for taking care of the error!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao edited a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
sunchao edited a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-894007146


   I'm not aware of any issue at the moment - we've already been using this (although a slightly different internal Hadoop version) for a while now. @steveloughran can offer more inputs from Hadoop/S3 side.
   
   Besides that, I wanted to add https://github.com/apache/spark/pull/33160 so that users can build Spark with older versions of Hadoop which do not support shaded client. If people feel this is useful I can try to push it to the finish line. I also wanted to add some user documentation on compiling Spark against different versions of Hadoop (besides the default one).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847240912


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43403/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853380133






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-872736822


   Oh, if you are using ORC, please try to bring SPARK-35783. It's irrelevant to this Hadoop topic, but it helps you reduce the traffic.
   - https://github.com/apache/spark/pull/32923


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714783706


   Oh..


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] steveloughran commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-868515080


   @arghya18 
   1. we havent seen  HADOOP-17755 in any of our testing, unless it is HADOOP-16109
   2. still waiting on that JIRA for you to provide config details. Like I've said: no info, we close the JIRA.
   
   > wanted to check if the above issues mentioned will impact me if I manually build Spark 3.1.1 with Hadoop 3.3.0/3.3.1?
   
   well, what can anyone say? without info from you all we can say is WORKSFORME
   
   The best thing you can do to help everyone is checkout and build with Hadoop 3.3.1 (which does contain a lot of changes related to the committers, including from @dongjoon-hyun), and see if the problem is still there.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848209166


   **[Test build #138938 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138938/testReport)** for PR 30135 at commit [`263570c`](https://github.com/apache/spark/commit/263570c4b448e062e9b8454a9300ea4f7b0a9506).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847242862


   **[Test build #138883 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138883/testReport)** for PR 30135 at commit [`3d6a648`](https://github.com/apache/spark/commit/3d6a64848f55bea124c24d715281005b9f8c4596).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-864466090


   Hi, All. FYI,
   
   From Apache Hadoop 3.3.1, we reverted [HADOOP-16878](https://issues.apache.org/jira/browse/HADOOP-16878) as the last revert commit on `branch-3.3.1`.
   - https://github.com/apache/hadoop/commit/a3b9c37a397ad4188041dd80621bdeefc46885f2
   
   However, Apache Spark Jenkins seems to hit a flakiness issue.
   - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2/lastCompletedBuild/testReport/org.apache.spark.deploy.yarn/ClientSuite/distribute_jars_archive/history/
   
   ```
   org.apache.hadoop.fs.PathOperationException: 
   `Source (file:/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2/resource-managers/yarn/target/tmp/spark-703b8e99-63cc-4ba6-a9bc-25c7cae8f5f9/testJar9120517778809167117.jar) and destination (/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2/resource-managers/yarn/target/tmp/spark-703b8e99-63cc-4ba6-a9bc-25c7cae8f5f9/testJar9120517778809167117.jar)
   are equal in the copy command.': Operation not supported
   ```
   
   Please note that this is a flakiness. I'm investigating this.
   
   cc @gengliangwang since he is the release manager for Apache Spark 3.2.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714810305


   Yeah it should work with Hadoop 3.2+ clusters since [Hadoop maintains wire compatibility across minor releases. ](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html). +1 on renaming this to hadoop-3 (and probably rename `hadoop-2.7` to `hadoop-2` as well).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] steveloughran commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-872974884


   > To my surprise the read is slower(with same resource and same config) in Hadoop 3.3.1 than Hadoop 3.2.0 without the mentioned issue. It is possible I am missing something.
   
   Shouldn't happen. really shouldn't happen. We do not see that on our TCP-DS Benchmarks.
   
   The main way I could see this happening is if the seek policy hasn't switched to random on the first backwards seek. Explicitly set it.
   
   ```
   spark.hadoop.fs.s3a.experimental.fadvise random
   ```
   
   
   Hadoop 3.3.1 has a  stats collection API (IOStatisics) for filesystems, streams, etc. 
   * call toString() on a stream to get its stats, inc #of bytes discarded in seeks, streams aborted
   * do the same for the FS to get the aggregate stats. 
   
   
   high counts of bytes discarded and aborts are signs of bad seek policy.    
   
   set these two logs at debug and see what they say.
   ```
   org.apache.hadoop.fs.s3a.S3AInputStream
   org.apache.hadoop.fs.s3a.S3AStorageStatistics
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-771308098


   Thank you for keeping working on this area.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847310474






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714794421


   This seems to hit `bouncycastle`.
   ```
   $ build/sbt "yarn/testOnly *.YarnClusterSuite" -Pyarn -Phadoop-3.2
   ...
   [info] YarnClusterSuite:
   [info] org.apache.spark.deploy.yarn.YarnClusterSuite *** ABORTED *** (580 milliseconds)
   [info]   java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/org/bouncycastle/operator/OperatorCreationException
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853595736


   **[Test build #139275 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139275/testReport)** for PR 30135 at commit [`547ccb0`](https://github.com/apache/spark/commit/547ccb0fb4094982ba4dfb44f53b69775624b50d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-868190212


   We are preparing Apache Spark 3.2.0 with Hadoop 3.3.1, @arghya18 .
   - SPARK-35831 is fixed.
   - SPARK-35868 is also fixed.
   - SPARK-35878 has the PR from @steveloughran , too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r638151954



##########
File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3
##########
@@ -58,9 +58,10 @@ flatbuffers-java/1.9.0//flatbuffers-java-1.9.0.jar
 generex/1.0.2//generex-1.0.2.jar
 gson/2.2.4//gson-2.2.4.jar
 guava/14.0.1//guava-14.0.1.jar
-hadoop-client-api/3.2.2//hadoop-client-api-3.2.2.jar
-hadoop-client-runtime/3.2.2//hadoop-client-runtime-3.2.2.jar
-hadoop-yarn-server-web-proxy/3.2.2//hadoop-yarn-server-web-proxy-3.2.2.jar
+hadoop-client-api/3.3.1//hadoop-client-api-3.3.1.jar
+hadoop-client-runtime/3.3.1//hadoop-client-runtime-3.3.1.jar
+hadoop-shaded-guava/1.1.0//hadoop-shaded-guava-1.1.0.jar

Review comment:
       Yes - Hadoop 3.3.1 has switched to shaded guava.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-858031988


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139584/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r638151514



##########
File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3
##########
@@ -58,9 +58,10 @@ flatbuffers-java/1.9.0//flatbuffers-java-1.9.0.jar
 generex/1.0.2//generex-1.0.2.jar
 gson/2.2.4//gson-2.2.4.jar
 guava/14.0.1//guava-14.0.1.jar
-hadoop-client-api/3.2.2//hadoop-client-api-3.2.2.jar
-hadoop-client-runtime/3.2.2//hadoop-client-runtime-3.2.2.jar
-hadoop-yarn-server-web-proxy/3.2.2//hadoop-yarn-server-web-proxy-3.2.2.jar
+hadoop-client-api/3.3.1//hadoop-client-api-3.3.1.jar
+hadoop-client-runtime/3.3.1//hadoop-client-runtime-3.3.1.jar
+hadoop-shaded-guava/1.1.0//hadoop-shaded-guava-1.1.0.jar

Review comment:
       This is a new dependency?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-854935690


   **[Test build #139342 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139342/testReport)** for PR 30135 at commit [`c6f1802`](https://github.com/apache/spark/commit/c6f1802749b52aaac89301c23a083d4c21c562aa).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] arghya18 commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
arghya18 commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-872730959


   @dongjoon-hyun Thanks.. I am testing more jobs for further statistics. BDW I am testing this on ORC.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-872736822


   Oh, if you are using ORC, please try to bring SPARK-35783. It's irrelevant to Hadoop, but it helps you reduce the traffic.
   - https://github.com/apache/spark/pull/32923


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r640208302



##########
File path: pom.xml
##########
@@ -197,6 +197,7 @@
     <maven-antrun.version>1.8</maven-antrun.version>
     <commons-crypto.version>1.1.0</commons-crypto.version>
     <commons-cli.version>1.2</commons-cli.version>
+    <bouncycastle.version>1.52</bouncycastle.version>

Review comment:
       To be clear, bouncycastle was also unshaded in Hadoop 3.2.2 which Spark currently use, but since https://issues.apache.org/jira/browse/YARN-8448 it becomes a runtime dependency for RM and therefore required by Spark tests (e.g., `MiniYARNCluster`).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-894006737


   +1 for backporting!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853380133






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r639390101



##########
File path: core/pom.xml
##########
@@ -413,6 +413,16 @@
       <artifactId>hadoop-minikdc</artifactId>
       <scope>test</scope>
     </dependency>
+    <dependency>
+      <groupId>org.bouncycastle</groupId>
+      <artifactId>bcprov-jdk15on</artifactId>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.bouncycastle</groupId>
+      <artifactId>bcpkix-jdk15on</artifactId>
+      <scope>test</scope>
+    </dependency>

Review comment:
       Just a question. What happens for `hadoop-2.7` profile? Can we put this under `hadoop-3.2` profile?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-850008994


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139032/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-864466090


   Hi, All. FYI,
   
   From Apache Hadoop 3.3.1, we reverted [HADOOP-16878](https://issues.apache.org/jira/browse/HADOOP-16878) as the last revert commit on `branch-3.3.1`.
   - https://github.com/apache/hadoop/commit/a3b9c37a397ad4188041dd80621bdeefc46885f2
   
   However, Apache Spark Jenkins seems to hit a flakiness issue.
   - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2/lastCompletedBuild/testReport/org.apache.spark.deploy.yarn/ClientSuite/distribute_jars_archive/history/
   
   ```
   org.apache.hadoop.fs.PathOperationException: 
   `Source (file:/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2/resource-managers/yarn/target/tmp/spark-703b8e99-63cc-4ba6-a9bc-25c7cae8f5f9/testJar9120517778809167117.jar) and destination (/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2/resource-managers/yarn/target/tmp/spark-703b8e99-63cc-4ba6-a9bc-25c7cae8f5f9/testJar9120517778809167117.jar)
   are equal in the copy command.': Operation not supported
   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:403)
   ```
   
   Please note that this is a flakiness and at the same time the error log is pointing `FileUtil.java:403` which is the code before reverting. I'm investigating this.
   
   cc @gengliangwang since he is the release manager for Apache Spark 3.2.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-872729358


   Thank you for sharing, @arghya18 . It's interesting. The read statistic increase is also observed in my environment, but TPCDS 1TB on S3 parquet performance was faster for me. I'll keep tracking HADOOP-17755 together.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-771277246


   This is staled because Spark can't use Hadoop 3.3.0 as it is (see HADOOP-16080). The work may be resumed once Hadoop 3.3.1 is released.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848374341


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43466/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dbtsai commented on a change in pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #30135:
URL: https://github.com/apache/spark/pull/30135#discussion_r640159422



##########
File path: pom.xml
##########
@@ -197,6 +197,7 @@
     <maven-antrun.version>1.8</maven-antrun.version>
     <commons-crypto.version>1.1.0</commons-crypto.version>
     <commons-cli.version>1.2</commons-cli.version>
+    <bouncycastle.version>1.52</bouncycastle.version>

Review comment:
       Wonder why hadoop doesn't shade bouncycastle?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847471392


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43420/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-850004475


   **[Test build #139032 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139032/testReport)** for PR 30135 at commit [`c56cb14`](https://github.com/apache/spark/commit/c56cb142414562c6caec8d4604dc3f6e7a8688a3).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-864466090


   Hi, All. FYI,
   
   From Apache Hadoop 3.3.1, we reverted HADOOP-16878 as the last commit on `branch-3.3.1`.
   - https://github.com/apache/hadoop/commit/a3b9c37a397ad4188041dd80621bdeefc46885f2
   
   However, Apache Spark Jenkins seems to hit a flakiness issue.
   - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2/lastCompletedBuild/testReport/org.apache.spark.deploy.yarn/ClientSuite/distribute_jars_archive/history/
   
   ```
   org.apache.hadoop.fs.PathOperationException: `Source (file:/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2/resource-managers/yarn/target/tmp/spark-703b8e99-63cc-4ba6-a9bc-25c7cae8f5f9/testJar9120517778809167117.jar) and destination (/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2/resource-managers/yarn/target/tmp/spark-703b8e99-63cc-4ba6-a9bc-25c7cae8f5f9/testJar9120517778809167117.jar) are equal in the copy command.': Operation not supported
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848374341


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43466/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-848073474


   **[Test build #138938 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138938/testReport)** for PR 30135 at commit [`263570c`](https://github.com/apache/spark/commit/263570c4b448e062e9b8454a9300ea4f7b0a9506).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-854201334


   @dongjoon-hyun interesting. This is related to https://issues.apache.org/jira/browse/HADOOP-16878, and I thought it's already reverted from RC3 so I reverted my earlier change of handling it in Spark code. Let me check later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847348795


   **[Test build #138887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138887/testReport)** for PR 30135 at commit [`edcee70`](https://github.com/apache/spark/commit/edcee7021006b61e1470ad993fdf4d82a7f5470b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847440186


   We need to remove "Stale" label, otherwise the bot will close it. :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] arghya18 commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
arghya18 commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-872763060


   @dongjoon-hyun  Ok.. I will do that but I thought read and time increase is impact of s3a implemented on Hadoop 3.3.1 as that changed only after I upgraded from Hadoop 3.2.0 to Hadoop 3.3.1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-854198489


   Interesting. The same test fails in both Jenkins and GitHub Action. Is something changed at this new RC?
   - `ClientSuite.distribute jars archive`
   ```
   org.apache.hadoop.fs.PathOperationException: `Source (file:/home/jenkins/workspace/SparkPullRequestBuilder/target/tmp/spark-8a380e11-5074-471e-b9c8-f3f764e2c4e6/testJar948737110654121459.jar) and destination (/home/jenkins/workspace/SparkPullRequestBuilder/target/tmp/spark-8a380e11-5074-471e-b9c8-f3f764e2c4e6/testJar948737110654121459.jar) are equal in the copy command.': Operation not supported
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [WIP][SPARK-29250] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714708467


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34781/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-868284598


   Thank you for sharing, @arghya18 . 
   HADOOP-17755 sounds like read-side issue and Magic committer is write-side feature. I don't think they are related. If you hit a magic committer issue, please file a JIRA at Apache Hadoop community.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-857882361


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44111/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30135: [WIP][SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-714787384


   cc @srowen , @dbtsai 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853683723


   **[Test build #139275 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139275/testReport)** for PR 30135 at commit [`547ccb0`](https://github.com/apache/spark/commit/547ccb0fb4094982ba4dfb44f53b69775624b50d).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sunchao commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.0

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847204081


   @srowen Yes I think we can just document it for now. If the upgrade is successful, I'm inclined to make a new "hadoop-3" profile for the next release (3.2.0?) - probably will send an email to dev first to gather opinions on this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-858142554


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139591/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-853420021






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30135: [SPARK-29250][BUILD] Upgrade to Hadoop 3.3.1

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30135:
URL: https://github.com/apache/spark/pull/30135#issuecomment-847486841


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138898/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org