You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/09 19:35:48 UTC
[GitHub] [spark] sunchao opened a new pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
sunchao opened a new pull request #34855:
URL: https://github.com/apache/spark/pull/34855
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Upgrade to Hadoop 3.3.2.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-1061262634
Re-triggered via empty commit. I did it manually by clicking the "Re-run all jobs" button which wasn't reflected here somehow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r822266139
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -69,7 +69,7 @@ private[hive] object IsolatedClientLoader extends Logging {
// If the error message contains hadoop, it is probably because the hadoop
// version cannot be resolved.
val fallbackVersion = if (VersionUtils.isHadoop3) {
- "3.3.1"
+ "3.3.2"
} else {
"2.7.4"
Review comment:
That sounds like independent improvement idea. Could you file a JIRA for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990199407
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146048/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-1058739724
hmm somehow `YarnClustereSuite` started failing after 3.3.2. I'll need to check what caused the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-1060074165
Is there any update, @sunchao ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r821316594
##########
File path: dev/deps/spark-deps-hadoop-3-hive-2.3
##########
@@ -205,9 +205,13 @@ objenesis/3.2//objenesis-3.2.jar
okhttp/3.12.12//okhttp-3.12.12.jar
okio/1.14.0//okio-1.14.0.jar
opencsv/2.3//opencsv-2.3.jar
+opentracing-api/0.33.0//opentracing-api-0.33.0.jar
+opentracing-noop/0.33.0//opentracing-noop-0.33.0.jar
+opentracing-util/0.33.0//opentracing-util-0.33.0.jar
orc-core/1.7.3//orc-core-1.7.3.jar
orc-mapreduce/1.7.3//orc-mapreduce-1.7.3.jar
orc-shims/1.7.3//orc-shims-1.7.3.jar
+org.jacoco.agent/0.8.5/runtime/org.jacoco.agent-0.8.5-runtime.jar
Review comment:
Are you sure it's test only? Didn't think those appeared in these drops file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990197495
**[Test build #146048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146048/testReport)** for PR 34855 at commit [`997590e`](https://github.com/apache/spark/commit/997590e87ed1c644dc1a68929eb4ecba106c2d03).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990199374
**[Test build #146048 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146048/testReport)** for PR 34855 at commit [`997590e`](https://github.com/apache/spark/commit/997590e87ed1c644dc1a68929eb4ecba106c2d03).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r767148632
##########
File path: pom.xml
##########
@@ -120,7 +120,7 @@
<sbt.project.name>spark</sbt.project.name>
<slf4j.version>1.7.30</slf4j.version>
<log4j.version>1.2.17</log4j.version>
- <hadoop.version>3.3.1</hadoop.version>
+ <hadoop.version>3.3.2</hadoop.version>
Review comment:
should we update https://github.com/apache/spark/pull/34830#discussion_r767148540 together?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r820964517
##########
File path: dev/deps/spark-deps-hadoop-3-hive-2.3
##########
@@ -205,9 +205,13 @@ objenesis/3.2//objenesis-3.2.jar
okhttp/3.12.12//okhttp-3.12.12.jar
okio/1.14.0//okio-1.14.0.jar
opencsv/2.3//opencsv-2.3.jar
+opentracing-api/0.33.0//opentracing-api-0.33.0.jar
+opentracing-noop/0.33.0//opentracing-noop-0.33.0.jar
+opentracing-util/0.33.0//opentracing-util-0.33.0.jar
orc-core/1.7.3//orc-core-1.7.3.jar
orc-mapreduce/1.7.3//orc-mapreduce-1.7.3.jar
orc-shims/1.7.3//orc-shims-1.7.3.jar
+org.jacoco.agent/0.8.5/runtime/org.jacoco.agent-0.8.5-runtime.jar
Review comment:
jacoco is Java code coverage library, I was surprised that it would become a dependency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-1061022437
> It seems to pass locally. Could you re-trigger the test simply, @sunchao ?
>
> ```
> [info] YarnClusterSuite:
> [info] - run Spark in yarn-client mode (10 seconds, 131 milliseconds)
> [info] - run Spark in yarn-cluster mode (9 seconds, 90 milliseconds)
> [info] - run Spark in yarn-client mode with unmanaged am (8 seconds, 78 milliseconds)
> [info] - run Spark in yarn-client mode with different configurations, ensuring redaction (10 seconds, 102 milliseconds)
> [info] - run Spark in yarn-cluster mode with different configurations, ensuring redaction (10 seconds, 96 milliseconds)
> [info] - yarn-cluster should respect conf overrides in SparkHadoopUtil (SPARK-16414, SPARK-23630) (9 seconds, 116 milliseconds)
> [info] - SPARK-35672: run Spark in yarn-client mode with additional jar using URI scheme 'local' (10 seconds, 111 milliseconds)
> [info] - SPARK-35672: run Spark in yarn-cluster mode with additional jar using URI scheme 'local' (9 seconds, 96 milliseconds)
> [info] - SPARK-35672: run Spark in yarn-client mode with additional jar using URI scheme 'local' and gateway-replacement path (8 seconds, 79 milliseconds)
> [info] - SPARK-35672: run Spark in yarn-cluster mode with additional jar using URI scheme 'local' and gateway-replacement path (9 seconds, 90 milliseconds)
> [info] - SPARK-35672: run Spark in yarn-cluster mode with additional jar using URI scheme 'local' and gateway-replacement path containing an environment variable (9 seconds, 100 milliseconds)
> ...
> ```
I manually test with mvn locally, and there will be UT failed:
```
YarnClusterSuite:
- run Spark in yarn-client mode
- run Spark in yarn-cluster mode
- run Spark in yarn-client mode with unmanaged am
- run Spark in yarn-client mode with different configurations, ensuring redaction
- run Spark in yarn-cluster mode with different configurations, ensuring redaction
- yarn-cluster should respect conf overrides in SparkHadoopUtil (SPARK-16414, SPARK-23630)
- SPARK-35672: run Spark in yarn-client mode with additional jar using URI scheme 'local'
- SPARK-35672: run Spark in yarn-cluster mode with additional jar using URI scheme 'local'
- SPARK-35672: run Spark in yarn-client mode with additional jar using URI scheme 'local' and gateway-replacement path
- SPARK-35672: run Spark in yarn-cluster mode with additional jar using URI scheme 'local' and gateway-replacement path
- SPARK-35672: run Spark in yarn-cluster mode with additional jar using URI scheme 'local' and gateway-replacement path containing an environment variable
- SPARK-35672: run Spark in yarn-client mode with additional jar using URI scheme 'file'
- SPARK-35672: run Spark in yarn-cluster mode with additional jar using URI scheme 'file'
- run Spark in yarn-cluster mode unsuccessfully
- run Spark in yarn-cluster mode failure after sc initialized
- run Python application in yarn-client mode *** FAILED ***
LOST did not equal FINISHED SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/xxx/spark-source/assembly/target/scala-2.12/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/xxx/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] (BaseYarnClusterSuite.scala:233)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990318317
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r766873294
##########
File path: pom.xml
##########
@@ -309,6 +309,17 @@
</extraJavaTestArgs>
</properties>
<repositories>
+ <repository>
Review comment:
Yes, will remove this section once the official 3.3.2 release is out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r821215672
##########
File path: dev/deps/spark-deps-hadoop-3-hive-2.3
##########
@@ -205,9 +205,13 @@ objenesis/3.2//objenesis-3.2.jar
okhttp/3.12.12//okhttp-3.12.12.jar
okio/1.14.0//okio-1.14.0.jar
opencsv/2.3//opencsv-2.3.jar
+opentracing-api/0.33.0//opentracing-api-0.33.0.jar
+opentracing-noop/0.33.0//opentracing-noop-0.33.0.jar
+opentracing-util/0.33.0//opentracing-util-0.33.0.jar
orc-core/1.7.3//orc-core-1.7.3.jar
orc-mapreduce/1.7.3//orc-mapreduce-1.7.3.jar
orc-shims/1.7.3//orc-shims-1.7.3.jar
+org.jacoco.agent/0.8.5/runtime/org.jacoco.agent-0.8.5-runtime.jar
Review comment:
Hmm let me check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r766850706
##########
File path: dev/deps/spark-deps-hadoop-3-hive-2.3
##########
@@ -189,9 +189,13 @@ objenesis/2.6//objenesis-2.6.jar
okhttp/3.12.12//okhttp-3.12.12.jar
okio/1.14.0//okio-1.14.0.jar
opencsv/2.3//opencsv-2.3.jar
+opentracing-api/0.33.0//opentracing-api-0.33.0.jar
Review comment:
The new dependencies look OK w.r.t licensing.
##########
File path: pom.xml
##########
@@ -309,6 +309,17 @@
</extraJavaTestArgs>
</properties>
<repositories>
+ <repository>
Review comment:
We'd remove this before merging? after it's released
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r767175899
##########
File path: pom.xml
##########
@@ -120,7 +120,7 @@
<sbt.project.name>spark</sbt.project.name>
<slf4j.version>1.7.30</slf4j.version>
<log4j.version>1.2.17</log4j.version>
- <hadoop.version>3.3.1</hadoop.version>
+ <hadoop.version>3.3.2</hadoop.version>
Review comment:
Good point. Will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r767178255
##########
File path: pom.xml
##########
@@ -120,7 +120,7 @@
<sbt.project.name>spark</sbt.project.name>
<slf4j.version>1.7.30</slf4j.version>
<log4j.version>1.2.17</log4j.version>
- <hadoop.version>3.3.1</hadoop.version>
+ <hadoop.version>3.3.2</hadoop.version>
Review comment:
> should we update [#34830 (comment)](https://github.com/apache/spark/pull/34830#discussion_r767148540) together?
+1 on. this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990237945
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50523/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r767175899
##########
File path: pom.xml
##########
@@ -120,7 +120,7 @@
<sbt.project.name>spark</sbt.project.name>
<slf4j.version>1.7.30</slf4j.version>
<log4j.version>1.2.17</log4j.version>
- <hadoop.version>3.3.1</hadoop.version>
+ <hadoop.version>3.3.2</hadoop.version>
Review comment:
Good point. Will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990387080
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146050/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r767148632
##########
File path: pom.xml
##########
@@ -120,7 +120,7 @@
<sbt.project.name>spark</sbt.project.name>
<slf4j.version>1.7.30</slf4j.version>
<log4j.version>1.2.17</log4j.version>
- <hadoop.version>3.3.1</hadoop.version>
+ <hadoop.version>3.3.2</hadoop.version>
Review comment:
should we update https://github.com/apache/spark/pull/34830#discussion_r767148540 together?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990385613
**[Test build #146050 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146050/testReport)** for PR 34855 at commit [`be530f0`](https://github.com/apache/spark/commit/be530f080c289ae0553c1a72a0f3bf4adfb291f0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-1062500259
@srowen , do you have any other concerns? Or, the last issue (LICENSE) is resolved and we are good to go?
<img width="849" alt="Screen Shot 2022-03-08 at 6 45 14 PM" src="https://user-images.githubusercontent.com/9700541/157362706-d75a3785-73c8-4368-adc4-ff42cb732519.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-1061038658
Thanks for helping to verify this @LuciferYang @dongjoon-hyun ! yea it seems a bit flaky. I tried to look into the YARN logs locally but couldn't find anything interesting. Let me try to re-trigger the GitHub workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-1061036680
Ya, your failed test case is already passed in the original GitHub Action run. Maybe you might hit some flaky test case failure which is still in this module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r821332614
##########
File path: dev/deps/spark-deps-hadoop-3-hive-2.3
##########
@@ -121,10 +120,11 @@ janino/3.0.16//janino-3.0.16.jar
javassist/3.25.0-GA//javassist-3.25.0-GA.jar
javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar
javolution/5.5.1//javolution-5.5.1.jar
+jaxb-api/2.2.11//jaxb-api-2.2.11.jar
jaxb-runtime/2.3.2//jaxb-runtime-2.3.2.jar
jcl-over-slf4j/1.7.32//jcl-over-slf4j-1.7.32.jar
jdo-api/3.0.1//jdo-api-3.0.1.jar
-jdom/1.1//jdom-1.1.jar
+jdom2/2.0.6//jdom2-2.0.6.jar
Review comment:
Updated. Can you check if it looks OK?
##########
File path: dev/deps/spark-deps-hadoop-3-hive-2.3
##########
@@ -121,10 +120,11 @@ janino/3.0.16//janino-3.0.16.jar
javassist/3.25.0-GA//javassist-3.25.0-GA.jar
javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar
javolution/5.5.1//javolution-5.5.1.jar
+jaxb-api/2.2.11//jaxb-api-2.2.11.jar
Review comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990284671
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50523/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-1061030993
Oh...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r821202723
##########
File path: dev/deps/spark-deps-hadoop-3-hive-2.3
##########
@@ -205,9 +205,13 @@ objenesis/3.2//objenesis-3.2.jar
okhttp/3.12.12//okhttp-3.12.12.jar
okio/1.14.0//okio-1.14.0.jar
opencsv/2.3//opencsv-2.3.jar
+opentracing-api/0.33.0//opentracing-api-0.33.0.jar
+opentracing-noop/0.33.0//opentracing-noop-0.33.0.jar
+opentracing-util/0.33.0//opentracing-util-0.33.0.jar
orc-core/1.7.3//orc-core-1.7.3.jar
orc-mapreduce/1.7.3//orc-mapreduce-1.7.3.jar
orc-shims/1.7.3//orc-shims-1.7.3.jar
+org.jacoco.agent/0.8.5/runtime/org.jacoco.agent-0.8.5-runtime.jar
Review comment:
oh, I missed this. Do you want to exclude this as an workaround, @sunchao ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-1061258741
I believe this is almost one. Could you review this once more, @viirya , @srowen , @HyukjinKwon , @AngersZhuuuu , @LuciferYang ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r821323085
##########
File path: dev/deps/spark-deps-hadoop-3-hive-2.3
##########
@@ -121,10 +120,11 @@ janino/3.0.16//janino-3.0.16.jar
javassist/3.25.0-GA//javassist-3.25.0-GA.jar
javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar
javolution/5.5.1//javolution-5.5.1.jar
+jaxb-api/2.2.11//jaxb-api-2.2.11.jar
Review comment:
this is CDDL 1.0 - see below for license tasks
##########
File path: dev/deps/spark-deps-hadoop-3-hive-2.3
##########
@@ -121,10 +120,11 @@ janino/3.0.16//janino-3.0.16.jar
javassist/3.25.0-GA//javassist-3.25.0-GA.jar
javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar
javolution/5.5.1//javolution-5.5.1.jar
+jaxb-api/2.2.11//jaxb-api-2.2.11.jar
jaxb-runtime/2.3.2//jaxb-runtime-2.3.2.jar
jcl-over-slf4j/1.7.32//jcl-over-slf4j-1.7.32.jar
jdo-api/3.0.1//jdo-api-3.0.1.jar
-jdom/1.1//jdom-1.1.jar
+jdom2/2.0.6//jdom2-2.0.6.jar
Review comment:
I'm checking licenses of these too - this is basically 3-clause BSD and could be listed as such in the LICENSE/NOTICE file with other 3-clause BSD. I think we might need the license file stored in licenses/ - https://raw.githubusercontent.com/hunterhacker/jdom/master/LICENSE.txt
##########
File path: hadoop-cloud/pom.xml
##########
@@ -267,6 +267,13 @@
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
+ <exclusion>
+ <!--
+ This is a code coverage library introduced by aliyun-java-sdk-core, only for testing
+ -->
+ <groupId>org.jacoco</groupId>
+ <artifactId>org.jacoco.agent</artifactId>
Review comment:
OK yeah I think we can safely exclude it. Probably a mistake that it wasn't test-scope in the original artifact
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r822244027
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -69,7 +69,7 @@ private[hive] object IsolatedClientLoader extends Logging {
// If the error message contains hadoop, it is probably because the hadoop
// version cannot be resolved.
val fallbackVersion = if (VersionUtils.isHadoop3) {
- "3.3.1"
+ "3.3.2"
} else {
"2.7.4"
Review comment:
By the way, can we read the hadoop version of the project configuration here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-1062529550
Thank you, @sunchao , @viirya , @srowen , @HyukjinKwon , @LuciferYang , @AngersZhuuuu .
Merged to master for Apache Spark 3.3.
Also, cc @MaxGekk since he is the release manager for Apache Spark 3.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-1062529550
Thank you, @sunchao , @srowen , @HyukjinKwon , @LuciferYang , @AngersZhuuuu .
Merged to master for Apache Spark 3.3.
Also, cc @MaxGekk since he is the release manager for Apache Spark 3.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r822266733
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -69,7 +69,7 @@ private[hive] object IsolatedClientLoader extends Logging {
// If the error message contains hadoop, it is probably because the hadoop
// version cannot be resolved.
val fallbackVersion = if (VersionUtils.isHadoop3) {
- "3.3.1"
+ "3.3.2"
} else {
"2.7.4"
Review comment:
> That sounds like independent improvement idea. Could you file a JIRA for that?
Yea, will try to do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #34855:
URL: https://github.com/apache/spark/pull/34855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r822357513
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -69,7 +69,7 @@ private[hive] object IsolatedClientLoader extends Logging {
// If the error message contains hadoop, it is probably because the hadoop
// version cannot be resolved.
val fallbackVersion = if (VersionUtils.isHadoop3) {
- "3.3.1"
+ "3.3.2"
} else {
"2.7.4"
Review comment:
I'm not sure this is easy since in this case the Hadoop version specified via `hadoop.version` in `pom.xml` is customized and is not `3.3.2`, which is why it can't be fetched from Maven.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r821317822
##########
File path: dev/deps/spark-deps-hadoop-3-hive-2.3
##########
@@ -205,9 +205,13 @@ objenesis/3.2//objenesis-3.2.jar
okhttp/3.12.12//okhttp-3.12.12.jar
okio/1.14.0//okio-1.14.0.jar
opencsv/2.3//opencsv-2.3.jar
+opentracing-api/0.33.0//opentracing-api-0.33.0.jar
+opentracing-noop/0.33.0//opentracing-noop-0.33.0.jar
+opentracing-util/0.33.0//opentracing-util-0.33.0.jar
orc-core/1.7.3//orc-core-1.7.3.jar
orc-mapreduce/1.7.3//orc-mapreduce-1.7.3.jar
orc-shims/1.7.3//orc-shims-1.7.3.jar
+org.jacoco.agent/0.8.5/runtime/org.jacoco.agent-0.8.5-runtime.jar
Review comment:
I tracked to this commit which introduced it: https://github.com/aliyun/aliyun-openapi-java-sdk/commit/e0d21a34ca4e5d3caa82bd1ebdf8e60e8b4cd5ee, which looks only used in test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-1061031321
https://github.com/apache/spark/pull/34855#issuecomment-1061022437
Sorry, this may be my bad. I re-run it twice and succeeded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-1061259161
To @sunchao . It seems that it's not re-triggered yet. You may want to add an empty commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-1061257766
Sure @dongjoon-hyun . Just re-triggered the jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990213215
**[Test build #146050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146050/testReport)** for PR 34855 at commit [`be530f0`](https://github.com/apache/spark/commit/be530f080c289ae0553c1a72a0f3bf4adfb291f0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r767178255
##########
File path: pom.xml
##########
@@ -120,7 +120,7 @@
<sbt.project.name>spark</sbt.project.name>
<slf4j.version>1.7.30</slf4j.version>
<log4j.version>1.2.17</log4j.version>
- <hadoop.version>3.3.1</hadoop.version>
+ <hadoop.version>3.3.2</hadoop.version>
Review comment:
> should we update [#34830 (comment)](https://github.com/apache/spark/pull/34830#discussion_r767148540) together?
+1 on. this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990302961
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50525/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990213215
**[Test build #146050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146050/testReport)** for PR 34855 at commit [`be530f0`](https://github.com/apache/spark/commit/be530f080c289ae0553c1a72a0f3bf4adfb291f0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990250688
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50525/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990318317
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990197495
**[Test build #146048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146048/testReport)** for PR 34855 at commit [`997590e`](https://github.com/apache/spark/commit/997590e87ed1c644dc1a68929eb4ecba106c2d03).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990199407
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146048/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34855: [WIP][SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34855:
URL: https://github.com/apache/spark/pull/34855#issuecomment-990387080
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146050/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34855: [SPARK-37600][BUILD] Upgrade to Hadoop 3.3.2
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34855:
URL: https://github.com/apache/spark/pull/34855#discussion_r821315839
##########
File path: dev/deps/spark-deps-hadoop-3-hive-2.3
##########
@@ -205,9 +205,13 @@ objenesis/3.2//objenesis-3.2.jar
okhttp/3.12.12//okhttp-3.12.12.jar
okio/1.14.0//okio-1.14.0.jar
opencsv/2.3//opencsv-2.3.jar
+opentracing-api/0.33.0//opentracing-api-0.33.0.jar
+opentracing-noop/0.33.0//opentracing-noop-0.33.0.jar
+opentracing-util/0.33.0//opentracing-util-0.33.0.jar
orc-core/1.7.3//orc-core-1.7.3.jar
orc-mapreduce/1.7.3//orc-mapreduce-1.7.3.jar
orc-shims/1.7.3//orc-shims-1.7.3.jar
+org.jacoco.agent/0.8.5/runtime/org.jacoco.agent-0.8.5-runtime.jar
Review comment:
This is a test-only dependency brought in by `aliyun-java-sdk-core` in `hadoop-cloud-storage`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org