You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/06/27 14:58:56 UTC
[spark] branch master updated: [SPARK-44197][BUILD][FOLLOWUP] Update `IsolatedClientLoader` hadoop version
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 24960d84a9d [SPARK-44197][BUILD][FOLLOWUP] Update `IsolatedClientLoader` hadoop version
24960d84a9d is described below
commit 24960d84a9dac17728822f3e783335f221c49da3
Author: panbingkun <pb...@gmail.com>
AuthorDate: Tue Jun 27 07:58:44 2023 -0700
[SPARK-44197][BUILD][FOLLOWUP] Update `IsolatedClientLoader` hadoop version
### What changes were proposed in this pull request?
The pr aims to follow up SPARK-44197.
### Why are the changes needed?
When the Hadoop version that Spark relies on is upgraded from `3.3.5` to `3.3.6`, the corresponding versions in `IsolatedClientLoader` should also be upgraded synchronously.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GA.
Closes #41758 from panbingkun/SPARK-44197_FOLLOWUP.
Authored-by: panbingkun <pb...@gmail.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
assembly/README | 2 +-
resource-managers/kubernetes/integration-tests/README.md | 2 +-
.../scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/assembly/README b/assembly/README
index a380d8cb330..3dde243d3e6 100644
--- a/assembly/README
+++ b/assembly/README
@@ -9,4 +9,4 @@ This module is off by default. To activate it specify the profile in the command
If you need to build an assembly for a different version of Hadoop the
hadoop-version system property needs to be set as in this example:
- -Dhadoop.version=3.3.5
+ -Dhadoop.version=3.3.6
diff --git a/resource-managers/kubernetes/integration-tests/README.md b/resource-managers/kubernetes/integration-tests/README.md
index 2944c189ed4..909e5b652d4 100644
--- a/resource-managers/kubernetes/integration-tests/README.md
+++ b/resource-managers/kubernetes/integration-tests/README.md
@@ -129,7 +129,7 @@ properties to Maven. For example:
mvn integration-test -am -pl :spark-kubernetes-integration-tests_2.12 \
-Pkubernetes -Pkubernetes-integration-tests \
- -Phadoop-3 -Dhadoop.version=3.3.5 \
+ -Phadoop-3 -Dhadoop.version=3.3.6 \
-Dspark.kubernetes.test.sparkTgz=spark-3.0.0-SNAPSHOT-bin-example.tgz \
-Dspark.kubernetes.test.imageTag=sometag \
-Dspark.kubernetes.test.imageRepo=docker.io/somerepo \
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
index 64718a9d35c..2765e6af521 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
@@ -66,7 +66,7 @@ private[hive] object IsolatedClientLoader extends Logging {
case e: RuntimeException if e.getMessage.contains("hadoop") =>
// If the error message contains hadoop, it is probably because the hadoop
// version cannot be resolved.
- val fallbackVersion = "3.3.5"
+ val fallbackVersion = "3.3.6"
logWarning(s"Failed to resolve Hadoop artifacts for the version $hadoopVersion. We " +
s"will change the hadoop version from $hadoopVersion to $fallbackVersion and try " +
"again. It is recommended to set jars used by Hive metastore client through " +
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org