You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2020/02/11 17:14:57 UTC
[spark] branch branch-3.0 updated: [SPARK-30783] Exclude
hive-service-rpc
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 79c99d0 [SPARK-30783] Exclude hive-service-rpc
79c99d0 is described below
commit 79c99d0ee02de79df9700123a13d34afaaf21602
Author: Yin Huai <yh...@databricks.com>
AuthorDate: Wed Feb 12 00:12:45 2020 +0800
[SPARK-30783] Exclude hive-service-rpc
### What changes were proposed in this pull request?
Exclude hive-service-rpc from build.
### Why are the changes needed?
hive-service-rpc 2.3.6 and spark sql's thrift server module have duplicate classes. Leaving hive-service-rpc 2.3.6 in the class path means that spark can pick up classes defined in hive instead of its thrift server module, which can cause hard to debug runtime errors due to class loading order and compilation errors for applications depend on spark.
If you compare hive-service-rpc 2.3.6's jar (https://search.maven.org/remotecontent?filepath=org/apache/hive/hive-service-rpc/2.3.6/hive-service-rpc-2.3.6.jar) and spark thrift server's jar (e.g. https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-hive-thriftserver_2.12/3.0.0-SNAPSHOT/spark-hive-thriftserver_2.12-3.0.0-20200207.021914-364.jar), you will see that all of classes provided by hive-service-rpc-2.3.6.jar are covered by spark thrift server's jar. h [...]
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Existing tests.
Closes #27533 from yhuai/SPARK-30783.
Authored-by: Yin Huai <yh...@databricks.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
(cherry picked from commit ea626b6acf0de0ff3b0678372f30ba6f84ae2b09)
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 1 -
dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 1 -
pom.xml | 20 ++++++++++++++++++++
3 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-2.3 b/dev/deps/spark-deps-hadoop-2.7-hive-2.3
index 1b57250..4f4d8b1 100644
--- a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2.7-hive-2.3
@@ -87,7 +87,6 @@ hive-jdbc/2.3.6//hive-jdbc-2.3.6.jar
hive-llap-common/2.3.6//hive-llap-common-2.3.6.jar
hive-metastore/2.3.6//hive-metastore-2.3.6.jar
hive-serde/2.3.6//hive-serde-2.3.6.jar
-hive-service-rpc/2.3.6//hive-service-rpc-2.3.6.jar
hive-shims-0.23/2.3.6//hive-shims-0.23-2.3.6.jar
hive-shims-common/2.3.6//hive-shims-common-2.3.6.jar
hive-shims-scheduler/2.3.6//hive-shims-scheduler-2.3.6.jar
diff --git a/dev/deps/spark-deps-hadoop-3.2-hive-2.3 b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
index ffd2364..18e4246 100644
--- a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
@@ -86,7 +86,6 @@ hive-jdbc/2.3.6//hive-jdbc-2.3.6.jar
hive-llap-common/2.3.6//hive-llap-common-2.3.6.jar
hive-metastore/2.3.6//hive-metastore-2.3.6.jar
hive-serde/2.3.6//hive-serde-2.3.6.jar
-hive-service-rpc/2.3.6//hive-service-rpc-2.3.6.jar
hive-shims-0.23/2.3.6//hive-shims-0.23-2.3.6.jar
hive-shims-common/2.3.6//hive-shims-common-2.3.6.jar
hive-shims-scheduler/2.3.6//hive-shims-scheduler-2.3.6.jar
diff --git a/pom.xml b/pom.xml
index a8d6ac9..925fa28 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1453,6 +1453,11 @@
<artifactId>hive-service</artifactId>
</exclusion>
<exclusion>
+ <!-- All classes are covered by spark's hive-thriftserver module -->
+ <groupId>${hive.group}</groupId>
+ <artifactId>hive-service-rpc</artifactId>
+ </exclusion>
+ <exclusion>
<groupId>${hive.group}</groupId>
<artifactId>hive-shims</artifactId>
</exclusion>
@@ -1509,6 +1514,11 @@
<artifactId>hive-service</artifactId>
</exclusion>
<exclusion>
+ <!-- All classes are covered by spark's hive-thriftserver module -->
+ <groupId>${hive.group}</groupId>
+ <artifactId>hive-service-rpc</artifactId>
+ </exclusion>
+ <exclusion>
<groupId>${hive.group}</groupId>
<artifactId>hive-shims</artifactId>
</exclusion>
@@ -1762,6 +1772,11 @@
<artifactId>hive-service</artifactId>
</exclusion>
<exclusion>
+ <!-- All classes are covered by spark's hive-thriftserver module -->
+ <groupId>${hive.group}</groupId>
+ <artifactId>hive-service-rpc</artifactId>
+ </exclusion>
+ <exclusion>
<groupId>${hive.group}</groupId>
<artifactId>hive-shims</artifactId>
</exclusion>
@@ -1911,6 +1926,11 @@
<artifactId>groovy-all</artifactId>
</exclusion>
<!-- Begin of Hive 2.3 exclusion -->
+ <exclusion>
+ <!-- All classes are covered by spark's hive-thriftserver module -->
+ <groupId>${hive.group}</groupId>
+ <artifactId>hive-service-rpc</artifactId>
+ </exclusion>
<!-- parquet-hadoop-bundle:1.8.1 conflict with 1.10.1 -->
<exclusion>
<groupId>org.apache.parquet</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org