You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yin Huai (Jira)" <ji...@apache.org> on 2020/02/10 22:46:00 UTC
[jira] [Updated] (SPARK-30783) Hive 2.3 profile should exclude
hive-service-rpc
[ https://issues.apache.org/jira/browse/SPARK-30783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yin Huai updated SPARK-30783:
-----------------------------
Description:
hive-service-rpc 2.3.6 and spark sql's thrift server module have duplicate classes. Leaving hive-service-rpc 2.3.6 in the class path means that spark can pick up classes defined in hive instead of its thrift server module, which can cause hard to debug runtime errors due to class loading order and compilation errors for applications depend on spark.
If you compare hive-service-rpc 2.3.6's jar ([https://search.maven.org/remotecontent?filepath=org/apache/hive/hive-service-rpc/2.3.6/hive-service-rpc-2.3.6.jar]) and spark thrift server's jar (e.g. [https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-hive-thriftserver_2.12/3.0.0-SNAPSHOT/spark-hive-thriftserver_2.12-3.0.0-20200207.021914-364.jar),] you will see that all of classes provided by hive-service-rpc-2.3.6.jar are covered by spark thrift server's jar. I am attaching the list of jar contents for your reference.
was:hive-service-rpc 2.3.6 and spark sql's thrift server module have duplicate classes. Leaving hive-service-rpc 2.3.6 in the class path means that spark can pick up classes defined in hive instead of its thrift server module, which can cause hard to debug runtime errors due to class loading order and compilation errors for applications depend on spark.
> Hive 2.3 profile should exclude hive-service-rpc
> ------------------------------------------------
>
> Key: SPARK-30783
> URL: https://issues.apache.org/jira/browse/SPARK-30783
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Yin Huai
> Assignee: Yin Huai
> Priority: Blocker
>
> hive-service-rpc 2.3.6 and spark sql's thrift server module have duplicate classes. Leaving hive-service-rpc 2.3.6 in the class path means that spark can pick up classes defined in hive instead of its thrift server module, which can cause hard to debug runtime errors due to class loading order and compilation errors for applications depend on spark.
>
> If you compare hive-service-rpc 2.3.6's jar ([https://search.maven.org/remotecontent?filepath=org/apache/hive/hive-service-rpc/2.3.6/hive-service-rpc-2.3.6.jar]) and spark thrift server's jar (e.g. [https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-hive-thriftserver_2.12/3.0.0-SNAPSHOT/spark-hive-thriftserver_2.12-3.0.0-20200207.021914-364.jar),] you will see that all of classes provided by hive-service-rpc-2.3.6.jar are covered by spark thrift server's jar. I am attaching the list of jar contents for your reference.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org