[jira] [Created] (SPARK-33376) Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader

["Chao Sun (Jira)" <ji...@apache.org>]

Chao Sun created SPARK-33376:
--------------------------------

             Summary: Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
                 Key: SPARK-33376
                 URL: https://issues.apache.org/jira/browse/SPARK-33376
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.0.1
            Reporter: Chao Sun


Currently, when initializing {{IsolatedClientLoader}}, ppl can specify to either share Hadoop classes from Spark or not. In the latter case it's supposed to only loads the Hadoop classes from the Hive jars themselves.

However this feature is currently used in two cases: 1) unit tests, 2) when the Hadoop version defined in Maven can not be found when {{spark.sql.hive.metastore.jars == "maven"}}. Also when {{sharesHadoopClasses}} is false, it isn't really only using Hadoop classes from Hive jars: Spark also download {{hadoop-client}} jar and put it together with the Hive jars, and the Hadoop version used by {{hadoop-client}} is the same version used by Spark itself. This could potentially cause issues because we are mixing two versions of Hadoop jars in the classpath.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org