You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by li...@apache.org on 2017/03/04 02:44:37 UTC
spark git commit: [SPARK-19084][SQL] Ensure context class loader is
set when initializing Hive.
Repository: spark
Updated Branches:
refs/heads/master a6a7a95e2 -> 9e5b4ce72
[SPARK-19084][SQL] Ensure context class loader is set when initializing Hive.
A change in Hive 2.2 (most probably HIVE-13149) causes this code path to fail,
since the call to "state.getConf.setClassLoader" does not actually change the
context's class loader. Spark doesn't yet officially support Hive 2.2, but some
distribution-specific metastore client libraries may have that change (as certain
versions of CDH already do), and this also makes it easier to support 2.2 when it
comes out.
Tested with existing unit tests; we've also used this patch extensively with Hive
metastore client jars containing the offending patch.
Author: Marcelo Vanzin <va...@cloudera.com>
Closes #17154 from vanzin/SPARK-19804.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9e5b4ce7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9e5b4ce7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9e5b4ce7
Branch: refs/heads/master
Commit: 9e5b4ce727cf262a14a411efded85ee1e50a88ed
Parents: a6a7a95
Author: Marcelo Vanzin <va...@cloudera.com>
Authored: Fri Mar 3 18:44:31 2017 -0800
Committer: Xiao Li <ga...@gmail.com>
Committed: Fri Mar 3 18:44:31 2017 -0800
----------------------------------------------------------------------
.../apache/spark/sql/hive/client/HiveClientImpl.scala | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/9e5b4ce7/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
----------------------------------------------------------------------
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index 8f98c8f..7acaa9a 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -269,16 +269,21 @@ private[hive] class HiveClientImpl(
*/
def withHiveState[A](f: => A): A = retryLocked {
val original = Thread.currentThread().getContextClassLoader
- // Set the thread local metastore client to the client associated with this HiveClientImpl.
- Hive.set(client)
+ val originalConfLoader = state.getConf.getClassLoader
// The classloader in clientLoader could be changed after addJar, always use the latest
- // classloader
+ // classloader. We explicitly set the context class loader since "conf.setClassLoader" does
+ // not do that, and the Hive client libraries may need to load classes defined by the client's
+ // class loader.
+ Thread.currentThread().setContextClassLoader(clientLoader.classLoader)
state.getConf.setClassLoader(clientLoader.classLoader)
+ // Set the thread local metastore client to the client associated with this HiveClientImpl.
+ Hive.set(client)
// setCurrentSessionState will use the classLoader associated
// with the HiveConf in `state` to override the context class loader of the current
// thread.
shim.setCurrentSessionState(state)
val ret = try f finally {
+ state.getConf.setClassLoader(originalConfLoader)
Thread.currentThread().setContextClassLoader(original)
HiveCatalogMetrics.incrementHiveClientCalls(1)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org