You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ma...@apache.org on 2015/07/21 22:12:51 UTC
spark git commit: [SPARK-9206] [SQL] Fix HiveContext classloading for
GCS connector.
Repository: spark
Updated Branches:
refs/heads/master 60c0ce134 -> c07838b5a
[SPARK-9206] [SQL] Fix HiveContext classloading for GCS connector.
IsolatedClientLoader.isSharedClass includes all of com.google.\*, presumably
for Guava, protobuf, and/or other shared Google libraries, but needs to
count com.google.cloud.\* as "hive classes" when determining which ClassLoader
to use. Otherwise, things like HiveContext.parquetFile will throw a
ClassCastException when fs.defaultFS is set to a Google Cloud Storage (gs://)
path. On StackOverflow: http://stackoverflow.com/questions/31478955
EDIT: Adding yhuai who worked on the relevant classloading isolation pieces.
Author: Dennis Huo <dh...@google.com>
Closes #7549 from dennishuo/dhuo-fix-hivecontext-gcs and squashes the following commits:
1f8db07 [Dennis Huo] Fix HiveContext classloading for GCS connector.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c07838b5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c07838b5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c07838b5
Branch: refs/heads/master
Commit: c07838b5a9cdf96c0f49055ea1c397e0f0e915d2
Parents: 60c0ce1
Author: Dennis Huo <dh...@google.com>
Authored: Tue Jul 21 13:12:11 2015 -0700
Committer: Michael Armbrust <mi...@databricks.com>
Committed: Tue Jul 21 13:12:11 2015 -0700
----------------------------------------------------------------------
.../org/apache/spark/sql/hive/client/IsolatedClientLoader.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/c07838b5/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
----------------------------------------------------------------------
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
index 3d609a6..97fb981 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
@@ -125,7 +125,7 @@ private[hive] class IsolatedClientLoader(
name.contains("log4j") ||
name.startsWith("org.apache.spark.") ||
name.startsWith("scala.") ||
- name.startsWith("com.google") ||
+ (name.startsWith("com.google") && !name.startsWith("com.google.cloud")) ||
name.startsWith("java.lang.") ||
name.startsWith("java.net") ||
sharedPrefixes.exists(name.startsWith)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org