incubator-livy git commit: [LIVY-380] Livy should overwrite spark.sql.catalogImplementation if Hive classes are not present on classpath

[js...@apache.org]

Repository: incubator-livy
Updated Branches:
  refs/heads/master d4bd76f09 -> fc4a9e60a


[LIVY-380] Livy should overwrite spark.sql.catalogImplementation if Hive classes are not present on classpath

In SparkContextInitializer.spark2CreateContext, livy checks for whether catalog implementation is "hive" or "in-memory" to properly initialize the SparkSession.
If the config is set to "hive" (this is the case if livy.repl.enable-hive-context is set to "true"), livy will further check if hive classes are present or not.
It only invokes the enableHiveSupport method of SparkSession.Builder if hive classes are found.
If not, it proceeds to initializing a SparkSession without Hive support enabled.
However, in Spark's code base, when creating SparkSession, it checks for spark.sql.catalogImplementation to determine which session state to initialize.
If spark.sql.catalogImplementation is set to "hive", it will initialize a HiveSessionState.
However, since Livy has already checked that hive classes are not present, it should not go ahead invoking SparkSession.builder.getOrCreate without overwriting spark.sql.catalogImplementation to "in-memory".

Author: Min Shen <ms...@linkedin.com>

Closes #24 from Victsm/master.


Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/fc4a9e60
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/fc4a9e60
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/fc4a9e60

Branch: refs/heads/master
Commit: fc4a9e60aac5672c134b6eafe27e049cca067897
Parents: d4bd76f
Author: Min Shen <ms...@linkedin.com>
Authored: Thu Aug 3 08:48:04 2017 +0800
Committer: jerryshao <ss...@hortonworks.com>
Committed: Thu Aug 3 08:48:04 2017 +0800

----------------------------------------------------------------------
 .../main/scala/org/apache/livy/repl/SparkContextInitializer.scala  | 2 ++
 1 file changed, 2 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/fc4a9e60/repl/src/main/scala/org/apache/livy/repl/SparkContextInitializer.scala
----------------------------------------------------------------------
diff --git a/repl/src/main/scala/org/apache/livy/repl/SparkContextInitializer.scala b/repl/src/main/scala/org/apache/livy/repl/SparkContextInitializer.scala
index a4a57f9..4d47824 100644
--- a/repl/src/main/scala/org/apache/livy/repl/SparkContextInitializer.scala
+++ b/repl/src/main/scala/org/apache/livy/repl/SparkContextInitializer.scala
@@ -93,6 +93,8 @@ trait SparkContextInitializer extends Logging {
         spark = builder.getClass.getMethod("getOrCreate").invoke(builder)
         info("Created Spark session (with Hive support).")
       } else {
+        builder.getClass.getMethod("config", classOf[String], classOf[String])
+          .invoke(builder, "spark.sql.catalogImplementation", "in-memory")
         spark = builder.getClass.getMethod("getOrCreate").invoke(builder)
         info("Created Spark session.")
       }