You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by an...@apache.org on 2016/05/20 06:44:18 UTC
spark git commit: [SPARK-15417][SQL][PYTHON] PySpark shell always
uses in-memory catalog
Repository: spark
Updated Branches:
refs/heads/master 257375019 -> c32b1b162
[SPARK-15417][SQL][PYTHON] PySpark shell always uses in-memory catalog
## What changes were proposed in this pull request?
There is no way to use the Hive catalog in `pyspark-shell`. This is because we used to create a `SparkContext` before calling `SparkSession.enableHiveSupport().getOrCreate()`, which just gets the existing `SparkContext` instead of creating a new one. As a result, `spark.sql.catalogImplementation` was never propagated.
## How was this patch tested?
Manual.
Author: Andrew Or <an...@databricks.com>
Closes #13203 from andrewor14/fix-pyspark-shell.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c32b1b16
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c32b1b16
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c32b1b16
Branch: refs/heads/master
Commit: c32b1b162e7e5ecc5c823f79ba9f23cbd1407dbf
Parents: 2573750
Author: Andrew Or <an...@databricks.com>
Authored: Thu May 19 23:44:10 2016 -0700
Committer: Andrew Or <an...@databricks.com>
Committed: Thu May 19 23:44:10 2016 -0700
----------------------------------------------------------------------
python/pyspark/shell.py | 8 +++++---
python/pyspark/sql/session.py | 6 ++++++
2 files changed, 11 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/c32b1b16/python/pyspark/shell.py
----------------------------------------------------------------------
diff --git a/python/pyspark/shell.py b/python/pyspark/shell.py
index adaa3b5..ef46d30 100644
--- a/python/pyspark/shell.py
+++ b/python/pyspark/shell.py
@@ -35,12 +35,11 @@ from pyspark.storagelevel import StorageLevel
if os.environ.get("SPARK_EXECUTOR_URI"):
SparkContext.setSystemProperty("spark.executor.uri", os.environ["SPARK_EXECUTOR_URI"])
-sc = SparkContext()
-atexit.register(lambda: sc.stop())
+SparkContext._ensure_initialized()
try:
# Try to access HiveConf, it will raise exception if Hive is not added
- sc._jvm.org.apache.hadoop.hive.conf.HiveConf()
+ SparkContext._jvm.org.apache.hadoop.hive.conf.HiveConf()
spark = SparkSession.builder\
.enableHiveSupport()\
.getOrCreate()
@@ -49,6 +48,9 @@ except py4j.protocol.Py4JError:
except TypeError:
spark = SparkSession(sc)
+sc = spark.sparkContext
+atexit.register(lambda: sc.stop())
+
# for compatibility
sqlContext = spark._wrapped
sqlCtx = sqlContext
http://git-wip-us.apache.org/repos/asf/spark/blob/c32b1b16/python/pyspark/sql/session.py
----------------------------------------------------------------------
diff --git a/python/pyspark/sql/session.py b/python/pyspark/sql/session.py
index 0e04b88..2419475 100644
--- a/python/pyspark/sql/session.py
+++ b/python/pyspark/sql/session.py
@@ -204,6 +204,12 @@ class SparkSession(object):
@property
@since(2.0)
+ def sparkContext(self):
+ """Returns the underlying :class:`SparkContext`."""
+ return self._sc
+
+ @property
+ @since(2.0)
def conf(self):
"""Runtime configuration interface for Spark.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org