You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by da...@apache.org on 2016/05/11 21:15:29 UTC
spark git commit: [SPARK-15270] [SQL] Use SparkSession Builder to
build a session with HiveSupport
Repository: spark
Updated Branches:
refs/heads/master 40a949aae -> de9c85cca
[SPARK-15270] [SQL] Use SparkSession Builder to build a session with HiveSupport
## What changes were proposed in this pull request?
Before:
Creating a hiveContext was failing
```python
from pyspark.sql import HiveContext
hc = HiveContext(sc)
```
with
```
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "spark-2.0/python/pyspark/sql/context.py", line 458, in __init__
sparkSession = SparkSession.withHiveSupport(sparkContext)
File "spark-2.0/python/pyspark/sql/session.py", line 192, in withHiveSupport
jsparkSession = sparkContext._jvm.SparkSession.withHiveSupport(sparkContext._jsc.sc())
File "spark-2.0/python/lib/py4j-0.9.2-src.zip/py4j/java_gateway.py", line 1048, in __getattr__
py4j.protocol.Py4JError: org.apache.spark.sql.SparkSession.withHiveSupport does not exist in the JVM
```
Now:
```python
>>> from pyspark.sql import HiveContext
>>> hc = HiveContext(sc)
>>> hc.range(0, 100)
DataFrame[id: bigint]
>>> hc.range(0, 100).count()
100
```
## How was this patch tested?
Existing Tests, tested manually in python shell
Author: Sandeep Singh <sa...@techaddict.me>
Closes #13056 from techaddict/SPARK-15270.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/de9c85cc
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/de9c85cc
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/de9c85cc
Branch: refs/heads/master
Commit: de9c85ccaacd12de9837eb88eae0a7e7ededd679
Parents: 40a949a
Author: Sandeep Singh <sa...@techaddict.me>
Authored: Wed May 11 14:15:18 2016 -0700
Committer: Davies Liu <da...@gmail.com>
Committed: Wed May 11 14:15:18 2016 -0700
----------------------------------------------------------------------
python/pyspark/sql/context.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/de9c85cc/python/pyspark/sql/context.py
----------------------------------------------------------------------
diff --git a/python/pyspark/sql/context.py b/python/pyspark/sql/context.py
index 78ab2e8..02e742c 100644
--- a/python/pyspark/sql/context.py
+++ b/python/pyspark/sql/context.py
@@ -455,7 +455,7 @@ class HiveContext(SQLContext):
def __init__(self, sparkContext, jhiveContext=None):
if jhiveContext is None:
- sparkSession = SparkSession.withHiveSupport(sparkContext)
+ sparkSession = SparkSession.builder.enableHiveSupport().getOrCreate()
else:
sparkSession = SparkSession(sparkContext, jhiveContext.sparkSession())
SQLContext.__init__(self, sparkContext, sparkSession, jhiveContext)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org