You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2017/01/13 10:36:07 UTC
spark git commit: [SPARK-18687][PYSPARK][SQL] Backward compatibility
- creating a Dataframe on a new SQLContext object fails with a Derby error
Repository: spark
Updated Branches:
refs/heads/master b040cef2e -> 285a7798e
[SPARK-18687][PYSPARK][SQL] Backward compatibility - creating a Dataframe on a new SQLContext object fails with a Derby error
Change is for SQLContext to reuse the active SparkSession during construction if the sparkContext supplied is the same as the currently active SparkContext. Without this change, a new SparkSession is instantiated that results in a Derby error when attempting to create a dataframe using a new SQLContext object even though the SparkContext supplied to the new SQLContext is same as the currently active one. Refer https://issues.apache.org/jira/browse/SPARK-18687 for details on the error and a repro.
Existing unit tests and a new unit test added to pyspark-sql:
/python/run-tests --python-executables=python --modules=pyspark-sql
Please review http://spark.apache.org/contributing.html before opening a pull request.
Author: Vinayak <vi...@in.ibm.com>
Author: Vinayak Joshi <vi...@users.noreply.github.com>
Closes #16119 from vijoshi/SPARK-18687_master.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/285a7798
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/285a7798
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/285a7798
Branch: refs/heads/master
Commit: 285a7798e267311730b0163d37d726a81465468a
Parents: b040cef
Author: Vinayak <vi...@in.ibm.com>
Authored: Fri Jan 13 18:35:12 2017 +0800
Committer: Wenchen Fan <we...@databricks.com>
Committed: Fri Jan 13 18:35:51 2017 +0800
----------------------------------------------------------------------
python/pyspark/sql/context.py | 2 +-
python/pyspark/sql/tests.py | 7 ++++++-
2 files changed, 7 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/285a7798/python/pyspark/sql/context.py
----------------------------------------------------------------------
diff --git a/python/pyspark/sql/context.py b/python/pyspark/sql/context.py
index de4c335..c22f4b8 100644
--- a/python/pyspark/sql/context.py
+++ b/python/pyspark/sql/context.py
@@ -73,7 +73,7 @@ class SQLContext(object):
self._jsc = self._sc._jsc
self._jvm = self._sc._jvm
if sparkSession is None:
- sparkSession = SparkSession(sparkContext)
+ sparkSession = SparkSession.builder.getOrCreate()
if jsqlContext is None:
jsqlContext = sparkSession._jwrapped
self.sparkSession = sparkSession
http://git-wip-us.apache.org/repos/asf/spark/blob/285a7798/python/pyspark/sql/tests.py
----------------------------------------------------------------------
diff --git a/python/pyspark/sql/tests.py b/python/pyspark/sql/tests.py
index d178285..a825028 100644
--- a/python/pyspark/sql/tests.py
+++ b/python/pyspark/sql/tests.py
@@ -47,7 +47,7 @@ else:
import unittest
from pyspark import SparkContext
-from pyspark.sql import SparkSession, HiveContext, Column, Row
+from pyspark.sql import SparkSession, SQLContext, HiveContext, Column, Row
from pyspark.sql.types import *
from pyspark.sql.types import UserDefinedType, _infer_type
from pyspark.tests import ReusedPySparkTestCase, SparkSubmitTests
@@ -206,6 +206,11 @@ class SQLTests(ReusedPySparkTestCase):
cls.spark.stop()
shutil.rmtree(cls.tempdir.name, ignore_errors=True)
+ def test_sqlcontext_reuses_sparksession(self):
+ sqlContext1 = SQLContext(self.sc)
+ sqlContext2 = SQLContext(self.sc)
+ self.assertTrue(sqlContext1.sparkSession is sqlContext2.sparkSession)
+
def test_row_should_be_read_only(self):
row = Row(a=1, b=2)
self.assertEqual(1, row.a)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org