You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ue...@apache.org on 2020/09/16 17:20:51 UTC
[spark] branch branch-3.0 updated: [SPARK-32897][PYTHON] Don't show
a deprecation warning at SparkSession.builder.getOrCreate
This is an automated email from the ASF dual-hosted git repository.
ueshin pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new aa9563e [SPARK-32897][PYTHON] Don't show a deprecation warning at SparkSession.builder.getOrCreate
aa9563e is described below
commit aa9563eb6afa97ef5cf35c947218cacbc31d69ff
Author: HyukjinKwon <gu...@apache.org>
AuthorDate: Wed Sep 16 10:13:47 2020 -0700
[SPARK-32897][PYTHON] Don't show a deprecation warning at SparkSession.builder.getOrCreate
### What changes were proposed in this pull request?
In PySpark shell, if you call `SparkSession.builder.getOrCreate` as below:
```python
import warnings
from pyspark.sql import SparkSession, SQLContext
warnings.simplefilter('always', DeprecationWarning)
spark.stop()
SparkSession.builder.getOrCreate()
```
it shows the deprecation warning as below:
```
/.../spark/python/pyspark/sql/context.py:72: DeprecationWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
DeprecationWarning)
```
via https://github.com/apache/spark/blob/d3304268d3046116d39ec3d54a8e319dce188f36/python/pyspark/sql/session.py#L222
We shouldn't print the deprecation warning from it. This is the only place ^.
### Why are the changes needed?
To prevent to inform users that `SparkSession.builder.getOrCreate` is deprecated mistakenly.
### Does this PR introduce _any_ user-facing change?
Yes, it won't show a deprecation warning to end users for calling `SparkSession.builder.getOrCreate`.
### How was this patch tested?
Manually tested as above.
Closes #29768 from HyukjinKwon/SPARK-32897.
Authored-by: HyukjinKwon <gu...@apache.org>
Signed-off-by: Takuya UESHIN <ue...@databricks.com>
(cherry picked from commit 657e39a3346daf0c67cff3cf90fe68176c479747)
Signed-off-by: Takuya UESHIN <ue...@databricks.com>
---
python/pyspark/sql/context.py | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/sql/context.py b/python/pyspark/sql/context.py
index 0b7a7da..d9e5080 100644
--- a/python/pyspark/sql/context.py
+++ b/python/pyspark/sql/context.py
@@ -72,9 +72,10 @@ class SQLContext(object):
>>> df.rdd.map(lambda x: (x.i, x.s, x.d, x.l, x.b, x.time, x.row.a, x.list)).collect()
[(1, u'string', 1.0, 1, True, datetime.datetime(2014, 8, 1, 14, 1, 5), 1, [1, 2, 3])]
"""
- warnings.warn(
- "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
- DeprecationWarning)
+ if sparkSession is None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ DeprecationWarning)
self._sc = sparkContext
self._jsc = self._sc._jsc
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org