You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2017/10/19 09:06:25 UTC
[GitHub] spark pull request #19535: [WIP][PYTHON] Mark/print deprecation warnings as ...
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/19535
[WIP][PYTHON] Mark/print deprecation warnings as DeprecationWarning for deprecated APIs
## What changes were proposed in this pull request?
This PR proposes to mark the existing warnings as `DeprecationWarning` and print out warnings for deprecated functions.
This could be actually useful for Spark app developers. I use PyCharm and this IDE can detect this specific `DeprecationWarning`:
**Before**
<img src="https://user-images.githubusercontent.com/6477701/31762664-df68d9f8-b4f6-11e7-8773-f0468f70a2cc.png" height="45" />
**After**
<img src="https://user-images.githubusercontent.com/6477701/31762662-de4d6868-b4f6-11e7-98dc-3c8446a0c28a.png" height="70" />
For console usage, `DeprecationWarning` is usually disabled (see https://docs.python.org/2/library/warnings.html#warning-categories and https://docs.python.org/3/library/warnings.html#warning-categories):
```
>>> import warnings
>>> filter(lambda f: f[2] == DeprecationWarning, warnings.filters)
[('ignore', <_sre.SRE_Pattern object at 0x10ba58c00>, <type 'exceptions.DeprecationWarning'>, <_sre.SRE_Pattern object at 0x10bb04138>, 0), ('ignore', None, <type 'exceptions.DeprecationWarning'>, None, 0)]
```
so, it won't actually mess up the terminal much unless it is intended.
If this is intendedly enabled, it'd should as below:
```
>>> import warnings
>>> warnings.simplefilter('always', DeprecationWarning)
>>>
>>> from pyspark.sql import functions
>>> functions.approxCountDistinct("a")
.../spark/python/pyspark/sql/functions.py:232: DeprecationWarning: Deprecated in 2.1, use approx_count_distinct instead.
"Deprecated in 2.1, use approx_count_distinct instead.", DeprecationWarning)
...
```
These instances were found by:
```
cd python/pyspark
grep -r "Deprecated" .
grep -r "deprecated" .
grep -r "deprecate" .
```
## How was this patch tested?
Manually tested.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark deprecated-warning
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19535.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19535
----
commit 376162a5976877faf87446ac9a2774d01b44e6e3
Author: hyukjinkwon <gu...@gmail.com>
Date: 2017-10-19T08:10:03Z
Mark/print deprecation warnings as DeprecationWarningfor deprecated APIs
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19535
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82956/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19535
**[Test build #82906 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82906/testReport)** for PR 19535 at commit [`376162a`](https://github.com/apache/spark/commit/376162a5976877faf87446ac9a2774d01b44e6e3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/19535
Looks good at high level.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19535
**[Test build #82956 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82956/testReport)** for PR 19535 at commit [`f8a7d2b`](https://github.com/apache/spark/commit/f8a7d2b4f20838915b50f9e7233df42e77f7dc84).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19535
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82906/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19535
Merged to master
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19535
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [WIP][PYTHON] Mark/print deprecation warnings as Depreca...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19535
@holdenk and @rxin, I didn't open a JIRA yet and I want to have your opinions first. Do you guys like this idea?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [WIP][PYTHON] Mark/print deprecation warnings as Depreca...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19535
Let me probably open up a JIRA and link it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19535
Thanks @felixcheung. Will update soon.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19535: [SPARK-22313][PYTHON] Mark/print deprecation warn...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19535#discussion_r159020029
--- Diff: python/pyspark/streaming/flume.py ---
@@ -54,8 +54,13 @@ def createStream(ssc, hostname, port,
:param bodyDecoder: A function used to decode body (default is utf8_decoder)
:return: A DStream object
- .. note:: Deprecated in 2.3.0
+ .. note:: Deprecated in 2.3.0. Flume support is deprecated as of Spark 2.3.0.
+ See SPARK-22142.
"""
+ warnings.warn(
--- End diff --
Sure, I took a quick look and I think this one is actually not being tested and seems that's why .. will double check and take a closer look tonight (KST).
I have seen few mistakes about this so far and .. I am working on Python coverage BTW - https://issues.apache.org/jira/browse/SPARK-7721
Anyway, it was my stupid mistake. Thanks ..
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19535: [SPARK-22313][PYTHON] Mark/print deprecation warn...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19535#discussion_r159019418
--- Diff: python/pyspark/streaming/flume.py ---
@@ -54,8 +54,13 @@ def createStream(ssc, hostname, port,
:param bodyDecoder: A function used to decode body (default is utf8_decoder)
:return: A DStream object
- .. note:: Deprecated in 2.3.0
+ .. note:: Deprecated in 2.3.0. Flume support is deprecated as of Spark 2.3.0.
+ See SPARK-22142.
"""
+ warnings.warn(
--- End diff --
Yes, it is not. Will make a followup after double checking other files too. Thank you.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19535
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19535: [SPARK-22313][PYTHON] Mark/print deprecation warn...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19535#discussion_r145655586
--- Diff: python/pyspark/sql/functions.py ---
@@ -44,6 +45,14 @@ def _(col):
return _
+def _wrap_deprecated_function(func, message):
+ """ Wrap the deprecated function to print out deprecation warnings"""
+ def _(col):
+ warnings.warn(message, DeprecationWarning)
+ return func(col)
--- End diff --
Here, I intendedly avoided `*args` and `**kwargs` to keep the argument signature printed in pydoc, `help(...)`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19535: [SPARK-22313][PYTHON] Mark/print deprecation warn...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/19535#discussion_r159013024
--- Diff: python/pyspark/streaming/flume.py ---
@@ -54,8 +54,13 @@ def createStream(ssc, hostname, port,
:param bodyDecoder: A function used to decode body (default is utf8_decoder)
:return: A DStream object
- .. note:: Deprecated in 2.3.0
+ .. note:: Deprecated in 2.3.0. Flume support is deprecated as of Spark 2.3.0.
+ See SPARK-22142.
"""
+ warnings.warn(
--- End diff --
Seems `warnings` is not imported in this file?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19535
**[Test build #82956 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82956/testReport)** for PR 19535 at commit [`f8a7d2b`](https://github.com/apache/spark/commit/f8a7d2b4f20838915b50f9e7233df42e77f7dc84).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19535: [SPARK-22313][PYTHON] Mark/print deprecation warn...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/19535#discussion_r146024844
--- Diff: python/pyspark/streaming/flume.py ---
@@ -56,6 +56,7 @@ def createStream(ssc, hostname, port,
.. note:: Deprecated in 2.3.0
"""
+ warnings.warn("Deprecated in 2.3.0.", DeprecationWarning)
--- End diff --
for these, could you provide more information? link to the doc on deprecating DStream in python?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19535: [SPARK-22313][PYTHON] Mark/print deprecation warn...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/19535#discussion_r146024871
--- Diff: python/pyspark/streaming/kafka.py ---
@@ -58,6 +60,7 @@ def createStream(ssc, zkQuorum, groupId, topics, kafkaParams=None,
.. note:: Deprecated in 2.3.0
"""
+ warnings.warn("Deprecated in 2.3.0.", DeprecationWarning)
--- End diff --
ditto here
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19535
**[Test build #82906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82906/testReport)** for PR 19535 at commit [`376162a`](https://github.com/apache/spark/commit/376162a5976877faf87446ac9a2774d01b44e6e3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19535: [SPARK-22313][PYTHON] Mark/print deprecation warn...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19535
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19535
**[Test build #82957 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82957/testReport)** for PR 19535 at commit [`5ac0697`](https://github.com/apache/spark/commit/5ac0697bf17be37de7ec3a7c5268c2b208ac49ee).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19535
**[Test build #82957 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82957/testReport)** for PR 19535 at commit [`5ac0697`](https://github.com/apache/spark/commit/5ac0697bf17be37de7ec3a7c5268c2b208ac49ee).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19535: [SPARK-22313][PYTHON] Mark/print deprecation warn...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19535#discussion_r145648171
--- Diff: python/pyspark/mllib/classification.py ---
@@ -311,7 +311,7 @@ def train(cls, data, iterations=100, step=1.0, miniBatchFraction=1.0,
"""
warnings.warn(
"Deprecated in 2.0.0. Use ml.classification.LogisticRegression or "
- "LogisticRegressionWithLBFGS.")
+ "LogisticRegressionWithLBFGS.", DeprecationWarning)
--- End diff --
Another example:
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19535: [SPARK-22313][PYTHON] Mark/print deprecation warn...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/19535#discussion_r159019845
--- Diff: python/pyspark/streaming/flume.py ---
@@ -54,8 +54,13 @@ def createStream(ssc, hostname, port,
:param bodyDecoder: A function used to decode body (default is utf8_decoder)
:return: A DStream object
- .. note:: Deprecated in 2.3.0
+ .. note:: Deprecated in 2.3.0. Flume support is deprecated as of Spark 2.3.0.
+ See SPARK-22142.
"""
+ warnings.warn(
--- End diff --
thank you :) It will be good to also check why master build does not fail since python should complain about it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19535
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19535
Thanks @srowen, @rxin and @felixcheung.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19535: [SPARK-22313][PYTHON] Mark/print deprecation warnings as...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19535
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82957/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org