You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Peter Parente (JIRA)" <ji...@apache.org> on 2017/05/20 20:37:04 UTC
[jira] [Updated] (SPARK-20818) Tab-to-autocomplete in IPython +
Python3 results in job execution
[ https://issues.apache.org/jira/browse/SPARK-20818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Parente updated SPARK-20818:
----------------------------------
Description:
Using Spark in a Jupyter Notebook 5.0 with an IPython 6.0 kernel and Python 3, when I press Tab to autocomplete the function names on a DataFrame or RDD, Spark executes the job graph constructed thus far. This only appears to happen for certain autocompletions, namely completions that resolve to functions having the @ignore_unicode_prefix decorator applied to them, and only in Python 3 (never in Python 2).
https://github.com/apache/spark/blob/master/python/pyspark/rdd.py#L146
Indeed, this function does have a special case for Python 3 in which it rewrites function docstrings. Why IPython autocompletion has a bad interaction with this logic is unknown (to me at least) at the moment.
I reproduced this bug on Spark 2.0.2. The code in the decorator hasn't changed in 2.1.x, so the bug likely impacts that version as well.
was:
Using Spark in a Jupyter Notebook 5.0 with an IPython 6.0 kernel and Python 3, when I press Tab to autocomplete the function names on an RDD, Spark executes the job graph constructed thus far. This only appears to happen for certain autocompletions, namely completions that resolve to functions having the @ignore_unicode_prefix decorator applied to them, and only in Python 3 (never in Python 2).
https://github.com/apache/spark/blob/master/python/pyspark/rdd.py#L146
Indeed, this function does have a special case for Python 3 in which it rewrites function docstrings. Why IPython autocompletion has a bad interaction with this logic is unknown (to me at least) at the moment.
I reproduced this bug on Spark 2.0.2. The code in the decorator hasn't changed in 2.1.x, so the bug likely impacts that version as well.
> Tab-to-autocomplete in IPython + Python3 results in job execution
> -----------------------------------------------------------------
>
> Key: SPARK-20818
> URL: https://issues.apache.org/jira/browse/SPARK-20818
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.0.2
> Reporter: Peter Parente
> Labels: ipython, jupyter
>
> Using Spark in a Jupyter Notebook 5.0 with an IPython 6.0 kernel and Python 3, when I press Tab to autocomplete the function names on a DataFrame or RDD, Spark executes the job graph constructed thus far. This only appears to happen for certain autocompletions, namely completions that resolve to functions having the @ignore_unicode_prefix decorator applied to them, and only in Python 3 (never in Python 2).
> https://github.com/apache/spark/blob/master/python/pyspark/rdd.py#L146
> Indeed, this function does have a special case for Python 3 in which it rewrites function docstrings. Why IPython autocompletion has a bad interaction with this logic is unknown (to me at least) at the moment.
> I reproduced this bug on Spark 2.0.2. The code in the decorator hasn't changed in 2.1.x, so the bug likely impacts that version as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org