You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Paul Shearer (JIRA)" <ji...@apache.org> on 2016/04/20 12:49:25 UTC
[jira] [Comment Edited] (SPARK-13973) `ipython notebook` is going
away...
[ https://issues.apache.org/jira/browse/SPARK-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249628#comment-15249628 ]
Paul Shearer edited comment on SPARK-13973 at 4/20/16 10:48 AM:
----------------------------------------------------------------
The problem with this change is that it creates a bug for users who simply want the IPython interactive shell, as opposed to the notebook. `ipython` with no arguments starts the IPython shell, but `jupyter` with no arguments results in the following error:
{noformat}
usage: jupyter [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir]
[--paths] [--json]
[subcommand]
jupyter: error: one of the arguments --version subcommand --config-dir --data-dir --runtime-dir --paths is required
{noformat}
I can't speak for the general Python community but as a data scientist, personally I find the IPython notebook only suitable for very basic exploratory analysis - any sort of application development is much better served by the IPython shell, so I'm always using the shell and rarely the notebook. So I prefer the old script.
Perhaps the best answer is to stop maintaining an unsustainable backwards compatibility. The committed change broke the pyspark startup script in my case, and the old startup script will eventually be broken when `ipython notebook` is deprecated. So perhaps `IPYTHON=1` should just result in some kind of error message prompting the user to switch to the new PYSPARK_DRIVER_PYTHON config style. Most Spark users knows the installation process is not seamless and requires mucking about with environment variables - they might as well be told to do it in a way that's convenient to the development team.
was (Author: pshearer):
The problem with this change is that it creates a bug for users who simply want the IPython interactive shell, as opposed to the notebook. `ipython` with no arguments starts the IPython shell, but `jupyter` with no arguments results in the following error:
{noformat}
usage: jupyter [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir]
[--paths] [--json]
[subcommand]
jupyter: error: one of the arguments --version subcommand --config-dir --data-dir --runtime-dir --paths is required
{noformat}
I can't speak for the general Python community but as a data scientist, personally I find the IPython notebook only suitable for very basic exploratory analysis - any sort of application development is much better served by the IPython shell, so I'm always using the shell and rarely the notebook.
It seems like maintaining this old configuration switch is no longer sustainable. The change breaks it for my case, and the old state will eventually be broken when `ipython notebook` is deprecated. So perhaps `IPYTHON=1` should just result in some kind of error message prompting the user to switch to the new PYSPARK_DRIVER_PYTHON config style.
> `ipython notebook` is going away...
> -----------------------------------
>
> Key: SPARK-13973
> URL: https://issues.apache.org/jira/browse/SPARK-13973
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Environment: spark-1.6.1-bin-hadoop2.6
> Anaconda2-2.5.0-Linux-x86_64
> Reporter: Bogdan Pirvu
> Assignee: Rekha Joshi
> Priority: Trivial
> Fix For: 2.0.0
>
>
> Starting {{pyspark}} with following environment variables:
> {code:none}
> export IPYTHON=1
> export IPYTHON_OPTS="notebook --no-browser"
> {code}
> yields this warning
> {code:none}
> [TerminalIPythonApp] WARNING | Subcommand `ipython notebook` is deprecated and will be removed in future versions.
> [TerminalIPythonApp] WARNING | You likely want to use `jupyter notebook`... continue in 5 sec. Press Ctrl-C to quit now.
> {code}
> Changing line 52 from
> {code:none}
> PYSPARK_DRIVER_PYTHON="ipython"
> {code}
> to
> {code:none}
> PYSPARK_DRIVER_PYTHON="jupyter"
> {code}
> in https://github.com/apache/spark/blob/master/bin/pyspark works for me to solve this issue, but I'm not sure if it's sustainable as I'm not familiar with the rest of the code...
> This is the relevant part of my Python environment:
> {code:none}
> ipython 4.1.2 py27_0
> ipython-genutils 0.1.0 <pip>
> ipython_genutils 0.1.0 py27_0
> ipywidgets 4.1.1 py27_0
> ...
> jupyter 1.0.0 py27_1
> jupyter-client 4.2.1 <pip>
> jupyter-console 4.1.1 <pip>
> jupyter-core 4.1.0 <pip>
> jupyter_client 4.2.1 py27_0
> jupyter_console 4.1.1 py27_0
> jupyter_core 4.1.0 py27_0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org