You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/01/10 05:30:04 UTC
[1/3] git commit: Simplify and fix pyspark script.
Updated Branches:
refs/heads/master 4b074fac0 -> 300eaa994
Simplify and fix pyspark script.
This patch removes compatibility for IPython < 1.0 but fixes the launch
script and makes it much simpler.
I tested this using the three commands in the PySpark documentation page:
1. IPYTHON=1 ./pyspark
2. IPYTHON_OPTS="notebook" ./pyspark
3. IPYTHON_OPTS="notebook --pylab inline" ./pyspark
There are two changes:
- We rely on PYTHONSTARTUP env var to start PySpark
- Removed the quotes around $IPYTHON_OPTS... having quotes
gloms them together as a single argument passed to `exec` which
seemed to cause ipython to fail (it instead expects them as
multiple arguments).
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/82a1d38a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/82a1d38a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/82a1d38a
Branch: refs/heads/master
Commit: 82a1d38aea3b10930a2659b9c0e7ad2fb2c2ab4a
Parents: 6ccf8ce
Author: Patrick Wendell <pw...@gmail.com>
Authored: Tue Jan 7 17:46:02 2014 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Tue Jan 7 17:55:25 2014 -0800
----------------------------------------------------------------------
bin/pyspark | 8 +-------
docs/python-programming-guide.md | 5 +++--
2 files changed, 4 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/82a1d38a/bin/pyspark
----------------------------------------------------------------------
diff --git a/bin/pyspark b/bin/pyspark
index d6810f4..a70da56 100755
--- a/bin/pyspark
+++ b/bin/pyspark
@@ -51,7 +51,6 @@ export PYSPARK_PYTHON
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
# Load the PySpark shell.py script when ./pyspark is used interactively:
-export OLD_PYTHONSTARTUP=$PYTHONSTARTUP
export PYTHONSTARTUP=$FWDIR/python/pyspark/shell.py
if [ -n "$IPYTHON_OPTS" ]; then
@@ -59,12 +58,7 @@ if [ -n "$IPYTHON_OPTS" ]; then
fi
if [[ "$IPYTHON" = "1" ]] ; then
- # IPython <1.0.0 doesn't honor PYTHONSTARTUP, while 1.0.0+ does.
- # Hence we clear PYTHONSTARTUP and use the -c "%run $IPYTHONSTARTUP" command which works on all versions
- # We also force interactive mode with "-i"
- IPYTHONSTARTUP=$PYTHONSTARTUP
- PYTHONSTARTUP=
- exec ipython "$IPYTHON_OPTS" -i -c "%run $IPYTHONSTARTUP"
+ exec ipython $IPYTHON_OPTS
else
exec "$PYSPARK_PYTHON" "$@"
fi
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/82a1d38a/docs/python-programming-guide.md
----------------------------------------------------------------------
diff --git a/docs/python-programming-guide.md b/docs/python-programming-guide.md
index dc187b3..c4236f8 100644
--- a/docs/python-programming-guide.md
+++ b/docs/python-programming-guide.md
@@ -99,8 +99,9 @@ $ MASTER=local[4] ./bin/pyspark
## IPython
-It is also possible to launch PySpark in [IPython](http://ipython.org), the enhanced Python interpreter.
-To do this, set the `IPYTHON` variable to `1` when running `bin/pyspark`:
+It is also possible to launch PySpark in [IPython](http://ipython.org), the
+enhanced Python interpreter. PySpark works with IPython 1.0.0 and later. To
+use IPython, set the `IPYTHON` variable to `1` when running `bin/pyspark`:
{% highlight bash %}
$ IPYTHON=1 ./bin/pyspark
[2/3] git commit: Small fix suggested by josh
Posted by pw...@apache.org.
Small fix suggested by josh
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/77ca9e1b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/77ca9e1b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/77ca9e1b
Branch: refs/heads/master
Commit: 77ca9e1ba845c8cbb1566f803b591f6a826b0f1d
Parents: 82a1d38
Author: Patrick Wendell <pw...@gmail.com>
Authored: Thu Jan 9 18:41:00 2014 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Thu Jan 9 18:41:00 2014 -0800
----------------------------------------------------------------------
bin/pyspark | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/77ca9e1b/bin/pyspark
----------------------------------------------------------------------
diff --git a/bin/pyspark b/bin/pyspark
index a70da56..ed6f8da 100755
--- a/bin/pyspark
+++ b/bin/pyspark
@@ -51,6 +51,7 @@ export PYSPARK_PYTHON
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
# Load the PySpark shell.py script when ./pyspark is used interactively:
+export OLD_PYTHONSTARTUP=$PYTHONSTARTUP
export PYTHONSTARTUP=$FWDIR/python/pyspark/shell.py
if [ -n "$IPYTHON_OPTS" ]; then
[3/3] git commit: Merge pull request #353 from
pwendell/ipython-simplify
Posted by pw...@apache.org.
Merge pull request #353 from pwendell/ipython-simplify
Simplify and fix pyspark script.
This patch removes compatibility for IPython < 1.0 but fixes the launch
script and makes it much simpler.
I tested this using the three commands in the PySpark documentation page:
1. IPYTHON=1 ./pyspark
2. IPYTHON_OPTS="notebook" ./pyspark
3. IPYTHON_OPTS="notebook --pylab inline" ./pyspark
There are two changes:
- We rely on PYTHONSTARTUP env var to start PySpark
- Removed the quotes around $IPYTHON_OPTS... having quotes
gloms them together as a single argument passed to `exec` which
seemed to cause ipython to fail (it instead expects them as
multiple arguments).
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/300eaa99
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/300eaa99
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/300eaa99
Branch: refs/heads/master
Commit: 300eaa994c399a0c991c1e39b4dd864a7aa4bdc6
Parents: 4b074fa 77ca9e1
Author: Patrick Wendell <pw...@gmail.com>
Authored: Thu Jan 9 20:29:51 2014 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Thu Jan 9 20:29:51 2014 -0800
----------------------------------------------------------------------
bin/pyspark | 7 +------
docs/python-programming-guide.md | 5 +++--
2 files changed, 4 insertions(+), 8 deletions(-)
----------------------------------------------------------------------