You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by fe...@apache.org on 2017/03/03 06:53:10 UTC
zeppelin git commit: ZEPPELIN-2195. Use PYSPARK_PYTHON and
PYSPARK_DRIVER_PYTHON over zeppelin.pyspark.python
Repository: zeppelin
Updated Branches:
refs/heads/master df320481b -> bab985461
ZEPPELIN-2195. Use PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON over zeppelin.pyspark.python
### What is this PR for?
`zeppelin.pyspark.python` is zeppelin configuration for the python exec on driver side, it won't affect executor side. It would be better to use `PYSPARK_PYTHON` and `PYSPARK_DRIVER_PYTHON` which is what spark use officially. So that user can define their own python exec in interpreter setting for different version of python rather than defining them `zeppelin-env.sh` which is shared globally.
### What type of PR is it?
[ Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2195
### How should this be tested?
Tested it manually.
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zj...@apache.org>
Closes #2079 from zjffdu/ZEPPELIN-2195 and squashes the following commits:
fa71cb2 [Jeff Zhang] address comments
fd89a1e [Jeff Zhang] ZEPPELIN-2195. Use PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON over zeppelin.pyspark.python
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/bab98546
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/bab98546
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/bab98546
Branch: refs/heads/master
Commit: bab985461c507f276e648f335997acc340416f56
Parents: df32048
Author: Jeff Zhang <zj...@apache.org>
Authored: Tue Feb 28 17:42:51 2017 +0800
Committer: Felix Cheung <fe...@apache.org>
Committed: Thu Mar 2 22:51:06 2017 -0800
----------------------------------------------------------------------
docs/install/upgrade.md | 5 +++++
docs/interpreter/spark.md | 11 +++++++++--
.../org/apache/zeppelin/spark/PySparkInterpreter.java | 11 ++++++++++-
3 files changed, 24 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/bab98546/docs/install/upgrade.md
----------------------------------------------------------------------
diff --git a/docs/install/upgrade.md b/docs/install/upgrade.md
index f310196..44e38a3 100644
--- a/docs/install/upgrade.md
+++ b/docs/install/upgrade.md
@@ -56,3 +56,8 @@ So, copying `notebook` and `conf` directory should be enough.
- From 0.7, we uses `pegdown` as the `markdown.parser.type` option for the `%md` interpreter. Rendered markdown might be different from what you expected
- From 0.7 note.json format has been changed to support multiple outputs in a paragraph. Zeppelin will automatically convert old format to new format. 0.6 or lower version can read new note.json format but output will not be displayed. For the detail, see [ZEPPELIN-212](http://issues.apache.org/jira/browse/ZEPPELIN-212) and [pull request](https://github.com/apache/zeppelin/pull/1658).
- From 0.7 note storage layer will utilize `GitNotebookRepo` by default instead of `VFSNotebookRepo` storage layer, which is an extension of latter one with versioning capabilities on top of it.
+
+### Upgrading from Zeppelin 0.7 to 0.8
+
+ - From 0.8, we recommend to use `PYSPARK_PYTHON` and `PYSPARK_DRIVER_PYTHON` instead of `zeppelin.pyspark.python` as `zeppelin.pyspark.python` only effects driver. You can use `PYSPARK_PYTHON` and `PYSPARK_DRIVER_PYTHON` as using them in spark.
+
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/bab98546/docs/interpreter/spark.md
----------------------------------------------------------------------
diff --git a/docs/interpreter/spark.md b/docs/interpreter/spark.md
index 6d92561..a19eda2 100644
--- a/docs/interpreter/spark.md
+++ b/docs/interpreter/spark.md
@@ -104,9 +104,16 @@ You can also set other Spark properties which are not listed in the table. For a
<td>Local repository for dependency loader</td>
</tr>
<tr>
- <td>zeppelin.pyspark.python</td>
+ <td>PYSPARK_PYTHON</td>
<td>python</td>
- <td>Python command to run pyspark with</td>
+ <td>Python binary executable to use for PySpark in both driver and workers (default is <code>python</code>).
+ Property <code>spark.pyspark.python</code> take precedence if it is set</td>
+ </tr>
+ <tr>
+ <td>PYSPARK_DRIVER_PYTHON</td>
+ <td>python</td>
+ <td>Python binary executable to use for PySpark in driver only (default is <code>PYSPARK_PYTHON</code>).
+ Property <code>spark.pyspark.driver.python</code> take precedence if it is set</td>
</tr>
<tr>
<td>zeppelin.spark.concurrentSQL</td>
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/bab98546/spark/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
----------------------------------------------------------------------
diff --git a/spark/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java b/spark/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
index 0679fcc..b7dc67d 100644
--- a/spark/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
+++ b/spark/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
@@ -200,7 +200,16 @@ public class PySparkInterpreter extends Interpreter implements ExecuteResultHand
gatewayServer.start();
// Run python shell
- CommandLine cmd = CommandLine.parse(getProperty("zeppelin.pyspark.python"));
+ // Choose python in the order of
+ // PYSPARK_DRIVER_PYTHON > PYSPARK_PYTHON > zeppelin.pyspark.python
+ String pythonExec = getProperty("zeppelin.pyspark.python");
+ if (System.getenv("PYSPARK_PYTHON") != null) {
+ pythonExec = System.getenv("PYSPARK_PYTHON");
+ }
+ if (System.getenv("PYSPARK_DRIVER_PYTHON") != null) {
+ pythonExec = System.getenv("PYSPARK_DRIVER_PYTHON");
+ }
+ CommandLine cmd = CommandLine.parse(pythonExec);
cmd.addArgument(scriptPath, false);
cmd.addArgument(Integer.toString(port), false);
cmd.addArgument(Integer.toString(getSparkInterpreter().getSparkVersion().toNumber()), false);