You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@toree.apache.org by "Eric Chang (JIRA)" <ji...@apache.org> on 2016/04/28 02:12:13 UTC
[jira] [Updated] (TOREE-310) Allow override of python executable
used for pyspark
[ https://issues.apache.org/jira/browse/TOREE-310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Chang updated TOREE-310:
-----------------------------
Description:
We're using virtualenvs when running pyspark, and it would be great to be able to use a virtualenv as the python executable used by the Spark Driver (i.e. {{--master yarn-client}}). This value is currently hard-coded to {{python}} in org/apache/toree/kernel/interpreter/pyspark/PySparkProcess.scala.
I have a branch on my repo which adds an optional kernel parameter PYTHON_EXEC:
{code}
...
"SPARK_HOME": "/usr/lib/spark",
"PYTHON_EXEC" : "/usr/local/python/virtualenvs/myvenv/bin/python",
...
{code}
If {{PYTHON_EXEC}} is unspecified, the default of {{python}} is used.
Here's the diff of the branch, please let me know if it's ok for me to issue a PR against the main repo: https://github.com/ericchang/incubator-toree/compare/ericchang:master...custom-python-exec
was:
We're using virtualenvs when running pyspark, and it would be great to be able to use a virtualenv as the python executable used by the Spark Driver (i.e. {{--master yarn-client}}). This value is currently hard-coded to {{python}} in org/apache/toree/kernel/interpreter/pyspark/PySparkProcess.scala.
I have a branch on my repo which adds an optional kernel parameter PYTHON_EXEC:
{code}
...
"SPARK_HOME": "/usr/lib/spark",
"PYTHON_EXEC" : "/usr/local/python/virtualenvs/myvenv/bin/python",
...
{code}
Here's the diff of the branch, please let me know if it's ok for me to issue a PR against the main repo: https://github.com/ericchang/incubator-toree/compare/ericchang:master...custom-python-exec
> Allow override of python executable used for pyspark
> ----------------------------------------------------
>
> Key: TOREE-310
> URL: https://issues.apache.org/jira/browse/TOREE-310
> Project: TOREE
> Issue Type: Improvement
> Reporter: Eric Chang
>
> We're using virtualenvs when running pyspark, and it would be great to be able to use a virtualenv as the python executable used by the Spark Driver (i.e. {{--master yarn-client}}). This value is currently hard-coded to {{python}} in org/apache/toree/kernel/interpreter/pyspark/PySparkProcess.scala.
> I have a branch on my repo which adds an optional kernel parameter PYTHON_EXEC:
> {code}
> ...
> "SPARK_HOME": "/usr/lib/spark",
> "PYTHON_EXEC" : "/usr/local/python/virtualenvs/myvenv/bin/python",
> ...
> {code}
> If {{PYTHON_EXEC}} is unspecified, the default of {{python}} is used.
> Here's the diff of the branch, please let me know if it's ok for me to issue a PR against the main repo: https://github.com/ericchang/incubator-toree/compare/ericchang:master...custom-python-exec
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)