[jira] [Closed] (TOREE-310) Allow override of python executable used for pyspark

["Gino Bustelo (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TOREE-310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gino Bustelo closed TOREE-310.
------------------------------

> Allow override of python executable used for pyspark
> ----------------------------------------------------
>
>                 Key: TOREE-310
>                 URL: https://issues.apache.org/jira/browse/TOREE-310
>             Project: TOREE
>          Issue Type: Improvement
>            Reporter: Eric Chang
>             Fix For: 0.1.0
>
>
> We're using virtualenvs when running pyspark, and it would be great to be able to use a virtualenv as the python executable used by the Spark Driver (i.e. {{--master yarn-client}}).  This value is currently hard-coded to {{python}} in org/apache/toree/kernel/interpreter/pyspark/PySparkProcess.scala.
> I have a branch on my repo which adds an optional kernel parameter PYTHON_EXEC:
> {code}
> ...
>     "SPARK_HOME": "/usr/lib/spark",
>     "PYTHON_EXEC" : "/usr/local/python/virtualenvs/myvenv/bin/python",
> ...
> {code}
> If {{PYTHON_EXEC}} is unspecified, the default of {{python}} is used.
> Here's the diff of the branch, please let me know if it's ok for me to issue a PR against the main repo: https://github.com/ericchang/incubator-toree/compare/ericchang:master...custom-python-exec



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)