[jira] [Created] (TOREE-310) Allow override of python executable used for pyspark

["Eric Chang (JIRA)" <ji...@apache.org>]

Eric Chang created TOREE-310:
--------------------------------

             Summary: Allow override of python executable used for pyspark
                 Key: TOREE-310
                 URL: https://issues.apache.org/jira/browse/TOREE-310
             Project: TOREE
          Issue Type: Improvement
            Reporter: Eric Chang


We're using virtualenvs when running pyspark, and it would be great to be able to use a virtualenv as the python executable used by the Spark Driver (i.e. {{--master yarn-client}}).  This value is currently hard-coded to {{python}} in org/apache/toree/kernel/interpreter/pyspark/PySparkProcess.scala.

I have a branch on my repo which adds an optional kernel parameter PYTHON_EXEC:
{code}
...
    "SPARK_HOME": "/usr/lib/spark",
    "PYTHON_EXEC" : "/usr/local/python/virtualenvs/myvenv/bin/python",
...
{code}

Here's the diff of the branch, please let me know if it's ok for me to issue a PR against the main repo: https://github.com/ericchang/incubator-toree/compare/ericchang:master...custom-python-exec



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)