You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@zeppelin.apache.org by "Mathew (JIRA)" <ji...@apache.org> on 2018/08/16 04:22:00 UTC

[jira] [Created] (ZEPPELIN-3720) PYSPARK_PYTHON / PYSPARK_DRIVER_PYTHON ignored.

Mathew created ZEPPELIN-3720:
--------------------------------

             Summary: PYSPARK_PYTHON / PYSPARK_DRIVER_PYTHON ignored.
                 Key: ZEPPELIN-3720
                 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3720
             Project: Zeppelin
          Issue Type: Bug
    Affects Versions: 0.8.0
         Environment: Spark 2.2.2 (Cloudera)

Zeppelin 0.8.0

RHEL 7.4
            Reporter: Mathew


I am not 100% sure if the issue is on my end, but no matter what combination of the following spark interpreter configs I use, the PYSPARK_PYTHON present in my spark-env.sh takes precedence.
 * PYSPARK_PYTHON
 * PYSPARK_DRIVER_PYTHON
 * spark.pyspark.python
 * spark.pyspark.driver.python
 * spark.yarn.appMasterEnv.PYSPARK_PYTHON
 * spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON

 

Note that this is only is yarn client mode, as in yarn cluster mode, it ignores whats in spark-env.sh.

When I use spark-submit it works fine, and picks up my custom python executable path.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)