You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2016/07/27 20:52:20 UTC
[jira] [Resolved] (SPARK-16110) Can't set Python via spark-submit
for YARN cluster mode when PYSPARK_PYTHON & PYSPARK_DRIVER_PYTHON are set
[ https://issues.apache.org/jira/browse/SPARK-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin resolved SPARK-16110.
------------------------------------
Resolution: Fixed
Assignee: Kevin Grealish
Fix Version/s: 2.1.0
> Can't set Python via spark-submit for YARN cluster mode when PYSPARK_PYTHON & PYSPARK_DRIVER_PYTHON are set
> -----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-16110
> URL: https://issues.apache.org/jira/browse/SPARK-16110
> Project: Spark
> Issue Type: Bug
> Components: PySpark, YARN
> Affects Versions: 1.6.1
> Environment: Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-38-generic x86_64), Spark 1.6.1, Azure HDInsight 3.4)
> Reporter: Kevin Grealish
> Assignee: Kevin Grealish
> Labels: easyfix
> Fix For: 2.1.0
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> When a cluster has PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment variables set (needed for using non-system Python e.g. /usr/bin/anaconda/bin/python), then you are unable to override this per submission in YARN cluster mode.
> When using spark-submit (in this case via LIVY) to submit with an override:
> spark-submit --master yarn --deploy-mode cluster --conf 'spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON=python3' --conf' 'spark.yarn.appMasterEnv.PYSPARK_PYTHON=python3' probe.py
> the environment variable values will override the conf settings. A workaround for some can be to unset the env vars but that is not always possible (e.g. submitting batch via LIVY where you can only pass through the parameters to spark-submit).
> Expectation is that the conf values above override the environment variables.
> Fix is to change the order of application of conf and env vars in the yarn client.
> Related discussion:https://issues.cloudera.org/browse/LIVY-159
> Backporting this to 1.6 would be great and unblocking for me.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org