[jira] [Created] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

["Thomas Graves (JIRA)" <ji...@apache.org>]

Thomas Graves created SPARK-22151:
-------------------------------------

             Summary: PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly
                 Key: SPARK-22151
                 URL: https://issues.apache.org/jira/browse/SPARK-22151
             Project: Spark
          Issue Type: Bug
          Components: YARN
    Affects Versions: 2.1.1
            Reporter: Thomas Graves


the code looks at the env variables:
val pythonPathStr = (sys.env.get("PYTHONPATH") ++ pythonPath)
But when you set spark.yarn.appMasterEnv it puts it into the local env. 

So the python path set in spark.yarn.appMasterEnv isn't properly set.

You can work around if you are running in cluster mode by setting it on the client like:

PYTHONPATH=./addon/python/ spark-submit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org