[GitHub] spark pull request #21468: [SPARK-22151] : PYTHONPATH not picked up from the...

[vanzin <gi...@git.apache.org>]

Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21468#discussion_r192857531
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -813,8 +813,14 @@ private[spark] class Client(
         if (pythonPath.nonEmpty) {
           val pythonPathStr = (sys.env.get("PYTHONPATH") ++ pythonPath)
             .mkString(ApplicationConstants.CLASS_PATH_SEPARATOR)
    -      env("PYTHONPATH") = pythonPathStr
    -      sparkConf.setExecutorEnv("PYTHONPATH", pythonPathStr)
    +      val newValue =
    --- End diff --
    
    You could just say `env.get("PYTHONPATH") ++=: pythonPath` before turning the list into a string.
    
    But there's also two extra questions here:
    - precedence; should the env come before or after the files added with `py-files`? I kinda think after makes more sense, since files are generally provided in the command line.
    - should `appMasterEnv` be reflected in executors? With your code it is. I'm not so sure it should.



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org