[GitHub] spark pull request #19840: [SPARK-22640][PYSPARK][YARN]switch python exec on...

[jiangxb1987 <gi...@git.apache.org>]

Github user jiangxb1987 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19840#discussion_r154028953
  
    --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
    @@ -57,9 +57,14 @@ private[spark] abstract class BasePythonRunner[IN, OUT](
     
       require(funcs.length == argOffsets.length, "argOffsets should have the same length as funcs")
     
    +  private val env = SparkEnv.get
    +  private val conf = env.conf
    +
       // All the Python functions should have the same exec, version and envvars.
       protected val envVars = funcs.head.funcs.head.envVars
    -  protected val pythonExec = funcs.head.funcs.head.pythonExec
    +  protected val pythonExec = conf.getOption("spark.executorEnv.PYSPARK_DRIVER_PYTHON")
    --- End diff --
    
    Shouldn't this be handled in `deploy/PythonRunner`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org