You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jackey Lee (Jira)" <ji...@apache.org> on 2020/03/13 03:30:00 UTC

[jira] [Created] (SPARK-31142) Remove useless conf set in pyspark context

Jackey Lee created SPARK-31142:
----------------------------------

             Summary: Remove useless conf set in pyspark context
                 Key: SPARK-31142
                 URL: https://issues.apache.org/jira/browse/SPARK-31142
             Project: Spark
          Issue Type: Improvement
          Components: PySpark, Spark Core
    Affects Versions: 3.0.0, 3.1.0
            Reporter: Jackey Lee


In Pyspark/context.py, we extract the configuration of the "spark.executorEnv" prefix from conf and put it in env. But in PythonWorkerFactory, we have already obtained these environment variables from ProgressBuilder, thus we do not need the extra copy.
In addition, in some special environment deployments, user-configured environment variables such as "spark.executorEnv.Path=/usr/lib:$PATH" will be automatically parsed as actual machine confs. If we still directly copy and overwrite it, the configuration will be a wrong config.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org