You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jungtaek Lim (JIRA)" <ji...@apache.org> on 2019/04/26 06:11:00 UTC

[jira] [Created] (SPARK-27575) Spark overwrites existing value of spark.yarn.dist.* instead of merging value

Jungtaek Lim created SPARK-27575:
------------------------------------

             Summary: Spark overwrites existing value of spark.yarn.dist.* instead of merging value
                 Key: SPARK-27575
                 URL: https://issues.apache.org/jira/browse/SPARK-27575
             Project: Spark
          Issue Type: Bug
          Components: Spark Submit, YARN
    Affects Versions: 2.4.0
            Reporter: Jungtaek Lim


If we specify `--files` arg when submitting app where configuration has "spark.yarn.dist.files", SparkSubmit overwrites the new files (files provided in arg) instead of merging existing value in configuration.

Same issue happens also on "spark.yarn.dist.pyFiles", "spark.yarn.dist.jars", "spark.yarn.dist.archives".

 

While I encountered the issue in Spark 2.4.0, I can see the issue from codebase in master branch and also branch-2.3 as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org