[jira] [Resolved] (SPARK-20741) SparkSubmit does not clean up after uploading spark_libs to the distributed cache

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-20741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-20741.
-------------------------------
       Resolution: Fixed
    Fix Version/s: 2.2.0

Issue resolved by pull request 17986
[https://github.com/apache/spark/pull/17986]

> SparkSubmit does not clean up after uploading spark_libs to the distributed cache
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-20741
>                 URL: https://issues.apache.org/jira/browse/SPARK-20741
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Submit
>    Affects Versions: 2.1.1
>            Reporter: Lior Regev
>            Priority: Minor
>             Fix For: 2.2.0
>
>
> Running spark submit has to distribute Spark's JARs to a distributed cache in order for the executors to access it.
> When neither spark.yarn.jars or spark.yarn.archive is provided, SparkSubmit creates a ZIP of all the JARs in $SPARK_HOME/jars and uploads it to the distributed cache.
> After uploading the ZIP file, SparkSubmit does not delete the local copy of it. This, in turn can cause the disk on the local machine to fill up (200MB at a time) until no more submissions are possible.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org