You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Saisai Shao (JIRA)" <ji...@apache.org> on 2016/05/26 07:24:12 UTC

[jira] [Commented] (SPARK-15548) Shuffle file remains with dynamic allocation on YARN

    [ https://issues.apache.org/jira/browse/SPARK-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15301670#comment-15301670 ] 

Saisai Shao commented on SPARK-15548:
-------------------------------------

From my understanding this problem is not only a yarn specific problem, this problem should be existed in all the cluster managers which use external shuffle service. Basically external shuffle service doesn't know when to remove unnecessary shuffle data, so an easy way is to hold the data until application is finished. 

> Shuffle file remains with dynamic allocation on YARN
> ----------------------------------------------------
>
>                 Key: SPARK-15548
>                 URL: https://issues.apache.org/jira/browse/SPARK-15548
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.6.1
>            Reporter: He Tianyi
>
> With dynamic allocation on YARN, executor may get exited with shuffle file left on disk. 
> These files will not be removed until application exists. 
> This is buggy for long running application (e.g. serving interactive queries) .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org