You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hidayat Teonadi (JIRA)" <ji...@apache.org> on 2018/11/12 18:17:00 UTC
[jira] [Commented] (SPARK-26020) shuffle data from spark streaming
not cleaned up when External Shuffle Service is enabled
[ https://issues.apache.org/jira/browse/SPARK-26020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684200#comment-16684200 ]
Hidayat Teonadi commented on SPARK-26020:
-----------------------------------------
linking related ticket SPARK-17233
> shuffle data from spark streaming not cleaned up when External Shuffle Service is enabled
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-26020
> URL: https://issues.apache.org/jira/browse/SPARK-26020
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
> Affects Versions: 2.3.0
> Reporter: Hidayat Teonadi
> Priority: Major
>
> Hi, I'm running Spark Streaming on YARN and have enabled dynamic allocation + External Spark Shuffle Service. I'm noticing that during the lifetime of my spark streaming application, the nm appcache folder is building up with blockmgr directories (filled with shuffle_*.data).
> I understand why the data is not immediately cleaned up due to dynamic executor allocation, but will any cleanup of these directories be done during the lifetime of the spark streaming application ? Some of these shuffle data are generated as part of spark jobs/stages that have already completed.
> I've initially designed the application to run perpetually, but without any cleanup eventually the cluster will run out of disk and crash the application.
> [https://stackoverflow.com/questions/52923386/spark-streaming-job-doesnt-delete-shuffle-files] suggests a stop gap solution of cleaning up via cron.
> YARN-8991is the ticket I filed for YARN, who suggested me to file a ticket for spark. Appreciate any help.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org