You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Wenchen Fan (JIRA)" <ji...@apache.org> on 2019/02/01 05:49:00 UTC

[jira] [Assigned] (SPARK-26525) Fast release memory of ShuffleBlockFetcherIterator

     [ https://issues.apache.org/jira/browse/SPARK-26525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan reassigned SPARK-26525:
-----------------------------------

    Assignee: liupengcheng

> Fast release memory of ShuffleBlockFetcherIterator
> --------------------------------------------------
>
>                 Key: SPARK-26525
>                 URL: https://issues.apache.org/jira/browse/SPARK-26525
>             Project: Spark
>          Issue Type: Improvement
>          Components: Shuffle
>    Affects Versions: 2.3.2
>            Reporter: liupengcheng
>            Assignee: liupengcheng
>            Priority: Major
>
> Currently, spark would not release ShuffleBlockFetcherIterator until the whole task finished.
> In some conditions, it incurs memory leak.
> An example is Shuffle -> map -> Coalesce(shuffle = false). Each ShuffleBlockFetcherIterator contains  some metas about MapStatus(blocksByAddress) and each ShuffleMapTask will keep n(max to shuffle partitions) shuffleBlockFetcherIterator for they are refered by onCompleteCallbacks of TaskContext, in some case, it may take huge memory and the memory will not released until the task finished.
> Actually, We can release ShuffleBlockFetcherIterator as soon as it's consumed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org