You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "yangpengyu (JIRA)" <ji...@apache.org> on 2019/04/09 14:54:00 UTC
[jira] [Created] (SPARK-27417) CLONE - ExternalSorter and
ExternalAppendOnlyMap should free shuffle memory in their stop() methods
yangpengyu created SPARK-27417:
----------------------------------
Summary: CLONE - ExternalSorter and ExternalAppendOnlyMap should free shuffle memory in their stop() methods
Key: SPARK-27417
URL: https://issues.apache.org/jira/browse/SPARK-27417
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.3.1, 1.4.1, 1.5.1, 1.6.0
Reporter: yangpengyu
Assignee: Josh Rosen
Fix For: 1.6.0
I discovered multiple leaks of shuffle memory while working on my memory manager consolidation patch, which added the ability to do strict memory leak detection for the bookkeeping that used to be performed by the ShuffleMemoryManager. This uncovered a handful of places where tasks can acquire execution/shuffle memory but never release it, starving themselves of memory.
Problems that I found:
* {{ExternalSorter.stop()}} should release the sorter's shuffle/execution memory.
* BlockStoreShuffleReader should call {{ExternalSorter.stop()}} using a {{CompletionIterator}}.
* {{ExternalAppendOnlyMap}} exposes no equivalent of {{stop()}} for freeing its resources.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org