You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "yangpengyu (JIRA)" <ji...@apache.org> on 2019/04/09 14:41:00 UTC
[jira] [Commented] (SPARK-11293) ExternalSorter and
ExternalAppendOnlyMap should free shuffle memory in their stop() methods
[ https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16813495#comment-16813495 ]
yangpengyu commented on SPARK-11293:
------------------------------------
I hit the sam problem when I run TPCH test on spark1.6.0.
my dataset scale is SF=1000，
Environment as follows:
1Master 3 Worker,
onHeapMemory=10g,
offHeapMemory=20g,
24threads/Worker
the query3 and query17 detected memory leak.Some logs are as follows:
9/04/09 21:57:59 ERROR Executor: Managed memory leak detected; size = 536870912 bytes, TID = 2685
41 19/04/09 21:58:16 WARN TaskMemoryManager: leak 512.0 MB memory from org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@385b3b3f
42 19/04/09 21:58:16 ERROR Executor: Managed memory leak detected; size = 536870912 bytes, TID = 2683
43 19/04/09 21:58:16 WARN TaskMemoryManager: leak 512.0 MB memory from org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@35be7b55
44 19/04/09 21:58:16 ERROR Executor: Managed memory leak detected; size = 536870912 bytes, TID = 2703
45 19/04/09 21:58:20 WARN TaskMemoryManager: leak 512.0 MB memory from org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@50f93582
46 19/04/09 21:58:20 ERROR Executor: Managed memory leak detected; size = 536870912 bytes, TID = 2709
47 19/04/09 21:58:21 WARN TaskMemoryManager: leak 512.0 MB memory from org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@28e3ec7a
48 19/04/09 21:58:21 ERROR Executor: Managed memory leak detected; size = 536870912 bytes, TID = 2723
49 19/04/09 21:59:50 WARN TaskMemoryManager: leak 512.0 MB memory from org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@5b2f5dbc
50 19/04/09 21:59:50 ERROR Executor: Managed memory leak detected; size = 536870912 bytes, TID = 2687
51 19/04/09 22:00:50 WARN TransportChannelHandler: Exception in connection from hw083/172.18.11.83:42989
> ExternalSorter and ExternalAppendOnlyMap should free shuffle memory in their stop() methods
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-11293
> URL: https://issues.apache.org/jira/browse/SPARK-11293
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.1, 1.4.1, 1.5.1, 1.6.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Critical
> Fix For: 1.6.0
>
>
> I discovered multiple leaks of shuffle memory while working on my memory manager consolidation patch, which added the ability to do strict memory leak detection for the bookkeeping that used to be performed by the ShuffleMemoryManager. This uncovered a handful of places where tasks can acquire execution/shuffle memory but never release it, starving themselves of memory.
> Problems that I found:
> * {{ExternalSorter.stop()}} should release the sorter's shuffle/execution memory.
> * BlockStoreShuffleReader should call {{ExternalSorter.stop()}} using a {{CompletionIterator}}.
> * {{ExternalAppendOnlyMap}} exposes no equivalent of {{stop()}} for freeing its resources.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org