You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Michael Spector (JIRA)" <ji...@apache.org> on 2018/09/06 11:11:00 UTC
[jira] [Commented] (SPARK-23670) Memory leak of
SparkPlanGraphWrapper in sparkUI
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605642#comment-16605642 ]
Michael Spector commented on SPARK-23670:
-----------------------------------------
The bug still happens in Apache Spark 2.3.1:
!heapdump_OOM.png|width=1595,height=966!
These char[] objects occupy 88% of driver memory.
> Memory leak of SparkPlanGraphWrapper in sparkUI
> -----------------------------------------------
>
> Key: SPARK-23670
> URL: https://issues.apache.org/jira/browse/SPARK-23670
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Myroslav Lisniak
> Assignee: Myroslav Lisniak
> Priority: Major
> Fix For: 2.3.1, 2.4.0
>
> Attachments: heap.png, heapdump_OOM.png
>
>
> Memory leak on driver for a long time running application. We have application using structured streaming and running 48 hours. But driver fails with out of memory after 25 hours. After investigating heap dump we found that most of the memory was occupied with a lot of *SparkPlanGraphWrapper* objects inside *InMemoryStore*.
> Application was run with option:
> --driver-memory 4G
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org