You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "David Vogelbacher (JIRA)" <ji...@apache.org> on 2018/11/01 14:43:00 UTC
[jira] [Commented] (SPARK-24437) Memory leak in
UnsafeHashedRelation
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671686#comment-16671686 ]
David Vogelbacher commented on SPARK-24437:
-------------------------------------------
Hey [~mgaido], I am seeing something similar in one of our long running applications.
The longer it runs the higher heap usage in the driver grows. Taking a heap dump, most of its space is taken up by {{UnsafeHashedRelation}} instances. Looking at their paths to GC, they are strongly referenced by {{InMemoryRelation}} instances:
!Screen Shot 2018-11-01 at 10.38.30 AM.png!
So it looks like if dataframes are cached the {{UnsafeHashedRelation}}s are never cleaned up, because they are strongly references by generated codegen plan?
The context cleaner can only clean them up when they are no longer strongly referenced.
> Memory leak in UnsafeHashedRelation
> -----------------------------------
>
> Key: SPARK-24437
> URL: https://issues.apache.org/jira/browse/SPARK-24437
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: gagan taneja
> Priority: Critical
> Attachments: Screen Shot 2018-05-30 at 2.05.40 PM.png, Screen Shot 2018-05-30 at 2.07.22 PM.png, Screen Shot 2018-11-01 at 10.38.30 AM.png
>
>
> There seems to memory leak with org.apache.spark.sql.execution.joins.UnsafeHashedRelation
> We have a long running instance of STS.
> With each query execution requiring Broadcast Join, UnsafeHashedRelation is getting added for cleanup in ContextCleaner. This reference of UnsafeHashedRelation is being held at some other Collection and not becoming eligible for GC and because of this ContextCleaner is not able to clean it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org