You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Tao Luo (JIRA)" <ji...@apache.org> on 2019/02/12 09:36:00 UTC

[jira] [Comment Edited] (SPARK-21492) Memory leak in SortMergeJoin

    [ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765378#comment-16765378 ] 

Tao Luo edited comment on SPARK-21492 at 2/12/19 9:35 AM:
----------------------------------------------------------

I'll take a stab at this jira, should have something to review today or tomorrow. 

[https://github.com/apache/spark/pull/23762] (took some inspiration from Zhan's patch)

I'd appreciate a review, and pointers on modifying code-gen portion.


was (Author: taoluo):
I'll take a stab at this jira, should have something to review today or tomorrow. 

> Memory leak in SortMergeJoin
> ----------------------------
>
>                 Key: SPARK-21492
>                 URL: https://issues.apache.org/jira/browse/SPARK-21492
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0, 2.3.0, 2.3.1, 3.0.0
>            Reporter: Zhan Zhang
>            Priority: Major
>
> In SortMergeJoin, if the iterator is not exhausted, there will be memory leak caused by the Sort. The memory is not released until the task end, and cannot be used by other operators causing performance drop or OOM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org