You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/10/08 08:31:00 UTC

[jira] [Assigned] (SPARK-40594) Eagerly release hashed relation in ShuffledHashJoin

     [ https://issues.apache.org/jira/browse/SPARK-40594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-40594:
------------------------------------

    Assignee:     (was: Apache Spark)

> Eagerly release hashed relation in ShuffledHashJoin
> ---------------------------------------------------
>
>                 Key: SPARK-40594
>                 URL: https://issues.apache.org/jira/browse/SPARK-40594
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.4.0
>            Reporter: XiDuo You
>            Priority: Major
>
> ShuffledHashJoin releases the built hashed relation at the end of task using taskCompletionListener. It is not always good enough for complex sql query.
> If a smj or window on the top of the shj, then the hashed relation in shj would be leak. All rows have been consumed in sort before smj or window then the buffer can not allocate the memory which is hold by hashed relation. Then it causes unnecessary spill.
> It is a common case in multi-join, since AQE supports convert smj to shj at runtime.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org