You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "XiDuo You (Jira)" <ji...@apache.org> on 2022/09/28 07:20:00 UTC

[jira] [Created] (SPARK-40594) Eagerly release hashed relation in ShuffledHashJoin

XiDuo You created SPARK-40594:
---------------------------------

             Summary: Eagerly release hashed relation in ShuffledHashJoin
                 Key: SPARK-40594
                 URL: https://issues.apache.org/jira/browse/SPARK-40594
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.4.0
            Reporter: XiDuo You


ShuffledHashJoin releases the built hashed relation at the end of task using taskCompletionListener. It is not always good enough for complex sql query.

If a smj on the top of the shj, then the hashed relation in shj would be leak. All rows have been consumed in sort before smj and then in smj the buffered rows can not allocate the memory which is hold by hashed relation. Then it causes unnecessary spill.

It is a common case in multi-join, since AQE supports convert smj to shj at runtime.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org