[jira] [Created] (SPARK-54502) Support off-heap memory mode for hash relations in broadcast hash join

["Hongze Zhang (Jira)" <ji...@apache.org>]

Hongze Zhang created SPARK-54502:
------------------------------------

             Summary: Support off-heap memory mode for hash relations in broadcast hash join
                 Key: SPARK-54502
                 URL: https://issues.apache.org/jira/browse/SPARK-54502
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 4.0.1
            Reporter: Hongze Zhang


Currently Spark is hard coded to use on-heap memory mode for broadcast hash join. It was because the broadcasted hash tables don't have a certain lifecycle so need to be managed by GC to avoid memory leak. Though this will cause waste of JVM heap memory when off-heap memory mode is enabled.

This will aim to improve the design and code to add off-heap support for broadcast hash join.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org