You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Szehon Ho (Jira)" <ji...@apache.org> on 2026/09/27 07:31:00 UTC

[jira] [Commented] (SPARK-54089) Add off-heap mode support for on-heap-only memory consumers

    [ https://issues.apache.org/jira/browse/SPARK-54089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18119771#comment-18119771 ] 

Szehon Ho commented on SPARK-54089:
-----------------------------------

Removing Fix Version/s 4.2.1 during release cleanup: this umbrella issue was reopened to add more subtasks and remains incomplete. Leaving it reopened for future work.

> Add off-heap mode support for on-heap-only memory consumers
> -----------------------------------------------------------
>
>                 Key: SPARK-54089
>                 URL: https://issues.apache.org/jira/browse/SPARK-54089
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 4.2.0
>            Reporter: Hongze Zhang
>            Priority: Major
>
> There are a few memory consumers that only support on-heap mode. Including:
> # LongToUnsafeRowMap (for long key hash join)
> # ExternalSorter (for non-serializable sort-based shuffle)
> It's ideal to make them support off-heap when Spark is running with off-heap memory mode. For 3rd accelerator plugins that leverage Spark off-heap memory, having the large allocations in vanilla Spark supporting off-heap mode will significantly ease the memory settings for running the mixed query plan that contains both vanilla Spark and offloaded computations. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org