You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "L. C. Hsieh (Jira)" <ji...@apache.org> on 2026/10/13 19:59:00 UTC

[jira] [Assigned] (SPARK-60112) Use identity equality for memory consumers

     [ https://issues.apache.org/jira/browse/SPARK-60112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

L. C. Hsieh reassigned SPARK-60112:
-----------------------------------

    Assignee: L. C. Hsieh

> Use identity equality for memory consumers
> ------------------------------------------
>
>                 Key: SPARK-60112
>                 URL: https://issues.apache.org/jira/browse/SPARK-60112
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 4.4.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Major
>              Labels: pull-request-available
>
> TaskMemoryManager keeps spillable consumers in a value-equality HashSet<MemoryConsumer>. A consumer whose equals is value-based (e.g. a Scala case class) is deduplicated against an equal but distinct consumer: the second instance is never offered for spilling and its memory is missing from the per-consumer memory breakdown on OOM. SPARK-60071 fixed this for HybridRowQueue. Tracking consumers in an identity-based set (e.g. Collections.newSetFromMap(new IdentityHashMap<>())) would protect every consumer in one place.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org