[jira] [Commented] (SPARK-60071) Use identity equality for HybridRowQueue

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-60071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126605#comment-18126605 ] 

Dongjoon Hyun commented on SPARK-60071:
---------------------------------------

This is reverted from branch-4.2 due to the compilation failure.
- https://github.com/apache/spark/commit/e8e8f37a7262b1081b4001c4a34fd1d4ac7caf9a

> Use identity equality for HybridRowQueue
> ----------------------------------------
>
>                 Key: SPARK-60071
>                 URL: https://issues.apache.org/jira/browse/SPARK-60071
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.5.8, 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 4.3.0
>
>
> HybridRowQueue is a case class, so two queues created in the same task with the same TaskMemoryManager, temp dir, number of fields, SerializerManager and lockFree flag are equal. Because HybridRowQueue is a MemoryConsumer (via HybridQueue), this breaks memory management:
> 1. TaskMemoryManager tracks consumers in a HashSet. A second queue that equals an already registered one is never added, so it is never offered for spilling when another consumer needs memory, and its memory is not attributed in the memory usage breakdown on OOM.
> 2. HybridQueue.spill(size, trigger) returns 0 when trigger == this, which uses equals. A queue therefore refuses to spill when an equal but distinct queue triggers the spill.
> This can happen in a real query, e.g. when several partitions evaluated by a Python UDF node are processed in one task (coalesce), or when both sides of a sort-merge join contain Python UDF eval nodes with the same input width.
> The fix is to give HybridRowQueue identity equality by making it a regular class.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org