[jira] [Resolved] (SPARK-59501) Unnecessary allocations in unchanged TreeNode traversals

["Gengliang Wang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gengliang Wang resolved SPARK-59501.
------------------------------------
    Fix Version/s: 4.4.0
       Resolution: Fixed

Issue resolved by pull request 58800
[https://github.com/apache/spark/pull/58800]

> Unnecessary allocations in unchanged TreeNode traversals
> --------------------------------------------------------
>
>                 Key: SPARK-59501
>                 URL: https://issues.apache.org/jira/browse/SPARK-59501
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Ben Hollis
>            Assignee: Ben Hollis
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 4.4.0
>
>
> Catalyst rules repeatedly traverse large, immutable plan and expression trees that they mostly leave unchanged. Even when a node and its subtree are not modified, the generic TreeNode traversal still allocates on every node.
> This cost is paid by every applicable analyzer and optimizer rule, on every pass, and scales with tree size, so queries whose analyzed plan expands into hundreds of branches amplify it. On a synthetic 500-branch compatibility view (4,001 analyzed plan nodes, ~378K expression occurrences), the unchanged-tree allocation overhead dominates optimizer time.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org