You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Nick Young (Jira)" <ji...@apache.org> on 2024/06/05 20:37:00 UTC

[jira] [Created] (SPARK-48544) Reduce memory pressure of TreeNode BitSets

Nick Young created SPARK-48544:
----------------------------------

             Summary: Reduce memory pressure of TreeNode BitSets
                 Key: SPARK-48544
                 URL: https://issues.apache.org/jira/browse/SPARK-48544
             Project: Spark
          Issue Type: Improvement
          Components: Optimizer
    Affects Versions: 4.0.0
            Reporter: Nick Young


By default, the TreeNode `treePatternBits` and `ineffectiveRules` variables are initialized to a `BitSet`, which takes up memory even when empty. For plans with a high number of TreeNodes, this adds unnecessary memory pressure, especially for large expression trees. Reduce the memory pressure by lazily initializing this variable.

For `treePatternBits` we will observe no benefit because these are always populated. However, for `ineffectiveRules` it is not necessary to initialize this for *every* node. We should make this variable lazy, being careful to not initialize even when `get` is called.

Related to SPARK-48047.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org