You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2026/08/27 05:34:27 UTC

[PR] Revert "[SPARK-58856][ML] Avoid per-leaf map for tree predictions" [spark]

zhengruifeng opened a new pull request, #58332:
URL: https://github.com/apache/spark/pull/58332

   ### What changes were proposed in this pull request?
   
   This PR reverts #57389 (`bc9ee80d0de08f4cca1ac8518e4b6bee64cc544d`) on `master`.
   
   It restores the transient lazy leaf-to-index map used by `predictLeaf` and removes the
   post-construction leaf-index field, initialization passes, and associated persistence assertions.
   The resulting eight files exactly match their versions immediately before #57389.
   
   ### Why are the changes needed?
   
   #57389 replaces the per-leaf map by constructing an unindexed node graph and then rebuilding the
   entire tree to attach indices. Although trees in an ensemble are processed sequentially, this adds
   O(number of nodes in one tree) temporary driver memory during training, loading, and old-model
   conversion. A sufficiently large individual tree can therefore cause an out-of-memory failure.
   
   Reverting removes that regression while an implementation that assigns immutable indices during
   initial node construction is considered separately in #58330.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No compared with a released Spark version. This restores the behavior that existed before the
   unreleased change in #57389. Leaf IDs and their traversal order are unchanged.
   
   ### How was this patch tested?
   
   The revert was verified to apply cleanly to current `master`, and all affected files were compared
   with their pre-#57389 versions. Targeted ML tests and Scala lint have not been run yet; the PR was
   opened first as requested.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: OpenAI Codex (GPT-5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] Revert "[SPARK-58856][ML] Avoid per-leaf map for tree predictions" [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #58332: Revert "[SPARK-58856][ML] Avoid per-leaf map for tree predictions"
URL: https://github.com/apache/spark/pull/58332


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] Revert "[SPARK-58856][ML] Avoid per-leaf map for tree predictions" [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #58332:
URL: https://github.com/apache/spark/pull/58332#issuecomment-5435616683

   **Merge Summary:**
   - merged into master https://github.com/apache/spark/commit/e2cc1d64ab363aa9fa5e4b0bdaffe20d93ff98df
   - merged into branch-4.x https://github.com/apache/spark/commit/85f46463a68e2a2b60270055a907fbc3dd2e737b
   
   *Posted by `merge_spark_pr.py`*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org