You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2026/07/24 00:33:20 UTC
[PR] [SPARK-58293][ML] Delay VectorIndexerModel attribute initialization [spark]
zhengruifeng opened a new pull request, #57483:
URL: https://github.com/apache/spark/pull/57483
### What changes were proposed in this pull request?
Makes `VectorIndexerModel.partialFeatureAttributes` lazy and adds a regression test showing that a high-dimensional model does not eagerly retain the derived attribute array.
### Why are the changes needed?
The attribute array is derived state used only while preparing output metadata. Constructing it eagerly allocates one attribute per feature when a model is fitted or loaded, even when it is never transformed. Deferring the allocation avoids that unnecessary retained state.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added a `VectorIndexerSuite` regression test using a high-dimensional model whose eager derived attributes would exceed the expected size bound.
The test suite was not run locally.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58293][ML] Delay VectorIndexerModel attribute initialization [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #57483:
URL: https://github.com/apache/spark/pull/57483#issuecomment-5068799729
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/6d6e2dc77bfc96638751687e49da2fe1b3936131
- merged into branch-4.x https://github.com/apache/spark/commit/7eb542ae5d33af2b83123de5d46d5d246675ba87
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58293][ML] Delay VectorIndexerModel attribute initialization [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #57483: [SPARK-58293][ML] Delay VectorIndexerModel attribute initialization
URL: https://github.com/apache/spark/pull/57483
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org