You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2026/08/04 08:43:41 UTC
[PR] [SPARK-58550][MLLIB] Delay GaussianMixture aggregation allocations [spark]
zhengruifeng opened a new pull request, #57750:
URL: https://github.com/apache/spark/pull/57750
### What changes were proposed in this pull request?
This PR delays allocation of `ExpectationSum` in `GaussianMixture` until an aggregation partition receives its first point. Empty partitions and intermediate tree branches use `null` as the aggregation identity.
### Why are the changes needed?
The previous `treeAggregate` zero value eagerly allocated the mixture weights, means, and one dense covariance matrix per cluster. That can be substantial for large models and is also serialized to every partition, even empty ones.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Not run (not requested). Added coverage using an RDD with an empty partition. Static checks: `git diff --check`, ASCII scan, and changed-line length scan.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58550][ML] Delay GaussianMixture aggregation allocations [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57750:
URL: https://github.com/apache/spark/pull/57750#issuecomment-5189643353
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/811e666add4063f45e9cf04b55015c2ee9521501
- merged into branch-4.x https://github.com/apache/spark/commit/2d2cfc492c3c79c1dbdf1863bec63be764d8bdea
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58550][ML] Delay GaussianMixture aggregation allocations [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57750:
URL: https://github.com/apache/spark/pull/57750#issuecomment-5189637757
Thank you @zhengruifeng and @HyukjinKwon!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58550][ML] Delay GaussianMixture aggregation allocations [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b closed pull request #57750: [SPARK-58550][ML] Delay GaussianMixture aggregation allocations
URL: https://github.com/apache/spark/pull/57750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org