You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2026/08/27 08:09:49 UTC
[PR] [SPARK-59048][ML] Reduce classification transform closure sizes [spark]
zhengruifeng opened a new pull request, #58336:
URL: https://github.com/apache/spark/pull/58336
### What changes were proposed in this pull request?
This PR adds protected column-expression hooks to `ClassificationModel` and
`ProbabilisticClassificationModel`. The shared transform implementations use these hooks to build
raw prediction, probability, and prediction columns. Their default implementations preserve the
existing row-based UDF behavior, while individual models can provide native Catalyst expressions or
UDFs backed by compact prediction state.
`LogisticRegressionModel` uses the new hooks to snapshot only coefficients, intercepts, mode, and
thresholds needed by each expression. Binary raw/probability thresholding uses native column
expressions, while the remaining UDFs call companion-object helpers without retaining the model.
### Why are the changes needed?
The previous classifier transform UDFs invoked bound model methods. As a result, their closures
retained the full model, including its parameter graph, even when a prediction stage needed only a
small subset of immutable scoring state.
The column-expression hooks provide a common design for classifiers to reduce those closures and to
use native Spark expressions where possible. This lowers driver memory pressure from retained query
plans on long-lived Spark Connect servers.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
The following checks were run locally:
```
build/sbt mllib/compile
build/sbt mllib/scalastyle
build/sbt 'mllib/testOnly org.apache.spark.ml.classification.ClassifierSuite org.apache.spark.ml.classification.LogisticRegressionSuite'
```
The focused test run completed 66 tests successfully, with one existing ignored test.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59048][ML] Reduce classification transform closure sizes [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #58336:
URL: https://github.com/apache/spark/pull/58336#issuecomment-5436711588
I ran an in-process comparison against a model subclass that recreated the previous bound-method
UDF expressions.
For closure size, the probe serialized each `ScalaUDF.function` with Spark's closure serializer
using a binary model with 4,096 features:
| Output columns | Before | After | Delta |
|---|---:|---:|---:|
| raw prediction | 41,825 B | 35,267 B | -15.7% |
| probability | 41,833 B | 35,527 B | -15.1% |
| prediction | 41,763 B | 35,184 B | -15.8% |
| raw + probability + prediction | 125,532 B (3 UDFs) | 37,605 B (2 UDFs) | -70.0% |
For execution time, the probe measured 20 alternating actions over 3,000,000 cached rows with 32
features. The table reports median times:
| Transform path | Before | After | Ratio |
|---|---:|---:|---:|
| prediction only | 214.710 ms | 191.852 ms | 0.894x |
| raw prediction to prediction | 501.883 ms | 503.178 ms | 1.003x |
| probability to prediction | 528.609 ms | 502.011 ms | 0.950x |
The temporary probe was removed after measurement. The affected binary and multinomial correctness
tests pass after the final UDF signature adjustment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59048][ML] Optimize ProbabilisticClassificationModel by reducing closure size and accelerating transform [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #58336:
URL: https://github.com/apache/spark/pull/58336#issuecomment-5486965577
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/2948ffa5589f65ad4885113031a8d2c1d681f2bd
- merged into branch-4.x https://github.com/apache/spark/commit/da266c169b0c7a37a07c9f1896991979c5e1989a
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59048][ML] Optimize ProbabilisticClassificationModel by reducing closure size and accelerating transform [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #58336: [SPARK-59048][ML] Optimize ProbabilisticClassificationModel by reducing closure size and accelerating transform
URL: https://github.com/apache/spark/pull/58336
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org