You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2026/09/10 10:13:51 UTC
[PR] [SPARK-59398][ML][SQL] Add a SQL expression for ML vector affine transformations [spark]
zhengruifeng opened a new pull request, #58692:
URL: https://github.com/apache/spark/pull/58692
### What changes were proposed in this pull request?
This PR adds an internal `VectorAffineTransform` Catalyst expression, registered as
`ml_vector_affine_transform`, that computes `vector(i) * scale(i) + shift(i)` directly on the SQL
struct representation of MLlib vectors.
The expression supports interpreted and code-generated evaluation for dense and sparse vectors.
It preserves sparse output when the input vector is sparse and the shift is zero; a nonzero shift
produces a dense vector. This PR only adds the expression and does not use it in ML implementations.
### Why are the changes needed?
Several ML feature transformations perform per-feature scaling and shifting through Scala UDFs.
This expression provides a reusable Catalyst building block for future optimizations without
deserializing vectors into MLlib objects.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New tests cover interpreted and code-generated evaluation with dense and sparse vectors, sparse
output preservation, nonzero shifts, null and empty vectors, infinite and NaN values, and
mismatched dimensions.
```text
build/sbt 'catalyst/testOnly *VectorAffineTransformSuite'
./dev/lint-scala
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59398][ML][SQL] Add an internal expression for ML vector scaling and shifting [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #58692:
URL: https://github.com/apache/spark/pull/58692#issuecomment-5675085195
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/737e541d258cd5943e84498a7a7ff356702060d8
- merged into branch-4.x https://github.com/apache/spark/commit/756932640a201955707a810474db3634bce06bb6
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59398][ML][SQL] Add an internal expression for ML vector scaling and shifting [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #58692: [SPARK-59398][ML][SQL] Add an internal expression for ML vector scaling and shifting
URL: https://github.com/apache/spark/pull/58692
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org