[jira] [Created] (SPARK-59535) Run PCA covariance decomposition on an executor

["Ruifeng Zheng (Jira)" <ji...@apache.org>]

Ruifeng Zheng created SPARK-59535:
-------------------------------------

             Summary: Run PCA covariance decomposition on an executor
                 Key: SPARK-59535
                 URL: https://issues.apache.org/jira/browse/SPARK-59535
             Project: Spark
          Issue Type: Improvement
          Components: MLlib
    Affects Versions: 5.0.0
            Reporter: Ruifeng Zheng


For PCA with at most 65535 features, RowMatrix currently aggregates a packed covariance matrix
across executors, returns that large aggregate to the driver, expands it into a dense matrix, and
runs the local Breeze SVD on the driver. The driver must hold the packed matrix, dense covariance,
and SVD workspace even though only the smaller principal-component result needs to be returned.

Add an internal tree-aggregation variant that preserves the final aggregate as a single-partition
RDD. Use it in RowMatrix so covariance expansion and PCA decomposition run in the final executor
task, and return only the public method result to the driver. Public APIs and numerical behavior
remain unchanged.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org