[jira] [Resolved] (SPARK-29814) Missing persist on sources in mllib.feature.PCA

["Sean R. Owen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-29814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean R. Owen resolved SPARK-29814.
----------------------------------
    Resolution: Duplicate

> Missing persist on sources in mllib.feature.PCA
> -----------------------------------------------
>
>                 Key: SPARK-29814
>                 URL: https://issues.apache.org/jira/browse/SPARK-29814
>             Project: Spark
>          Issue Type: Sub-task
>          Components: MLlib
>    Affects Versions: 2.4.3
>            Reporter: Dong Wang
>            Priority: Major
>
> The rdd is used in more than one actions: first() and actions in computePrincipalComponentsAndExplainedVariance(), so it needs to be persisted.
> {code:scala}
>   def fit(sources: RDD[Vector]): PCAModel = {
>     // first use rdd sources on action first()
>     val numFeatures = sources.first().size
>     require(k <= numFeatures,
>       s"source vector size $numFeatures must be no less than k=$k")
>     require(PCAUtil.memoryCost(k, numFeatures) < Int.MaxValue,
>       "The param k and numFeatures is too large for SVD computation. " +
>       "Try reducing the parameter k for PCA, or reduce the input feature " +
>       "vector dimension to make this tractable.")
>     val mat = new RowMatrix(sources)
>     // second use rdd sources
>     val (pc, explainedVariance) = mat.computePrincipalComponentsAndExplainedVariance(k)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org