You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by st553 <st...@gmail.com> on 2014/09/23 21:49:04 UTC
Re: spark1.0 principal component analysis
sowen wrote
> it seems that the singular values from the SVD aren't returned, so I don't
> know that you can access this directly
Its not clear to me why these aren't returned? The S matrix would be useful
to determine a reasonable value for K.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark1-0-principal-component-analysis-tp9249p14919.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark1.0 principal component analysis
Posted by "Evan R. Sparks" <ev...@gmail.com>.
In its current implementation, the principal components are computed in
MLlib in two steps:
1) In a distributed fashion, compute the covariance matrix - the result is
a local matrix.
2) On this local matrix, compute the SVD.
The sorting comes from the SVD. If you want to get the eigenvalues out, you
can simply run step 1 yourself on your RowMatrix via the (experimental)
computeCovariance() method, and then run SVD on the result using a library
like breeze.
- Evan
On Tue, Sep 23, 2014 at 12:49 PM, st553 <st...@gmail.com> wrote:
> sowen wrote
> > it seems that the singular values from the SVD aren't returned, so I
> don't
> > know that you can access this directly
>
> Its not clear to me why these aren't returned? The S matrix would be useful
> to determine a reasonable value for K.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark1-0-principal-component-analysis-tp9249p14919.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>