[jira] [Commented] (SPARK-21058) potential SVD optimization

["Sean Owen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-21058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16046424#comment-16046424 ] 

Sean Owen commented on SPARK-21058:
-----------------------------------

I think we discussed this separately. The Gramian method is indeed appropriate for tall, skinny matrices. Just because computing the Gramian is the hotspot doesn't mean there's a faster way. How would you efficiently compute the SVD of any huge matrix?

If the matrix isn't huge, you can pull it into memory and decompose it with a library, sure. But that's not a Spark problem.

There's also a different SVD implementation in GraphX.

No, I would not start work on anything until you've addressed the above. 

> potential SVD optimization
> --------------------------
>
>                 Key: SPARK-21058
>                 URL: https://issues.apache.org/jira/browse/SPARK-21058
>             Project: Spark
>          Issue Type: Improvement
>          Components: ML, MLlib
>    Affects Versions: 2.1.1
>            Reporter: Vincent
>
> In the current implementation, computeSVD will compute SVD for matrix A by computing AT*A first and svd on the Gramian matrix, we found that the Gramian matrix computation is the hot spot of the overall SVD computation. While svd on the Gramian matrix can benefit svd computation on the skinny matrix, for a non-skinny matrix, it could also become a huge overhead. So, is it possible to offer another option by computing svd on the original matrix instead of the Gramian matrix? We can decide which way to go by the ratio between numCols and numRows, or by simply settings from the user.
> We have observed a handsome gain on a toy dataset by svd on the original matrix instead of the Gramian matrix, if the proposal is acceptable, we will start to work on the patch and gather more performance data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org