You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Vincent (JIRA)" <ji...@apache.org> on 2017/06/12 03:04:00 UTC
[jira] [Created] (SPARK-21058) potential SVD optimization
Vincent created SPARK-21058:
-------------------------------
Summary: potential SVD optimization
Key: SPARK-21058
URL: https://issues.apache.org/jira/browse/SPARK-21058
Project: Spark
Issue Type: Improvement
Components: ML, MLlib
Affects Versions: 2.1.1
Reporter: Vincent
In the current implementation, computeSVD will compute SVD for matrix A by computing AT*A first and svd on the Gramian matrix, we found that the Gramian matrix computation is the hot spot of the overall SVD computation. While svd on the Gramian matrix can benefit svd computation on the skinny matrix, for a non-skinny matrix, it could also become a huge overhead. So, is it possible to offer another option by computing svd on the original matrix instead of the Gramian matrix? We can decide which way to go by the ratio between numCols and numRows, or by simply settings from the user.
We have observed a handsome gain on a toy dataset by svd on the original matrix instead of the Gramian matrix, if the proposal is acceptable, we will start to work on the patch and gather more performance data.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org