You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by me...@apache.org on 2014/07/12 08:26:53 UTC
git commit: use specialized axpy in RowMatrix for SVD
Repository: spark
Updated Branches:
refs/heads/master 559608693 -> d38887b8a
use specialized axpy in RowMatrix for SVD
After running some more tests on large matrix, found that the BV axpy (breeze/linalg/Vector.scala, axpy) is slower than the BSV axpy (breeze/linalg/operators/SparseVectorOps.scala, sv_dv_axpy), 8s v.s. 2s for each multiplication. The BV axpy operates on an iterator while BSV axpy directly operates on the underlying array. I think the overhead comes from creating the iterator (with a zip) and advancing the pointers.
Author: Li Pu <lp...@twitter.com>
Author: Xiangrui Meng <me...@databricks.com>
Author: Li Pu <li...@outlook.com>
Closes #1378 from vrilleup/master and squashes the following commits:
6fb01a3 [Li Pu] use specialized axpy in RowMatrix
5255f2a [Li Pu] Merge remote-tracking branch 'upstream/master'
7312ec1 [Li Pu] very minor comment fix
4c618e9 [Li Pu] Merge pull request #1 from mengxr/vrilleup-master
a461082 [Xiangrui Meng] make superscript show up correctly in doc
861ec48 [Xiangrui Meng] simplify axpy
62969fa [Xiangrui Meng] use BDV directly in symmetricEigs change the computation mode to local-svd, local-eigs, and dist-eigs update tests and docs
c273771 [Li Pu] automatically determine SVD compute mode and parameters
7148426 [Li Pu] improve RowMatrix multiply
5543cce [Li Pu] improve svd api
819824b [Li Pu] add flag for dense svd or sparse svd
eb15100 [Li Pu] fix binary compatibility
4c7aec3 [Li Pu] improve comments
e7850ed [Li Pu] use aggregate and axpy
827411b [Li Pu] fix EOF new line
9c80515 [Li Pu] use non-sparse implementation when k = n
fe983b0 [Li Pu] improve scala style
96d2ecb [Li Pu] improve eigenvalue sorting
e1db950 [Li Pu] SPARK-1782: svd for sparse matrix using ARPACK
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d38887b8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d38887b8
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d38887b8
Branch: refs/heads/master
Commit: d38887b8a0d00a11d7cf9393e7cb0918c3ec7a22
Parents: 5596086
Author: Li Pu <lp...@twitter.com>
Authored: Fri Jul 11 23:26:47 2014 -0700
Committer: Xiangrui Meng <me...@databricks.com>
Committed: Fri Jul 11 23:26:47 2014 -0700
----------------------------------------------------------------------
.../apache/spark/mllib/linalg/distributed/RowMatrix.scala | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/d38887b8/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
----------------------------------------------------------------------
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
index 711e32a..f4c403b 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
@@ -83,7 +83,13 @@ class RowMatrix(
seqOp = (U, r) => {
val rBrz = r.toBreeze
val a = rBrz.dot(vbr.value)
- brzAxpy(a, rBrz, U.asInstanceOf[BV[Double]])
+ rBrz match {
+ // use specialized axpy for better performance
+ case _: BDV[_] => brzAxpy(a, rBrz.asInstanceOf[BDV[Double]], U)
+ case _: BSV[_] => brzAxpy(a, rBrz.asInstanceOf[BSV[Double]], U)
+ case _ => throw new UnsupportedOperationException(
+ s"Do not support vector operation from type ${rBrz.getClass.getName}.")
+ }
U
},
combOp = (U1, U2) => U1 += U2