You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by phoenix bai <mi...@gmail.com> on 2014/08/21 11:07:22 UTC
Matrix multiplication in spark
there is RowMatrix implemented in spark.
and I check for a while but failed to find any matrix operations (like
multiplication etc) are defined in the class yet.
so, my question is, if I want to do matrix multiplication, (to do vector x
matrix multiplication to be precise), need to convert the vector/matrix to
the the matrix type defined in breeze package right?
thanks
Re: Matrix multiplication in spark
Posted by Victor Tso-Guillen <vt...@paxata.com>.
Scala defines transpose.
On Thu, Aug 21, 2014 at 4:22 PM, x <wa...@gmail.com> wrote:
> Yes.
> Now Spark API doesn't provide transpose function. You have to define it
> like below.
>
> def transpose(m: Array[Array[Double]]): Array[Array[Double]] = {
> (for {
> c <- m(0).indices
> } yield m.map(_(c)) ).toArray
> }
>
> xj @ Tokyo
>
>
> On Thu, Aug 21, 2014 at 10:12 PM, phoenix bai <mi...@gmail.com>
> wrote:
>
>> this is exactly what I was looking for. thank you.
>>
>> one thing though, it doesn`t have transpose() function defined, so I have
>> to do the transpose myself for the localMat in your case.
>> hoping it will be supported in the future :-)
>>
>>
>>
>> On Thu, Aug 21, 2014 at 7:30 PM, x <wa...@gmail.com> wrote:
>>
>>> You could create a distributed matrix with RowMatrix.
>>>
>>> val rmat = new RowMatrix(rows)
>>>
>>> And then make a local DenseMatrix.
>>>
>>> val localMat = Matrices.dense(m, n, mat)
>>>
>>> Then multiply them.
>>>
>>> rmat.multiply(localMat)
>>>
>>>
>>> xj @ Tokyo
>>>
>>> On Thu, Aug 21, 2014 at 6:37 PM, Sean Owen <so...@cloudera.com> wrote:
>>>
>>>> Are you trying to multiply dense or sparse matrices? if sparse, are
>>>> they very large -- meaning, are you looking for distributed
>>>> operations?
>>>>
>>>> On Thu, Aug 21, 2014 at 10:07 AM, phoenix bai <mi...@gmail.com>
>>>> wrote:
>>>> > there is RowMatrix implemented in spark.
>>>> > and I check for a while but failed to find any matrix operations (like
>>>> > multiplication etc) are defined in the class yet.
>>>> >
>>>> > so, my question is, if I want to do matrix multiplication, (to do
>>>> vector x
>>>> > matrix multiplication to be precise), need to convert the
>>>> vector/matrix to
>>>> > the the matrix type defined in breeze package right?
>>>> >
>>>> > thanks
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>
>>
>
Re: Matrix multiplication in spark
Posted by x <wa...@gmail.com>.
Yes.
Now Spark API doesn't provide transpose function. You have to define it
like below.
def transpose(m: Array[Array[Double]]): Array[Array[Double]] = {
(for {
c <- m(0).indices
} yield m.map(_(c)) ).toArray
}
xj @ Tokyo
On Thu, Aug 21, 2014 at 10:12 PM, phoenix bai <mi...@gmail.com> wrote:
> this is exactly what I was looking for. thank you.
>
> one thing though, it doesn`t have transpose() function defined, so I have
> to do the transpose myself for the localMat in your case.
> hoping it will be supported in the future :-)
>
>
>
> On Thu, Aug 21, 2014 at 7:30 PM, x <wa...@gmail.com> wrote:
>
>> You could create a distributed matrix with RowMatrix.
>>
>> val rmat = new RowMatrix(rows)
>>
>> And then make a local DenseMatrix.
>>
>> val localMat = Matrices.dense(m, n, mat)
>>
>> Then multiply them.
>>
>> rmat.multiply(localMat)
>>
>>
>> xj @ Tokyo
>>
>> On Thu, Aug 21, 2014 at 6:37 PM, Sean Owen <so...@cloudera.com> wrote:
>>
>>> Are you trying to multiply dense or sparse matrices? if sparse, are
>>> they very large -- meaning, are you looking for distributed
>>> operations?
>>>
>>> On Thu, Aug 21, 2014 at 10:07 AM, phoenix bai <mi...@gmail.com>
>>> wrote:
>>> > there is RowMatrix implemented in spark.
>>> > and I check for a while but failed to find any matrix operations (like
>>> > multiplication etc) are defined in the class yet.
>>> >
>>> > so, my question is, if I want to do matrix multiplication, (to do
>>> vector x
>>> > matrix multiplication to be precise), need to convert the
>>> vector/matrix to
>>> > the the matrix type defined in breeze package right?
>>> >
>>> > thanks
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>
Re: Matrix multiplication in spark
Posted by x <wa...@gmail.com>.
You could create a distributed matrix with RowMatrix.
val rmat = new RowMatrix(rows)
And then make a local DenseMatrix.
val localMat = Matrices.dense(m, n, mat)
Then multiply them.
rmat.multiply(localMat)
xj @ Tokyo
On Thu, Aug 21, 2014 at 6:37 PM, Sean Owen <so...@cloudera.com> wrote:
> Are you trying to multiply dense or sparse matrices? if sparse, are
> they very large -- meaning, are you looking for distributed
> operations?
>
> On Thu, Aug 21, 2014 at 10:07 AM, phoenix bai <mi...@gmail.com>
> wrote:
> > there is RowMatrix implemented in spark.
> > and I check for a while but failed to find any matrix operations (like
> > multiplication etc) are defined in the class yet.
> >
> > so, my question is, if I want to do matrix multiplication, (to do vector
> x
> > matrix multiplication to be precise), need to convert the vector/matrix
> to
> > the the matrix type defined in breeze package right?
> >
> > thanks
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Matrix multiplication in spark
Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.
We are working on a PRs to add block partitioned matrix formats and dense
matrix multiply methods. This should be out in the next few weeks or so.
The sparse methods still need some research on partitioning schemes etc.
and we will do that after the dense methods are in place.
Thanks
Shivaram
On Wed, Nov 5, 2014 at 2:00 AM, Duy Huynh <du...@gmail.com> wrote:
> ok great. when will this be ready?
>
> On Wed, Nov 5, 2014 at 4:27 AM, Xiangrui Meng <me...@gmail.com> wrote:
>
>> We are working on distributed block matrices. The main JIRA is at:
>>
>> https://issues.apache.org/jira/browse/SPARK-3434
>>
>> The goal is to support basic distributed linear algebra, (dense first
>> and then sparse).
>>
>> -Xiangrui
>>
>> On Wed, Nov 5, 2014 at 12:23 AM, ll <du...@gmail.com> wrote:
>> > @sowen.. i am looking for distributed operations, especially very large
>> > sparse matrix x sparse matrix multiplication. what is the best way to
>> > implement this in spark?
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Matrix-multiplication-in-spark-tp12562p18164.html
>> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>> >
>>
>
>
Re: Matrix multiplication in spark
Posted by Duy Huynh <du...@gmail.com>.
ok great. when will this be ready?
On Wed, Nov 5, 2014 at 4:27 AM, Xiangrui Meng <me...@gmail.com> wrote:
> We are working on distributed block matrices. The main JIRA is at:
>
> https://issues.apache.org/jira/browse/SPARK-3434
>
> The goal is to support basic distributed linear algebra, (dense first
> and then sparse).
>
> -Xiangrui
>
> On Wed, Nov 5, 2014 at 12:23 AM, ll <du...@gmail.com> wrote:
> > @sowen.. i am looking for distributed operations, especially very large
> > sparse matrix x sparse matrix multiplication. what is the best way to
> > implement this in spark?
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Matrix-multiplication-in-spark-tp12562p18164.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
Re: Matrix multiplication in spark
Posted by Xiangrui Meng <me...@gmail.com>.
We are working on distributed block matrices. The main JIRA is at:
https://issues.apache.org/jira/browse/SPARK-3434
The goal is to support basic distributed linear algebra, (dense first
and then sparse).
-Xiangrui
On Wed, Nov 5, 2014 at 12:23 AM, ll <du...@gmail.com> wrote:
> @sowen.. i am looking for distributed operations, especially very large
> sparse matrix x sparse matrix multiplication. what is the best way to
> implement this in spark?
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Matrix-multiplication-in-spark-tp12562p18164.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Matrix multiplication in spark
Posted by ll <du...@gmail.com>.
@sowen.. i am looking for distributed operations, especially very large
sparse matrix x sparse matrix multiplication. what is the best way to
implement this in spark?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Matrix-multiplication-in-spark-tp12562p18164.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Matrix multiplication in spark
Posted by Sean Owen <so...@cloudera.com>.
Are you trying to multiply dense or sparse matrices? if sparse, are
they very large -- meaning, are you looking for distributed
operations?
On Thu, Aug 21, 2014 at 10:07 AM, phoenix bai <mi...@gmail.com> wrote:
> there is RowMatrix implemented in spark.
> and I check for a while but failed to find any matrix operations (like
> multiplication etc) are defined in the class yet.
>
> so, my question is, if I want to do matrix multiplication, (to do vector x
> matrix multiplication to be precise), need to convert the vector/matrix to
> the the matrix type defined in breeze package right?
>
> thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org