Implementing linear albegra operations in the distributed linalg package

[José Manuel Abuín Mosquera <ab...@gmail.com>]

Hello,

I would like to add some linear algebra operations to all the 
DistributedMatrix classes that Spark actually handles (CoordinateMatrix, 
BlockMatrix, IndexedRowMatrix and RowMatrix), but first I would like do 
ask if you consider this useful. (For me, it is)

Of course, these operations will be distributed, but they will rely on 
the local implementation of mllib linalg. For example, when multiplying 
an IndexedRowMatrix by a DenseVector, the multiplication of one of the 
matrix rows by the vector will be performed by using the local 
implementation

What is your opinion about it?

Thank you

-- 
Jos� Manuel Abu�n Mosquera
Pre-doctoral researcher
Centro de Investigaci�n en Tecnolox�as da Informaci�n (CiTIUS)
University of Santiago de Compostela
15782 Santiago de Compostela, Spain

http://citius.usc.es/equipo/investigadores-en-formacion/josemanuel.abuin
http://jmabuin.github.io


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: Implementing linear albegra operations in the distributed linalg package

[Joseph Bradley <jo...@databricks.com>]

I agree that more distributed matrix ops would be good to have, but I think
there are a few things which need to happen first:
* Now that the spark.ml package has local linear algebra separate from the
spark.mllib package, we should migrate the distributed linear algebra
implementations over to spark.ml.
* This migration will require a bit of thinking about what the API should
look like.  Should it use Datasets?  If so, are there missing requirements
to fix within Datasets or local linear algebra?

I just created a JIRA; let's discuss more there:
https://issues.apache.org/jira/browse/SPARK-15882

Thanks for bringing this up!
Joseph

On Fri, Jun 3, 2016 at 4:02 AM, José Manuel Abuín Mosquera <
abuinjm@gmail.com> wrote:

> Hello,
>
> I would like to add some linear algebra operations to all the
> DistributedMatrix classes that Spark actually handles (CoordinateMatrix,
> BlockMatrix, IndexedRowMatrix and RowMatrix), but first I would like do ask
> if you consider this useful. (For me, it is)
>
> Of course, these operations will be distributed, but they will rely on the
> local implementation of mllib linalg. For example, when multiplying an
> IndexedRowMatrix by a DenseVector, the multiplication of one of the matrix
> rows by the vector will be performed by using the local implementation
>
> What is your opinion about it?
>
> Thank you
>
> --
> José Manuel Abuín Mosquera
> Pre-doctoral researcher
> Centro de Investigación en Tecnoloxías da Información (CiTIUS)
> University of Santiago de Compostela
> 15782 Santiago de Compostela, Spain
>
> http://citius.usc.es/equipo/investigadores-en-formacion/josemanuel.abuin
> http://jmabuin.github.io
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>