You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sis.apache.org by Martin Desruisseaux <ma...@geomatys.fr> on 2013/09/06 22:29:49 UTC
Needs a matrix library
Hello all
I have hit a point in Apache SIS where I need some matrix
implementations. Additions and multiplications are easy to implement,
but matrix inversions are more difficult. In Geotk I was using the
"vecmath" package (a derivative of legacy Sun Java3D library). However
vecmath is licensed under LGPL 2 [1], which in my understanding can not
work with Apache projects.
Does anyone would recommend a small library implementing Matrix objects
with basic operations? Note that I'm not looking for a full linear
algebra package. In particular, I would like a library with optimized
matrix implementation for the 3x3 and 4x4 cases - matrices of those
sizes will occur very often, and dedicated implementations make a real
difference both in performance and accuracy.
Martin
[1] https://java.net/projects/vecmath/sources/svn/content/trunk/LICENSE.txt
Re: Needs a matrix library
Posted by Adam Estrada <es...@gmail.com>.
I agree with that sentiment, Dr. Mattmann! It would be extremely cool to
see the distributed computation communities (eg. Spark and Hadoop) take
advantage of SIS. This is especially true for processing geospatial vector
data. Geospatial raster data is very splittable which makes it ideal for
this type of batch processing. Vector data is another beast all together
and I encourage folks in the aforementioned communities to think about how
to do this. I certainly have ideas and am all ears if someone would like to
chat about it!
Regards,
Adam
On Fri, Sep 6, 2013 at 9:41 PM, Mattmann, Chris A (398J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Thanks guys, just sharing a need here. SIS is a fully Java based
> geospatial library in development at Apache, aiming to support OGC
> standards. It would be great to figure out some synergy between Spark/Shark
> and SIS..
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Dmitriy Lyubimov <dl...@gmail.com>
> Reply-To: "dev@spark.incubator.apache.org" <dev@spark.incubator.apache.org
> >
> Date: Friday, September 6, 2013 5:25 PM
> To: "dev@spark.incubator.apache.org" <de...@spark.incubator.apache.org>
> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
> Subject: Re: Needs a matrix library
>
> >On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rx...@cs.berkeley.edu> wrote:
> >> They are asking about dedicated matrix libraries.
> >
> >Ah. I did not read the quoted email. Not sure why Chris was talking
> >about Pregel stuff, that doesn't seem what that question was about.
> >
> >>
> >> Neither GraphX nor Giraph are matrix libraries. These are systems that
> >> handle large scale graph processing, which could possibly be modeled as
> >> matrix computations. Hama looks like a BSP framework, so I am not sure
> >>if
> >> it has anything to do with matrix library either.
> >
> >+1
> >>
> >> For very small matrices (3x3, 4x4), the cost of going through jni to do
> >> native matrix operations will likely dominate the computation itself, so
> >> you are probably better off with a simple unrolled for loop in Java.
> >
> >+1 i guess this note is about JBlas and JBlas-based derivatives like
> >Breeze
> >
> >>
> >> I haven't looked into this myself, but I heard mahout-math is a decent
> >> library.
> >
> >+1 although for such tiny things like 3x3, 4x4 our cost-based
> >optimizations are probably not going to provide any noticeable bang.
> >Mahout in-core math is mostly for uniform cost-optimized support of
> >sparse vectors along with dense.
> >
> >Also, see if this makes sense, we are leaning towards commiting these
> >scala mappings in the current Mahout's trunk :[1]
> >
> >[1]
> >
> http://weatheringthrutechdays.blogspot.com/2013/07/scala-dsl-for-mahout-in
> >-core-linear.html
> >
> >-Dmitriy
> >
> >>
> >> --
> >> Reynold Xin, AMPLab, UC Berkeley
> >> http://rxin.org
> >>
> >>
> >>
> >> On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com>
> >>wrote:
> >>
> >>> keep forgetting this: what is graphx release roadmap?
> >>>
> >>> On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org>
> >>>wrote:
> >>> > Would it be more logical to use GraphX ?
> >>> > https://amplab.cs.berkeley.edu/publication/graphx-grades/
> >>> >
> >>> > Cos
> >>> >
> >>> > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
> >>> >> Thanks Roman, I was thinking Giraph too (knew it supported graphs
> >>>but
> >>> >> wasn't sure it supported matrices). If Giraph supports matrices,
> >>>big +1.
> >>> >>
> >>> >> Cheers,
> >>> >> Chris
> >>> >>
> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> >> Chris Mattmann, Ph.D.
> >>> >> Senior Computer Scientist
> >>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>> >> Office: 171-266B, Mailstop: 171-246
> >>> >> Email: chris.a.mattmann@nasa.gov
> >>> >> WWW: http://sunset.usc.edu/~mattmann/
> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> >> Adjunct Assistant Professor, Computer Science Department
> >>> >> University of Southern California, Los Angeles, CA 90089 USA
> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> -----Original Message-----
> >>> >> From: Roman Shaposhnik <rv...@apache.org>
> >>> >> Date: Friday, September 6, 2013 2:00 PM
> >>> >> To: <de...@spark.incubator.apache.org>
> >>> >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
> >>> >> Subject: Re: Needs a matrix library
> >>> >>
> >>> >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
> >>> >> ><ch...@jpl.nasa.gov> wrote:
> >>> >> >> Hey Martin,
> >>> >> >>
> >>> >> >> We may seriously consider using either Apache Hama here (which
> >>>will
> >>> >> >> bring in Hadoop):
> >>> >> >
> >>> >> >On that note I'd highly recommend taking a look at Apache Giraph
> >>> >> >as well: http://giraph.apache.org/
> >>> >> >
> >>> >> >Thanks,
> >>> >> >Roman.
> >>> >> >
> >>> >>
> >>>
>
>
Re: Needs a matrix library
Posted by Adam Estrada <es...@gmail.com>.
I agree with that sentiment, Dr. Mattmann! It would be extremely cool to
see the distributed computation communities (eg. Spark and Hadoop) take
advantage of SIS. This is especially true for processing geospatial vector
data. Geospatial raster data is very splittable which makes it ideal for
this type of batch processing. Vector data is another beast all together
and I encourage folks in the aforementioned communities to think about how
to do this. I certainly have ideas and am all ears if someone would like to
chat about it!
Regards,
Adam
On Fri, Sep 6, 2013 at 9:41 PM, Mattmann, Chris A (398J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Thanks guys, just sharing a need here. SIS is a fully Java based
> geospatial library in development at Apache, aiming to support OGC
> standards. It would be great to figure out some synergy between Spark/Shark
> and SIS..
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Dmitriy Lyubimov <dl...@gmail.com>
> Reply-To: "dev@spark.incubator.apache.org" <dev@spark.incubator.apache.org
> >
> Date: Friday, September 6, 2013 5:25 PM
> To: "dev@spark.incubator.apache.org" <de...@spark.incubator.apache.org>
> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
> Subject: Re: Needs a matrix library
>
> >On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rx...@cs.berkeley.edu> wrote:
> >> They are asking about dedicated matrix libraries.
> >
> >Ah. I did not read the quoted email. Not sure why Chris was talking
> >about Pregel stuff, that doesn't seem what that question was about.
> >
> >>
> >> Neither GraphX nor Giraph are matrix libraries. These are systems that
> >> handle large scale graph processing, which could possibly be modeled as
> >> matrix computations. Hama looks like a BSP framework, so I am not sure
> >>if
> >> it has anything to do with matrix library either.
> >
> >+1
> >>
> >> For very small matrices (3x3, 4x4), the cost of going through jni to do
> >> native matrix operations will likely dominate the computation itself, so
> >> you are probably better off with a simple unrolled for loop in Java.
> >
> >+1 i guess this note is about JBlas and JBlas-based derivatives like
> >Breeze
> >
> >>
> >> I haven't looked into this myself, but I heard mahout-math is a decent
> >> library.
> >
> >+1 although for such tiny things like 3x3, 4x4 our cost-based
> >optimizations are probably not going to provide any noticeable bang.
> >Mahout in-core math is mostly for uniform cost-optimized support of
> >sparse vectors along with dense.
> >
> >Also, see if this makes sense, we are leaning towards commiting these
> >scala mappings in the current Mahout's trunk :[1]
> >
> >[1]
> >
> http://weatheringthrutechdays.blogspot.com/2013/07/scala-dsl-for-mahout-in
> >-core-linear.html
> >
> >-Dmitriy
> >
> >>
> >> --
> >> Reynold Xin, AMPLab, UC Berkeley
> >> http://rxin.org
> >>
> >>
> >>
> >> On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com>
> >>wrote:
> >>
> >>> keep forgetting this: what is graphx release roadmap?
> >>>
> >>> On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org>
> >>>wrote:
> >>> > Would it be more logical to use GraphX ?
> >>> > https://amplab.cs.berkeley.edu/publication/graphx-grades/
> >>> >
> >>> > Cos
> >>> >
> >>> > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
> >>> >> Thanks Roman, I was thinking Giraph too (knew it supported graphs
> >>>but
> >>> >> wasn't sure it supported matrices). If Giraph supports matrices,
> >>>big +1.
> >>> >>
> >>> >> Cheers,
> >>> >> Chris
> >>> >>
> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> >> Chris Mattmann, Ph.D.
> >>> >> Senior Computer Scientist
> >>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>> >> Office: 171-266B, Mailstop: 171-246
> >>> >> Email: chris.a.mattmann@nasa.gov
> >>> >> WWW: http://sunset.usc.edu/~mattmann/
> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> >> Adjunct Assistant Professor, Computer Science Department
> >>> >> University of Southern California, Los Angeles, CA 90089 USA
> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> -----Original Message-----
> >>> >> From: Roman Shaposhnik <rv...@apache.org>
> >>> >> Date: Friday, September 6, 2013 2:00 PM
> >>> >> To: <de...@spark.incubator.apache.org>
> >>> >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
> >>> >> Subject: Re: Needs a matrix library
> >>> >>
> >>> >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
> >>> >> ><ch...@jpl.nasa.gov> wrote:
> >>> >> >> Hey Martin,
> >>> >> >>
> >>> >> >> We may seriously consider using either Apache Hama here (which
> >>>will
> >>> >> >> bring in Hadoop):
> >>> >> >
> >>> >> >On that note I'd highly recommend taking a look at Apache Giraph
> >>> >> >as well: http://giraph.apache.org/
> >>> >> >
> >>> >> >Thanks,
> >>> >> >Roman.
> >>> >> >
> >>> >>
> >>>
>
>
Re: Needs a matrix library
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Thanks guys, just sharing a need here. SIS is a fully Java based
geospatial library in development at Apache, aiming to support OGC
standards. It would be great to figure out some synergy between Spark/Shark
and SIS..
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Dmitriy Lyubimov <dl...@gmail.com>
Reply-To: "dev@spark.incubator.apache.org" <de...@spark.incubator.apache.org>
Date: Friday, September 6, 2013 5:25 PM
To: "dev@spark.incubator.apache.org" <de...@spark.incubator.apache.org>
Cc: "dev@sis.apache.org" <de...@sis.apache.org>
Subject: Re: Needs a matrix library
>On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rx...@cs.berkeley.edu> wrote:
>> They are asking about dedicated matrix libraries.
>
>Ah. I did not read the quoted email. Not sure why Chris was talking
>about Pregel stuff, that doesn't seem what that question was about.
>
>>
>> Neither GraphX nor Giraph are matrix libraries. These are systems that
>> handle large scale graph processing, which could possibly be modeled as
>> matrix computations. Hama looks like a BSP framework, so I am not sure
>>if
>> it has anything to do with matrix library either.
>
>+1
>>
>> For very small matrices (3x3, 4x4), the cost of going through jni to do
>> native matrix operations will likely dominate the computation itself, so
>> you are probably better off with a simple unrolled for loop in Java.
>
>+1 i guess this note is about JBlas and JBlas-based derivatives like
>Breeze
>
>>
>> I haven't looked into this myself, but I heard mahout-math is a decent
>> library.
>
>+1 although for such tiny things like 3x3, 4x4 our cost-based
>optimizations are probably not going to provide any noticeable bang.
>Mahout in-core math is mostly for uniform cost-optimized support of
>sparse vectors along with dense.
>
>Also, see if this makes sense, we are leaning towards commiting these
>scala mappings in the current Mahout's trunk :[1]
>
>[1]
>http://weatheringthrutechdays.blogspot.com/2013/07/scala-dsl-for-mahout-in
>-core-linear.html
>
>-Dmitriy
>
>>
>> --
>> Reynold Xin, AMPLab, UC Berkeley
>> http://rxin.org
>>
>>
>>
>> On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com>
>>wrote:
>>
>>> keep forgetting this: what is graphx release roadmap?
>>>
>>> On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org>
>>>wrote:
>>> > Would it be more logical to use GraphX ?
>>> > https://amplab.cs.berkeley.edu/publication/graphx-grades/
>>> >
>>> > Cos
>>> >
>>> > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
>>> >> Thanks Roman, I was thinking Giraph too (knew it supported graphs
>>>but
>>> >> wasn't sure it supported matrices). If Giraph supports matrices,
>>>big +1.
>>> >>
>>> >> Cheers,
>>> >> Chris
>>> >>
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Chris Mattmann, Ph.D.
>>> >> Senior Computer Scientist
>>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> >> Office: 171-266B, Mailstop: 171-246
>>> >> Email: chris.a.mattmann@nasa.gov
>>> >> WWW: http://sunset.usc.edu/~mattmann/
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Adjunct Assistant Professor, Computer Science Department
>>> >> University of Southern California, Los Angeles, CA 90089 USA
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> -----Original Message-----
>>> >> From: Roman Shaposhnik <rv...@apache.org>
>>> >> Date: Friday, September 6, 2013 2:00 PM
>>> >> To: <de...@spark.incubator.apache.org>
>>> >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
>>> >> Subject: Re: Needs a matrix library
>>> >>
>>> >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
>>> >> ><ch...@jpl.nasa.gov> wrote:
>>> >> >> Hey Martin,
>>> >> >>
>>> >> >> We may seriously consider using either Apache Hama here (which
>>>will
>>> >> >> bring in Hadoop):
>>> >> >
>>> >> >On that note I'd highly recommend taking a look at Apache Giraph
>>> >> >as well: http://giraph.apache.org/
>>> >> >
>>> >> >Thanks,
>>> >> >Roman.
>>> >> >
>>> >>
>>>
Re: Needs a matrix library
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Thanks guys, just sharing a need here. SIS is a fully Java based
geospatial library in development at Apache, aiming to support OGC
standards. It would be great to figure out some synergy between Spark/Shark
and SIS..
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Dmitriy Lyubimov <dl...@gmail.com>
Reply-To: "dev@spark.incubator.apache.org" <de...@spark.incubator.apache.org>
Date: Friday, September 6, 2013 5:25 PM
To: "dev@spark.incubator.apache.org" <de...@spark.incubator.apache.org>
Cc: "dev@sis.apache.org" <de...@sis.apache.org>
Subject: Re: Needs a matrix library
>On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rx...@cs.berkeley.edu> wrote:
>> They are asking about dedicated matrix libraries.
>
>Ah. I did not read the quoted email. Not sure why Chris was talking
>about Pregel stuff, that doesn't seem what that question was about.
>
>>
>> Neither GraphX nor Giraph are matrix libraries. These are systems that
>> handle large scale graph processing, which could possibly be modeled as
>> matrix computations. Hama looks like a BSP framework, so I am not sure
>>if
>> it has anything to do with matrix library either.
>
>+1
>>
>> For very small matrices (3x3, 4x4), the cost of going through jni to do
>> native matrix operations will likely dominate the computation itself, so
>> you are probably better off with a simple unrolled for loop in Java.
>
>+1 i guess this note is about JBlas and JBlas-based derivatives like
>Breeze
>
>>
>> I haven't looked into this myself, but I heard mahout-math is a decent
>> library.
>
>+1 although for such tiny things like 3x3, 4x4 our cost-based
>optimizations are probably not going to provide any noticeable bang.
>Mahout in-core math is mostly for uniform cost-optimized support of
>sparse vectors along with dense.
>
>Also, see if this makes sense, we are leaning towards commiting these
>scala mappings in the current Mahout's trunk :[1]
>
>[1]
>http://weatheringthrutechdays.blogspot.com/2013/07/scala-dsl-for-mahout-in
>-core-linear.html
>
>-Dmitriy
>
>>
>> --
>> Reynold Xin, AMPLab, UC Berkeley
>> http://rxin.org
>>
>>
>>
>> On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com>
>>wrote:
>>
>>> keep forgetting this: what is graphx release roadmap?
>>>
>>> On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org>
>>>wrote:
>>> > Would it be more logical to use GraphX ?
>>> > https://amplab.cs.berkeley.edu/publication/graphx-grades/
>>> >
>>> > Cos
>>> >
>>> > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
>>> >> Thanks Roman, I was thinking Giraph too (knew it supported graphs
>>>but
>>> >> wasn't sure it supported matrices). If Giraph supports matrices,
>>>big +1.
>>> >>
>>> >> Cheers,
>>> >> Chris
>>> >>
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Chris Mattmann, Ph.D.
>>> >> Senior Computer Scientist
>>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> >> Office: 171-266B, Mailstop: 171-246
>>> >> Email: chris.a.mattmann@nasa.gov
>>> >> WWW: http://sunset.usc.edu/~mattmann/
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Adjunct Assistant Professor, Computer Science Department
>>> >> University of Southern California, Los Angeles, CA 90089 USA
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> -----Original Message-----
>>> >> From: Roman Shaposhnik <rv...@apache.org>
>>> >> Date: Friday, September 6, 2013 2:00 PM
>>> >> To: <de...@spark.incubator.apache.org>
>>> >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
>>> >> Subject: Re: Needs a matrix library
>>> >>
>>> >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
>>> >> ><ch...@jpl.nasa.gov> wrote:
>>> >> >> Hey Martin,
>>> >> >>
>>> >> >> We may seriously consider using either Apache Hama here (which
>>>will
>>> >> >> bring in Hadoop):
>>> >> >
>>> >> >On that note I'd highly recommend taking a look at Apache Giraph
>>> >> >as well: http://giraph.apache.org/
>>> >> >
>>> >> >Thanks,
>>> >> >Roman.
>>> >> >
>>> >>
>>>
Re: Needs a matrix library
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rx...@cs.berkeley.edu> wrote:
> They are asking about dedicated matrix libraries.
Ah. I did not read the quoted email. Not sure why Chris was talking
about Pregel stuff, that doesn't seem what that question was about.
>
> Neither GraphX nor Giraph are matrix libraries. These are systems that
> handle large scale graph processing, which could possibly be modeled as
> matrix computations. Hama looks like a BSP framework, so I am not sure if
> it has anything to do with matrix library either.
+1
>
> For very small matrices (3x3, 4x4), the cost of going through jni to do
> native matrix operations will likely dominate the computation itself, so
> you are probably better off with a simple unrolled for loop in Java.
+1 i guess this note is about JBlas and JBlas-based derivatives like Breeze
>
> I haven't looked into this myself, but I heard mahout-math is a decent
> library.
+1 although for such tiny things like 3x3, 4x4 our cost-based
optimizations are probably not going to provide any noticeable bang.
Mahout in-core math is mostly for uniform cost-optimized support of
sparse vectors along with dense.
Also, see if this makes sense, we are leaning towards commiting these
scala mappings in the current Mahout's trunk :[1]
[1] http://weatheringthrutechdays.blogspot.com/2013/07/scala-dsl-for-mahout-in-core-linear.html
-Dmitriy
>
> --
> Reynold Xin, AMPLab, UC Berkeley
> http://rxin.org
>
>
>
> On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>
>> keep forgetting this: what is graphx release roadmap?
>>
>> On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org> wrote:
>> > Would it be more logical to use GraphX ?
>> > https://amplab.cs.berkeley.edu/publication/graphx-grades/
>> >
>> > Cos
>> >
>> > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
>> >> Thanks Roman, I was thinking Giraph too (knew it supported graphs but
>> >> wasn't sure it supported matrices). If Giraph supports matrices, big +1.
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> Chris Mattmann, Ph.D.
>> >> Senior Computer Scientist
>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >> Office: 171-266B, Mailstop: 171-246
>> >> Email: chris.a.mattmann@nasa.gov
>> >> WWW: http://sunset.usc.edu/~mattmann/
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> Adjunct Assistant Professor, Computer Science Department
>> >> University of Southern California, Los Angeles, CA 90089 USA
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Roman Shaposhnik <rv...@apache.org>
>> >> Date: Friday, September 6, 2013 2:00 PM
>> >> To: <de...@spark.incubator.apache.org>
>> >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
>> >> Subject: Re: Needs a matrix library
>> >>
>> >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
>> >> ><ch...@jpl.nasa.gov> wrote:
>> >> >> Hey Martin,
>> >> >>
>> >> >> We may seriously consider using either Apache Hama here (which will
>> >> >> bring in Hadoop):
>> >> >
>> >> >On that note I'd highly recommend taking a look at Apache Giraph
>> >> >as well: http://giraph.apache.org/
>> >> >
>> >> >Thanks,
>> >> >Roman.
>> >> >
>> >>
>>
Re: Needs a matrix library
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
BSD is compatible with ALv2 per:
http://www.apache.org/legal/3party.html#category-a
It's a Category A.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Adam Estrada <es...@gmail.com>
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Friday, September 6, 2013 8:41 PM
To: "dev@sis.apache.org" <de...@sis.apache.org>,
"shivaram@eecs.berkeley.edu" <sh...@eecs.berkeley.edu>
Cc: "dev@spark.incubator.apache.org" <de...@spark.incubator.apache.org>
Subject: Re: Needs a matrix library
>+1 to jblas. It has a BSD license though so it might not be compatible
>with
>the Apache v2 license. Anyone else want to weigh in on that?
>
>Adam
>
>
>On Fri, Sep 6, 2013 at 8:26 PM, Shivaram Venkataraman <
>shivaram@eecs.berkeley.edu> wrote:
>
>> For the machine learning library that is a part of Spark 0.8 we have
>>been
>> using jblas for local matrix operations. From some limited benchmarking
>> that we did, jblas is not much slower than optimized C++ libraries.
>>
>>http://blog.mikiobraun.de/2009/04/some-benchmark-numbers-for-jblas.htmlha
>>s
>> some more details.
>>
>> For more complex operations than addition and multiplication,
>>mahout-math
>> is a pretty good library. There was a great discussion on pros/cons of
>> different Java/Scala-based matrix libraries in
>> https://github.com/mesos/spark/pull/736
>>
>> Thanks
>> Shivaram
>>
>>
>> On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rx...@cs.berkeley.edu>
>>wrote:
>>
>> > They are asking about dedicated matrix libraries.
>> >
>> > Neither GraphX nor Giraph are matrix libraries. These are systems that
>> > handle large scale graph processing, which could possibly be modeled
>>as
>> > matrix computations. Hama looks like a BSP framework, so I am not
>>sure
>> if
>> > it has anything to do with matrix library either.
>> >
>> > For very small matrices (3x3, 4x4), the cost of going through jni to
>>do
>> > native matrix operations will likely dominate the computation itself,
>>so
>> > you are probably better off with a simple unrolled for loop in Java.
>> >
>> > I haven't looked into this myself, but I heard mahout-math is a decent
>> > library.
>> >
>> > --
>> > Reynold Xin, AMPLab, UC Berkeley
>> > http://rxin.org
>> >
>> >
>> >
>> > On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com>
>> > wrote:
>> >
>> > > keep forgetting this: what is graphx release roadmap?
>> > >
>> > > On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org>
>> > wrote:
>> > > > Would it be more logical to use GraphX ?
>> > > > https://amplab.cs.berkeley.edu/publication/graphx-grades/
>> > > >
>> > > > Cos
>> > > >
>> > > > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
>> > > >> Thanks Roman, I was thinking Giraph too (knew it supported graphs
>> but
>> > > >> wasn't sure it supported matrices). If Giraph supports matrices,
>>big
>> > +1.
>> > > >>
>> > > >> Cheers,
>> > > >> Chris
>> > > >>
>> > > >>
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > >> Chris Mattmann, Ph.D.
>> > > >> Senior Computer Scientist
>> > > >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > > >> Office: 171-266B, Mailstop: 171-246
>> > > >> Email: chris.a.mattmann@nasa.gov
>> > > >> WWW: http://sunset.usc.edu/~mattmann/
>> > > >>
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > >> Adjunct Assistant Professor, Computer Science Department
>> > > >> University of Southern California, Los Angeles, CA 90089 USA
>> > > >>
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >> -----Original Message-----
>> > > >> From: Roman Shaposhnik <rv...@apache.org>
>> > > >> Date: Friday, September 6, 2013 2:00 PM
>> > > >> To: <de...@spark.incubator.apache.org>
>> > > >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
>> > > >> Subject: Re: Needs a matrix library
>> > > >>
>> > > >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
>> > > >> ><ch...@jpl.nasa.gov> wrote:
>> > > >> >> Hey Martin,
>> > > >> >>
>> > > >> >> We may seriously consider using either Apache Hama here (which
>> will
>> > > >> >> bring in Hadoop):
>> > > >> >
>> > > >> >On that note I'd highly recommend taking a look at Apache Giraph
>> > > >> >as well: http://giraph.apache.org/
>> > > >> >
>> > > >> >Thanks,
>> > > >> >Roman.
>> > > >> >
>> > > >>
>> > >
>> >
>>
Re: Needs a matrix library
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
BSD is compatible with ALv2 per:
http://www.apache.org/legal/3party.html#category-a
It's a Category A.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Adam Estrada <es...@gmail.com>
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Friday, September 6, 2013 8:41 PM
To: "dev@sis.apache.org" <de...@sis.apache.org>,
"shivaram@eecs.berkeley.edu" <sh...@eecs.berkeley.edu>
Cc: "dev@spark.incubator.apache.org" <de...@spark.incubator.apache.org>
Subject: Re: Needs a matrix library
>+1 to jblas. It has a BSD license though so it might not be compatible
>with
>the Apache v2 license. Anyone else want to weigh in on that?
>
>Adam
>
>
>On Fri, Sep 6, 2013 at 8:26 PM, Shivaram Venkataraman <
>shivaram@eecs.berkeley.edu> wrote:
>
>> For the machine learning library that is a part of Spark 0.8 we have
>>been
>> using jblas for local matrix operations. From some limited benchmarking
>> that we did, jblas is not much slower than optimized C++ libraries.
>>
>>http://blog.mikiobraun.de/2009/04/some-benchmark-numbers-for-jblas.htmlha
>>s
>> some more details.
>>
>> For more complex operations than addition and multiplication,
>>mahout-math
>> is a pretty good library. There was a great discussion on pros/cons of
>> different Java/Scala-based matrix libraries in
>> https://github.com/mesos/spark/pull/736
>>
>> Thanks
>> Shivaram
>>
>>
>> On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rx...@cs.berkeley.edu>
>>wrote:
>>
>> > They are asking about dedicated matrix libraries.
>> >
>> > Neither GraphX nor Giraph are matrix libraries. These are systems that
>> > handle large scale graph processing, which could possibly be modeled
>>as
>> > matrix computations. Hama looks like a BSP framework, so I am not
>>sure
>> if
>> > it has anything to do with matrix library either.
>> >
>> > For very small matrices (3x3, 4x4), the cost of going through jni to
>>do
>> > native matrix operations will likely dominate the computation itself,
>>so
>> > you are probably better off with a simple unrolled for loop in Java.
>> >
>> > I haven't looked into this myself, but I heard mahout-math is a decent
>> > library.
>> >
>> > --
>> > Reynold Xin, AMPLab, UC Berkeley
>> > http://rxin.org
>> >
>> >
>> >
>> > On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com>
>> > wrote:
>> >
>> > > keep forgetting this: what is graphx release roadmap?
>> > >
>> > > On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org>
>> > wrote:
>> > > > Would it be more logical to use GraphX ?
>> > > > https://amplab.cs.berkeley.edu/publication/graphx-grades/
>> > > >
>> > > > Cos
>> > > >
>> > > > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
>> > > >> Thanks Roman, I was thinking Giraph too (knew it supported graphs
>> but
>> > > >> wasn't sure it supported matrices). If Giraph supports matrices,
>>big
>> > +1.
>> > > >>
>> > > >> Cheers,
>> > > >> Chris
>> > > >>
>> > > >>
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > >> Chris Mattmann, Ph.D.
>> > > >> Senior Computer Scientist
>> > > >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > > >> Office: 171-266B, Mailstop: 171-246
>> > > >> Email: chris.a.mattmann@nasa.gov
>> > > >> WWW: http://sunset.usc.edu/~mattmann/
>> > > >>
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > >> Adjunct Assistant Professor, Computer Science Department
>> > > >> University of Southern California, Los Angeles, CA 90089 USA
>> > > >>
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >> -----Original Message-----
>> > > >> From: Roman Shaposhnik <rv...@apache.org>
>> > > >> Date: Friday, September 6, 2013 2:00 PM
>> > > >> To: <de...@spark.incubator.apache.org>
>> > > >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
>> > > >> Subject: Re: Needs a matrix library
>> > > >>
>> > > >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
>> > > >> ><ch...@jpl.nasa.gov> wrote:
>> > > >> >> Hey Martin,
>> > > >> >>
>> > > >> >> We may seriously consider using either Apache Hama here (which
>> will
>> > > >> >> bring in Hadoop):
>> > > >> >
>> > > >> >On that note I'd highly recommend taking a look at Apache Giraph
>> > > >> >as well: http://giraph.apache.org/
>> > > >> >
>> > > >> >Thanks,
>> > > >> >Roman.
>> > > >> >
>> > > >>
>> > >
>> >
>>
Re: Needs a matrix library
Posted by Adam Estrada <es...@gmail.com>.
+1 to jblas. It has a BSD license though so it might not be compatible with
the Apache v2 license. Anyone else want to weigh in on that?
Adam
On Fri, Sep 6, 2013 at 8:26 PM, Shivaram Venkataraman <
shivaram@eecs.berkeley.edu> wrote:
> For the machine learning library that is a part of Spark 0.8 we have been
> using jblas for local matrix operations. From some limited benchmarking
> that we did, jblas is not much slower than optimized C++ libraries.
> http://blog.mikiobraun.de/2009/04/some-benchmark-numbers-for-jblas.htmlhas
> some more details.
>
> For more complex operations than addition and multiplication, mahout-math
> is a pretty good library. There was a great discussion on pros/cons of
> different Java/Scala-based matrix libraries in
> https://github.com/mesos/spark/pull/736
>
> Thanks
> Shivaram
>
>
> On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rx...@cs.berkeley.edu> wrote:
>
> > They are asking about dedicated matrix libraries.
> >
> > Neither GraphX nor Giraph are matrix libraries. These are systems that
> > handle large scale graph processing, which could possibly be modeled as
> > matrix computations. Hama looks like a BSP framework, so I am not sure
> if
> > it has anything to do with matrix library either.
> >
> > For very small matrices (3x3, 4x4), the cost of going through jni to do
> > native matrix operations will likely dominate the computation itself, so
> > you are probably better off with a simple unrolled for loop in Java.
> >
> > I haven't looked into this myself, but I heard mahout-math is a decent
> > library.
> >
> > --
> > Reynold Xin, AMPLab, UC Berkeley
> > http://rxin.org
> >
> >
> >
> > On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com>
> > wrote:
> >
> > > keep forgetting this: what is graphx release roadmap?
> > >
> > > On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > > > Would it be more logical to use GraphX ?
> > > > https://amplab.cs.berkeley.edu/publication/graphx-grades/
> > > >
> > > > Cos
> > > >
> > > > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
> > > >> Thanks Roman, I was thinking Giraph too (knew it supported graphs
> but
> > > >> wasn't sure it supported matrices). If Giraph supports matrices, big
> > +1.
> > > >>
> > > >> Cheers,
> > > >> Chris
> > > >>
> > > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >> Chris Mattmann, Ph.D.
> > > >> Senior Computer Scientist
> > > >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > >> Office: 171-266B, Mailstop: 171-246
> > > >> Email: chris.a.mattmann@nasa.gov
> > > >> WWW: http://sunset.usc.edu/~mattmann/
> > > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >> Adjunct Assistant Professor, Computer Science Department
> > > >> University of Southern California, Los Angeles, CA 90089 USA
> > > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> -----Original Message-----
> > > >> From: Roman Shaposhnik <rv...@apache.org>
> > > >> Date: Friday, September 6, 2013 2:00 PM
> > > >> To: <de...@spark.incubator.apache.org>
> > > >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
> > > >> Subject: Re: Needs a matrix library
> > > >>
> > > >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
> > > >> ><ch...@jpl.nasa.gov> wrote:
> > > >> >> Hey Martin,
> > > >> >>
> > > >> >> We may seriously consider using either Apache Hama here (which
> will
> > > >> >> bring in Hadoop):
> > > >> >
> > > >> >On that note I'd highly recommend taking a look at Apache Giraph
> > > >> >as well: http://giraph.apache.org/
> > > >> >
> > > >> >Thanks,
> > > >> >Roman.
> > > >> >
> > > >>
> > >
> >
>
Re: Needs a matrix library
Posted by Martin Desruisseaux <ma...@geomatys.fr>.
Le 09/09/13 02:32, Adam Estrada a écrit :
> Yep. JAMA looks good to me as well. I am not all that familiar with it and
> think that JBLAS would be good too but that seems like it would add quite a
> few unneeded dependencies.
I think that JBLAS could play a role a little bit later. For example
when drawing a contour map from a set of stations at random locations
(not on a grid), one method - among others - involve inverting a matrix
having as many rows and columns than stations. For such large matrix,
the industrially-proven LAPACK algorithms against numerical
instabilities would be very appreciated.
But in the particular case of the sis-referencing module, the matrix
size is the number of spatio-temporal dimensions + 1. For this reason I
expect matrix bigger than 5x5 to be quite unusual (while possible - for
example meteorologists use 2 time axes). For such small matrix, I think
that JAMA would be perfect. From my reading of javadoc, JBLAS uses pure
Java for O(1) operations like addition and multiplication by a vector,
but still uses native code for multiplication by an other matrix and
inversion (which we need).
Thanks all for your feedbacks!
Martin
Re: Needs a matrix library
Posted by Martin Desruisseaux <ma...@geomatys.fr>.
Le 09/09/13 02:32, Adam Estrada a écrit :
> Yep. JAMA looks good to me as well. I am not all that familiar with it and
> think that JBLAS would be good too but that seems like it would add quite a
> few unneeded dependencies.
I think that JBLAS could play a role a little bit later. For example
when drawing a contour map from a set of stations at random locations
(not on a grid), one method - among others - involve inverting a matrix
having as many rows and columns than stations. For such large matrix,
the industrially-proven LAPACK algorithms against numerical
instabilities would be very appreciated.
But in the particular case of the sis-referencing module, the matrix
size is the number of spatio-temporal dimensions + 1. For this reason I
expect matrix bigger than 5x5 to be quite unusual (while possible - for
example meteorologists use 2 time axes). For such small matrix, I think
that JAMA would be perfect. From my reading of javadoc, JBLAS uses pure
Java for O(1) operations like addition and multiplication by a vector,
but still uses native code for multiplication by an other matrix and
inversion (which we need).
Thanks all for your feedbacks!
Martin
Re: Needs a matrix library
Posted by Adam Estrada <es...@gmail.com>.
Yep. JAMA looks good to me as well. I am not all that familiar with it and
think that JBLAS would be good too but that seems like it would add quite a
few unneeded dependencies.
Adam
On Sun, Sep 8, 2013 at 6:38 PM, Chris Mattmann <ma...@apache.org> wrote:
> Hi Martin,
>
> OK, so looking at the license for JAMA:
>
> http://wordhoard.northwestern.edu/userman/thirdparty/jama.html
>
> and
> http://muuki88.github.io/jama-osgi/license.html
>
>
> Looks like the later is ALv2 licensed. So JAMA looks good to me,
> too.
>
> Cheers,
> Chris
>
> -----Original Message-----
>
> From: Martin Desruisseaux <ma...@geomatys.fr>
> Organization: Geomatys
> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
> Date: Sunday, September 8, 2013 1:23 PM
> To: "dev@sis.apache.org" <de...@sis.apache.org>
> Cc: "dev@spark.incubator.apache.org" <de...@spark.incubator.apache.org>
> Subject: Re: Needs a matrix library
>
> >Thanks all for the tips. So if I'm summarizing right:
> >
> > * Hama and Spark are designed for distributed computing. Given that
> > our need is for small matrices (usually no more than 5x5),
> > distributed computing would probably be too much. However I keep
> > Hama and Spark in mind for the SIS "Grid Coverage" (or Raster)
> > processing part, to come later.
> > * JBlas seems to be JNI wrappers around LAPACK and BLAS Fortran
> > libraries. For small matrix, the JNI cost may be larger than the
> > benefit. I will keep JBlas in mind for some computations that
> > require large matrix, but those computations are not expected to
> > occur in the "referencing" part of SIS.
> > * Other libraries under compatible license include Apache Commons Math
> > [1] and JAMA [2].
> >
> >
> >Given that Apache Commons is a large library (the JAR file is 1.6 Mb)
> >while JAMA is very small and focused on Matrix only (a 36.5 kb file), I
> >would be tempted to propose JAMA. Is there any though on that?
> >
> >
> > Martin
> >
> >
> >[1] http://commons.apache.org/proper/commons-math/
> >[2] http://math.nist.gov/javanumerics/jama/
> >
>
>
>
Re: Needs a matrix library
Posted by Adam Estrada <es...@gmail.com>.
Yep. JAMA looks good to me as well. I am not all that familiar with it and
think that JBLAS would be good too but that seems like it would add quite a
few unneeded dependencies.
Adam
On Sun, Sep 8, 2013 at 6:38 PM, Chris Mattmann <ma...@apache.org> wrote:
> Hi Martin,
>
> OK, so looking at the license for JAMA:
>
> http://wordhoard.northwestern.edu/userman/thirdparty/jama.html
>
> and
> http://muuki88.github.io/jama-osgi/license.html
>
>
> Looks like the later is ALv2 licensed. So JAMA looks good to me,
> too.
>
> Cheers,
> Chris
>
> -----Original Message-----
>
> From: Martin Desruisseaux <ma...@geomatys.fr>
> Organization: Geomatys
> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
> Date: Sunday, September 8, 2013 1:23 PM
> To: "dev@sis.apache.org" <de...@sis.apache.org>
> Cc: "dev@spark.incubator.apache.org" <de...@spark.incubator.apache.org>
> Subject: Re: Needs a matrix library
>
> >Thanks all for the tips. So if I'm summarizing right:
> >
> > * Hama and Spark are designed for distributed computing. Given that
> > our need is for small matrices (usually no more than 5x5),
> > distributed computing would probably be too much. However I keep
> > Hama and Spark in mind for the SIS "Grid Coverage" (or Raster)
> > processing part, to come later.
> > * JBlas seems to be JNI wrappers around LAPACK and BLAS Fortran
> > libraries. For small matrix, the JNI cost may be larger than the
> > benefit. I will keep JBlas in mind for some computations that
> > require large matrix, but those computations are not expected to
> > occur in the "referencing" part of SIS.
> > * Other libraries under compatible license include Apache Commons Math
> > [1] and JAMA [2].
> >
> >
> >Given that Apache Commons is a large library (the JAR file is 1.6 Mb)
> >while JAMA is very small and focused on Matrix only (a 36.5 kb file), I
> >would be tempted to propose JAMA. Is there any though on that?
> >
> >
> > Martin
> >
> >
> >[1] http://commons.apache.org/proper/commons-math/
> >[2] http://math.nist.gov/javanumerics/jama/
> >
>
>
>
Re: Needs a matrix library
Posted by Mike <sp...@good-with-numbers.com>.
Martin Desruisseaux wrote:
> * JBlas seems to be JNI wrappers around LAPACK and BLAS Fortran
> libraries.
According to http://mikiobraun.github.io/jblas/javadoc/ that's not true
for operations that are faster to perform in the JVM.
Re: Needs a matrix library
Posted by Mike <sp...@good-with-numbers.com>.
Martin Desruisseaux wrote:
> * JBlas seems to be JNI wrappers around LAPACK and BLAS Fortran
> libraries.
According to http://mikiobraun.github.io/jblas/javadoc/ that's not true
for operations that are faster to perform in the JVM.
Re: Needs a matrix library
Posted by Chris Mattmann <ma...@apache.org>.
Hi Martin,
OK, so looking at the license for JAMA:
http://wordhoard.northwestern.edu/userman/thirdparty/jama.html
and
http://muuki88.github.io/jama-osgi/license.html
Looks like the later is ALv2 licensed. So JAMA looks good to me,
too.
Cheers,
Chris
-----Original Message-----
From: Martin Desruisseaux <ma...@geomatys.fr>
Organization: Geomatys
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Sunday, September 8, 2013 1:23 PM
To: "dev@sis.apache.org" <de...@sis.apache.org>
Cc: "dev@spark.incubator.apache.org" <de...@spark.incubator.apache.org>
Subject: Re: Needs a matrix library
>Thanks all for the tips. So if I'm summarizing right:
>
> * Hama and Spark are designed for distributed computing. Given that
> our need is for small matrices (usually no more than 5x5),
> distributed computing would probably be too much. However I keep
> Hama and Spark in mind for the SIS "Grid Coverage" (or Raster)
> processing part, to come later.
> * JBlas seems to be JNI wrappers around LAPACK and BLAS Fortran
> libraries. For small matrix, the JNI cost may be larger than the
> benefit. I will keep JBlas in mind for some computations that
> require large matrix, but those computations are not expected to
> occur in the "referencing" part of SIS.
> * Other libraries under compatible license include Apache Commons Math
> [1] and JAMA [2].
>
>
>Given that Apache Commons is a large library (the JAR file is 1.6 Mb)
>while JAMA is very small and focused on Matrix only (a 36.5 kb file), I
>would be tempted to propose JAMA. Is there any though on that?
>
>
> Martin
>
>
>[1] http://commons.apache.org/proper/commons-math/
>[2] http://math.nist.gov/javanumerics/jama/
>
Re: Needs a matrix library
Posted by Chris Mattmann <ma...@apache.org>.
Hi Martin,
OK, so looking at the license for JAMA:
http://wordhoard.northwestern.edu/userman/thirdparty/jama.html
and
http://muuki88.github.io/jama-osgi/license.html
Looks like the later is ALv2 licensed. So JAMA looks good to me,
too.
Cheers,
Chris
-----Original Message-----
From: Martin Desruisseaux <ma...@geomatys.fr>
Organization: Geomatys
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Sunday, September 8, 2013 1:23 PM
To: "dev@sis.apache.org" <de...@sis.apache.org>
Cc: "dev@spark.incubator.apache.org" <de...@spark.incubator.apache.org>
Subject: Re: Needs a matrix library
>Thanks all for the tips. So if I'm summarizing right:
>
> * Hama and Spark are designed for distributed computing. Given that
> our need is for small matrices (usually no more than 5x5),
> distributed computing would probably be too much. However I keep
> Hama and Spark in mind for the SIS "Grid Coverage" (or Raster)
> processing part, to come later.
> * JBlas seems to be JNI wrappers around LAPACK and BLAS Fortran
> libraries. For small matrix, the JNI cost may be larger than the
> benefit. I will keep JBlas in mind for some computations that
> require large matrix, but those computations are not expected to
> occur in the "referencing" part of SIS.
> * Other libraries under compatible license include Apache Commons Math
> [1] and JAMA [2].
>
>
>Given that Apache Commons is a large library (the JAR file is 1.6 Mb)
>while JAMA is very small and focused on Matrix only (a 36.5 kb file), I
>would be tempted to propose JAMA. Is there any though on that?
>
>
> Martin
>
>
>[1] http://commons.apache.org/proper/commons-math/
>[2] http://math.nist.gov/javanumerics/jama/
>
Re: Needs a matrix library
Posted by Martin Desruisseaux <ma...@geomatys.fr>.
Thanks all for the tips. So if I'm summarizing right:
* Hama and Spark are designed for distributed computing. Given that
our need is for small matrices (usually no more than 5x5),
distributed computing would probably be too much. However I keep
Hama and Spark in mind for the SIS "Grid Coverage" (or Raster)
processing part, to come later.
* JBlas seems to be JNI wrappers around LAPACK and BLAS Fortran
libraries. For small matrix, the JNI cost may be larger than the
benefit. I will keep JBlas in mind for some computations that
require large matrix, but those computations are not expected to
occur in the "referencing" part of SIS.
* Other libraries under compatible license include Apache Commons Math
[1] and JAMA [2].
Given that Apache Commons is a large library (the JAR file is 1.6 Mb)
while JAMA is very small and focused on Matrix only (a 36.5 kb file), I
would be tempted to propose JAMA. Is there any though on that?
Martin
[1] http://commons.apache.org/proper/commons-math/
[2] http://math.nist.gov/javanumerics/jama/
Re: Needs a matrix library
Posted by Mike <sp...@good-with-numbers.com>.
Why do we have spark.util.Vector? Should it be replaced by jblas?
Re: Needs a matrix library
Posted by Adam Estrada <es...@gmail.com>.
+1 to jblas. It has a BSD license though so it might not be compatible with
the Apache v2 license. Anyone else want to weigh in on that?
Adam
On Fri, Sep 6, 2013 at 8:26 PM, Shivaram Venkataraman <
shivaram@eecs.berkeley.edu> wrote:
> For the machine learning library that is a part of Spark 0.8 we have been
> using jblas for local matrix operations. From some limited benchmarking
> that we did, jblas is not much slower than optimized C++ libraries.
> http://blog.mikiobraun.de/2009/04/some-benchmark-numbers-for-jblas.htmlhas
> some more details.
>
> For more complex operations than addition and multiplication, mahout-math
> is a pretty good library. There was a great discussion on pros/cons of
> different Java/Scala-based matrix libraries in
> https://github.com/mesos/spark/pull/736
>
> Thanks
> Shivaram
>
>
> On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rx...@cs.berkeley.edu> wrote:
>
> > They are asking about dedicated matrix libraries.
> >
> > Neither GraphX nor Giraph are matrix libraries. These are systems that
> > handle large scale graph processing, which could possibly be modeled as
> > matrix computations. Hama looks like a BSP framework, so I am not sure
> if
> > it has anything to do with matrix library either.
> >
> > For very small matrices (3x3, 4x4), the cost of going through jni to do
> > native matrix operations will likely dominate the computation itself, so
> > you are probably better off with a simple unrolled for loop in Java.
> >
> > I haven't looked into this myself, but I heard mahout-math is a decent
> > library.
> >
> > --
> > Reynold Xin, AMPLab, UC Berkeley
> > http://rxin.org
> >
> >
> >
> > On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com>
> > wrote:
> >
> > > keep forgetting this: what is graphx release roadmap?
> > >
> > > On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > > > Would it be more logical to use GraphX ?
> > > > https://amplab.cs.berkeley.edu/publication/graphx-grades/
> > > >
> > > > Cos
> > > >
> > > > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
> > > >> Thanks Roman, I was thinking Giraph too (knew it supported graphs
> but
> > > >> wasn't sure it supported matrices). If Giraph supports matrices, big
> > +1.
> > > >>
> > > >> Cheers,
> > > >> Chris
> > > >>
> > > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >> Chris Mattmann, Ph.D.
> > > >> Senior Computer Scientist
> > > >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > >> Office: 171-266B, Mailstop: 171-246
> > > >> Email: chris.a.mattmann@nasa.gov
> > > >> WWW: http://sunset.usc.edu/~mattmann/
> > > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >> Adjunct Assistant Professor, Computer Science Department
> > > >> University of Southern California, Los Angeles, CA 90089 USA
> > > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> -----Original Message-----
> > > >> From: Roman Shaposhnik <rv...@apache.org>
> > > >> Date: Friday, September 6, 2013 2:00 PM
> > > >> To: <de...@spark.incubator.apache.org>
> > > >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
> > > >> Subject: Re: Needs a matrix library
> > > >>
> > > >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
> > > >> ><ch...@jpl.nasa.gov> wrote:
> > > >> >> Hey Martin,
> > > >> >>
> > > >> >> We may seriously consider using either Apache Hama here (which
> will
> > > >> >> bring in Hadoop):
> > > >> >
> > > >> >On that note I'd highly recommend taking a look at Apache Giraph
> > > >> >as well: http://giraph.apache.org/
> > > >> >
> > > >> >Thanks,
> > > >> >Roman.
> > > >> >
> > > >>
> > >
> >
>
Re: Needs a matrix library
Posted by Martin Desruisseaux <ma...@geomatys.fr>.
Thanks all for the tips. So if I'm summarizing right:
* Hama and Spark are designed for distributed computing. Given that
our need is for small matrices (usually no more than 5x5),
distributed computing would probably be too much. However I keep
Hama and Spark in mind for the SIS "Grid Coverage" (or Raster)
processing part, to come later.
* JBlas seems to be JNI wrappers around LAPACK and BLAS Fortran
libraries. For small matrix, the JNI cost may be larger than the
benefit. I will keep JBlas in mind for some computations that
require large matrix, but those computations are not expected to
occur in the "referencing" part of SIS.
* Other libraries under compatible license include Apache Commons Math
[1] and JAMA [2].
Given that Apache Commons is a large library (the JAR file is 1.6 Mb)
while JAMA is very small and focused on Matrix only (a 36.5 kb file), I
would be tempted to propose JAMA. Is there any though on that?
Martin
[1] http://commons.apache.org/proper/commons-math/
[2] http://math.nist.gov/javanumerics/jama/
Re: Needs a matrix library
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Thank you Shivaram!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Shivaram Venkataraman <sh...@eecs.berkeley.edu>
Reply-To: "dev@spark.incubator.apache.org"
<de...@spark.incubator.apache.org>, "shivaram@eecs.berkeley.edu"
<sh...@eecs.berkeley.edu>
Date: Friday, September 6, 2013 5:26 PM
To: "dev@spark.incubator.apache.org" <de...@spark.incubator.apache.org>
Cc: "dev@sis.apache.org" <de...@sis.apache.org>
Subject: Re: Needs a matrix library
>For the machine learning library that is a part of Spark 0.8 we have been
>using jblas for local matrix operations. From some limited benchmarking
>that we did, jblas is not much slower than optimized C++ libraries.
>http://blog.mikiobraun.de/2009/04/some-benchmark-numbers-for-jblas.html
>has
>some more details.
>
>For more complex operations than addition and multiplication, mahout-math
>is a pretty good library. There was a great discussion on pros/cons of
>different Java/Scala-based matrix libraries in
>https://github.com/mesos/spark/pull/736
>
>Thanks
>Shivaram
>
>
>On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rx...@cs.berkeley.edu> wrote:
>
>> They are asking about dedicated matrix libraries.
>>
>> Neither GraphX nor Giraph are matrix libraries. These are systems that
>> handle large scale graph processing, which could possibly be modeled as
>> matrix computations. Hama looks like a BSP framework, so I am not sure
>>if
>> it has anything to do with matrix library either.
>>
>> For very small matrices (3x3, 4x4), the cost of going through jni to do
>> native matrix operations will likely dominate the computation itself, so
>> you are probably better off with a simple unrolled for loop in Java.
>>
>> I haven't looked into this myself, but I heard mahout-math is a decent
>> library.
>>
>> --
>> Reynold Xin, AMPLab, UC Berkeley
>> http://rxin.org
>>
>>
>>
>> On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com>
>> wrote:
>>
>> > keep forgetting this: what is graphx release roadmap?
>> >
>> > On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>> > > Would it be more logical to use GraphX ?
>> > > https://amplab.cs.berkeley.edu/publication/graphx-grades/
>> > >
>> > > Cos
>> > >
>> > > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
>> > >> Thanks Roman, I was thinking Giraph too (knew it supported graphs
>>but
>> > >> wasn't sure it supported matrices). If Giraph supports matrices,
>>big
>> +1.
>> > >>
>> > >> Cheers,
>> > >> Chris
>> > >>
>> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >> Chris Mattmann, Ph.D.
>> > >> Senior Computer Scientist
>> > >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > >> Office: 171-266B, Mailstop: 171-246
>> > >> Email: chris.a.mattmann@nasa.gov
>> > >> WWW: http://sunset.usc.edu/~mattmann/
>> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >> Adjunct Assistant Professor, Computer Science Department
>> > >> University of Southern California, Los Angeles, CA 90089 USA
>> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> -----Original Message-----
>> > >> From: Roman Shaposhnik <rv...@apache.org>
>> > >> Date: Friday, September 6, 2013 2:00 PM
>> > >> To: <de...@spark.incubator.apache.org>
>> > >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
>> > >> Subject: Re: Needs a matrix library
>> > >>
>> > >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
>> > >> ><ch...@jpl.nasa.gov> wrote:
>> > >> >> Hey Martin,
>> > >> >>
>> > >> >> We may seriously consider using either Apache Hama here (which
>>will
>> > >> >> bring in Hadoop):
>> > >> >
>> > >> >On that note I'd highly recommend taking a look at Apache Giraph
>> > >> >as well: http://giraph.apache.org/
>> > >> >
>> > >> >Thanks,
>> > >> >Roman.
>> > >> >
>> > >>
>> >
>>
Re: Needs a matrix library
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Thank you Shivaram!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Shivaram Venkataraman <sh...@eecs.berkeley.edu>
Reply-To: "dev@spark.incubator.apache.org"
<de...@spark.incubator.apache.org>, "shivaram@eecs.berkeley.edu"
<sh...@eecs.berkeley.edu>
Date: Friday, September 6, 2013 5:26 PM
To: "dev@spark.incubator.apache.org" <de...@spark.incubator.apache.org>
Cc: "dev@sis.apache.org" <de...@sis.apache.org>
Subject: Re: Needs a matrix library
>For the machine learning library that is a part of Spark 0.8 we have been
>using jblas for local matrix operations. From some limited benchmarking
>that we did, jblas is not much slower than optimized C++ libraries.
>http://blog.mikiobraun.de/2009/04/some-benchmark-numbers-for-jblas.html
>has
>some more details.
>
>For more complex operations than addition and multiplication, mahout-math
>is a pretty good library. There was a great discussion on pros/cons of
>different Java/Scala-based matrix libraries in
>https://github.com/mesos/spark/pull/736
>
>Thanks
>Shivaram
>
>
>On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rx...@cs.berkeley.edu> wrote:
>
>> They are asking about dedicated matrix libraries.
>>
>> Neither GraphX nor Giraph are matrix libraries. These are systems that
>> handle large scale graph processing, which could possibly be modeled as
>> matrix computations. Hama looks like a BSP framework, so I am not sure
>>if
>> it has anything to do with matrix library either.
>>
>> For very small matrices (3x3, 4x4), the cost of going through jni to do
>> native matrix operations will likely dominate the computation itself, so
>> you are probably better off with a simple unrolled for loop in Java.
>>
>> I haven't looked into this myself, but I heard mahout-math is a decent
>> library.
>>
>> --
>> Reynold Xin, AMPLab, UC Berkeley
>> http://rxin.org
>>
>>
>>
>> On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com>
>> wrote:
>>
>> > keep forgetting this: what is graphx release roadmap?
>> >
>> > On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>> > > Would it be more logical to use GraphX ?
>> > > https://amplab.cs.berkeley.edu/publication/graphx-grades/
>> > >
>> > > Cos
>> > >
>> > > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
>> > >> Thanks Roman, I was thinking Giraph too (knew it supported graphs
>>but
>> > >> wasn't sure it supported matrices). If Giraph supports matrices,
>>big
>> +1.
>> > >>
>> > >> Cheers,
>> > >> Chris
>> > >>
>> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >> Chris Mattmann, Ph.D.
>> > >> Senior Computer Scientist
>> > >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > >> Office: 171-266B, Mailstop: 171-246
>> > >> Email: chris.a.mattmann@nasa.gov
>> > >> WWW: http://sunset.usc.edu/~mattmann/
>> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >> Adjunct Assistant Professor, Computer Science Department
>> > >> University of Southern California, Los Angeles, CA 90089 USA
>> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> -----Original Message-----
>> > >> From: Roman Shaposhnik <rv...@apache.org>
>> > >> Date: Friday, September 6, 2013 2:00 PM
>> > >> To: <de...@spark.incubator.apache.org>
>> > >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
>> > >> Subject: Re: Needs a matrix library
>> > >>
>> > >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
>> > >> ><ch...@jpl.nasa.gov> wrote:
>> > >> >> Hey Martin,
>> > >> >>
>> > >> >> We may seriously consider using either Apache Hama here (which
>>will
>> > >> >> bring in Hadoop):
>> > >> >
>> > >> >On that note I'd highly recommend taking a look at Apache Giraph
>> > >> >as well: http://giraph.apache.org/
>> > >> >
>> > >> >Thanks,
>> > >> >Roman.
>> > >> >
>> > >>
>> >
>>
Re: Needs a matrix library
Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.
For the machine learning library that is a part of Spark 0.8 we have been
using jblas for local matrix operations. From some limited benchmarking
that we did, jblas is not much slower than optimized C++ libraries.
http://blog.mikiobraun.de/2009/04/some-benchmark-numbers-for-jblas.html has
some more details.
For more complex operations than addition and multiplication, mahout-math
is a pretty good library. There was a great discussion on pros/cons of
different Java/Scala-based matrix libraries in
https://github.com/mesos/spark/pull/736
Thanks
Shivaram
On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rx...@cs.berkeley.edu> wrote:
> They are asking about dedicated matrix libraries.
>
> Neither GraphX nor Giraph are matrix libraries. These are systems that
> handle large scale graph processing, which could possibly be modeled as
> matrix computations. Hama looks like a BSP framework, so I am not sure if
> it has anything to do with matrix library either.
>
> For very small matrices (3x3, 4x4), the cost of going through jni to do
> native matrix operations will likely dominate the computation itself, so
> you are probably better off with a simple unrolled for loop in Java.
>
> I haven't looked into this myself, but I heard mahout-math is a decent
> library.
>
> --
> Reynold Xin, AMPLab, UC Berkeley
> http://rxin.org
>
>
>
> On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
> > keep forgetting this: what is graphx release roadmap?
> >
> > On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> > > Would it be more logical to use GraphX ?
> > > https://amplab.cs.berkeley.edu/publication/graphx-grades/
> > >
> > > Cos
> > >
> > > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
> > >> Thanks Roman, I was thinking Giraph too (knew it supported graphs but
> > >> wasn't sure it supported matrices). If Giraph supports matrices, big
> +1.
> > >>
> > >> Cheers,
> > >> Chris
> > >>
> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >> Chris Mattmann, Ph.D.
> > >> Senior Computer Scientist
> > >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > >> Office: 171-266B, Mailstop: 171-246
> > >> Email: chris.a.mattmann@nasa.gov
> > >> WWW: http://sunset.usc.edu/~mattmann/
> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >> Adjunct Assistant Professor, Computer Science Department
> > >> University of Southern California, Los Angeles, CA 90089 USA
> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Roman Shaposhnik <rv...@apache.org>
> > >> Date: Friday, September 6, 2013 2:00 PM
> > >> To: <de...@spark.incubator.apache.org>
> > >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
> > >> Subject: Re: Needs a matrix library
> > >>
> > >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
> > >> ><ch...@jpl.nasa.gov> wrote:
> > >> >> Hey Martin,
> > >> >>
> > >> >> We may seriously consider using either Apache Hama here (which will
> > >> >> bring in Hadoop):
> > >> >
> > >> >On that note I'd highly recommend taking a look at Apache Giraph
> > >> >as well: http://giraph.apache.org/
> > >> >
> > >> >Thanks,
> > >> >Roman.
> > >> >
> > >>
> >
>
Re: Needs a matrix library
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rx...@cs.berkeley.edu> wrote:
> They are asking about dedicated matrix libraries.
Ah. I did not read the quoted email. Not sure why Chris was talking
about Pregel stuff, that doesn't seem what that question was about.
>
> Neither GraphX nor Giraph are matrix libraries. These are systems that
> handle large scale graph processing, which could possibly be modeled as
> matrix computations. Hama looks like a BSP framework, so I am not sure if
> it has anything to do with matrix library either.
+1
>
> For very small matrices (3x3, 4x4), the cost of going through jni to do
> native matrix operations will likely dominate the computation itself, so
> you are probably better off with a simple unrolled for loop in Java.
+1 i guess this note is about JBlas and JBlas-based derivatives like Breeze
>
> I haven't looked into this myself, but I heard mahout-math is a decent
> library.
+1 although for such tiny things like 3x3, 4x4 our cost-based
optimizations are probably not going to provide any noticeable bang.
Mahout in-core math is mostly for uniform cost-optimized support of
sparse vectors along with dense.
Also, see if this makes sense, we are leaning towards commiting these
scala mappings in the current Mahout's trunk :[1]
[1] http://weatheringthrutechdays.blogspot.com/2013/07/scala-dsl-for-mahout-in-core-linear.html
-Dmitriy
>
> --
> Reynold Xin, AMPLab, UC Berkeley
> http://rxin.org
>
>
>
> On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>
>> keep forgetting this: what is graphx release roadmap?
>>
>> On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org> wrote:
>> > Would it be more logical to use GraphX ?
>> > https://amplab.cs.berkeley.edu/publication/graphx-grades/
>> >
>> > Cos
>> >
>> > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
>> >> Thanks Roman, I was thinking Giraph too (knew it supported graphs but
>> >> wasn't sure it supported matrices). If Giraph supports matrices, big +1.
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> Chris Mattmann, Ph.D.
>> >> Senior Computer Scientist
>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >> Office: 171-266B, Mailstop: 171-246
>> >> Email: chris.a.mattmann@nasa.gov
>> >> WWW: http://sunset.usc.edu/~mattmann/
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> Adjunct Assistant Professor, Computer Science Department
>> >> University of Southern California, Los Angeles, CA 90089 USA
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Roman Shaposhnik <rv...@apache.org>
>> >> Date: Friday, September 6, 2013 2:00 PM
>> >> To: <de...@spark.incubator.apache.org>
>> >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
>> >> Subject: Re: Needs a matrix library
>> >>
>> >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
>> >> ><ch...@jpl.nasa.gov> wrote:
>> >> >> Hey Martin,
>> >> >>
>> >> >> We may seriously consider using either Apache Hama here (which will
>> >> >> bring in Hadoop):
>> >> >
>> >> >On that note I'd highly recommend taking a look at Apache Giraph
>> >> >as well: http://giraph.apache.org/
>> >> >
>> >> >Thanks,
>> >> >Roman.
>> >> >
>> >>
>>
Re: Needs a matrix library
Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.
For the machine learning library that is a part of Spark 0.8 we have been
using jblas for local matrix operations. From some limited benchmarking
that we did, jblas is not much slower than optimized C++ libraries.
http://blog.mikiobraun.de/2009/04/some-benchmark-numbers-for-jblas.html has
some more details.
For more complex operations than addition and multiplication, mahout-math
is a pretty good library. There was a great discussion on pros/cons of
different Java/Scala-based matrix libraries in
https://github.com/mesos/spark/pull/736
Thanks
Shivaram
On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rx...@cs.berkeley.edu> wrote:
> They are asking about dedicated matrix libraries.
>
> Neither GraphX nor Giraph are matrix libraries. These are systems that
> handle large scale graph processing, which could possibly be modeled as
> matrix computations. Hama looks like a BSP framework, so I am not sure if
> it has anything to do with matrix library either.
>
> For very small matrices (3x3, 4x4), the cost of going through jni to do
> native matrix operations will likely dominate the computation itself, so
> you are probably better off with a simple unrolled for loop in Java.
>
> I haven't looked into this myself, but I heard mahout-math is a decent
> library.
>
> --
> Reynold Xin, AMPLab, UC Berkeley
> http://rxin.org
>
>
>
> On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
> > keep forgetting this: what is graphx release roadmap?
> >
> > On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> > > Would it be more logical to use GraphX ?
> > > https://amplab.cs.berkeley.edu/publication/graphx-grades/
> > >
> > > Cos
> > >
> > > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
> > >> Thanks Roman, I was thinking Giraph too (knew it supported graphs but
> > >> wasn't sure it supported matrices). If Giraph supports matrices, big
> +1.
> > >>
> > >> Cheers,
> > >> Chris
> > >>
> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >> Chris Mattmann, Ph.D.
> > >> Senior Computer Scientist
> > >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > >> Office: 171-266B, Mailstop: 171-246
> > >> Email: chris.a.mattmann@nasa.gov
> > >> WWW: http://sunset.usc.edu/~mattmann/
> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >> Adjunct Assistant Professor, Computer Science Department
> > >> University of Southern California, Los Angeles, CA 90089 USA
> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Roman Shaposhnik <rv...@apache.org>
> > >> Date: Friday, September 6, 2013 2:00 PM
> > >> To: <de...@spark.incubator.apache.org>
> > >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
> > >> Subject: Re: Needs a matrix library
> > >>
> > >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
> > >> ><ch...@jpl.nasa.gov> wrote:
> > >> >> Hey Martin,
> > >> >>
> > >> >> We may seriously consider using either Apache Hama here (which will
> > >> >> bring in Hadoop):
> > >> >
> > >> >On that note I'd highly recommend taking a look at Apache Giraph
> > >> >as well: http://giraph.apache.org/
> > >> >
> > >> >Thanks,
> > >> >Roman.
> > >> >
> > >>
> >
>
Re: Needs a matrix library
Posted by Reynold Xin <rx...@cs.berkeley.edu>.
They are asking about dedicated matrix libraries.
Neither GraphX nor Giraph are matrix libraries. These are systems that
handle large scale graph processing, which could possibly be modeled as
matrix computations. Hama looks like a BSP framework, so I am not sure if
it has anything to do with matrix library either.
For very small matrices (3x3, 4x4), the cost of going through jni to do
native matrix operations will likely dominate the computation itself, so
you are probably better off with a simple unrolled for loop in Java.
I haven't looked into this myself, but I heard mahout-math is a decent
library.
--
Reynold Xin, AMPLab, UC Berkeley
http://rxin.org
On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> keep forgetting this: what is graphx release roadmap?
>
> On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org> wrote:
> > Would it be more logical to use GraphX ?
> > https://amplab.cs.berkeley.edu/publication/graphx-grades/
> >
> > Cos
> >
> > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
> >> Thanks Roman, I was thinking Giraph too (knew it supported graphs but
> >> wasn't sure it supported matrices). If Giraph supports matrices, big +1.
> >>
> >> Cheers,
> >> Chris
> >>
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Chris Mattmann, Ph.D.
> >> Senior Computer Scientist
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 171-266B, Mailstop: 171-246
> >> Email: chris.a.mattmann@nasa.gov
> >> WWW: http://sunset.usc.edu/~mattmann/
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Adjunct Assistant Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Roman Shaposhnik <rv...@apache.org>
> >> Date: Friday, September 6, 2013 2:00 PM
> >> To: <de...@spark.incubator.apache.org>
> >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
> >> Subject: Re: Needs a matrix library
> >>
> >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
> >> ><ch...@jpl.nasa.gov> wrote:
> >> >> Hey Martin,
> >> >>
> >> >> We may seriously consider using either Apache Hama here (which will
> >> >> bring in Hadoop):
> >> >
> >> >On that note I'd highly recommend taking a look at Apache Giraph
> >> >as well: http://giraph.apache.org/
> >> >
> >> >Thanks,
> >> >Roman.
> >> >
> >>
>
Re: Needs a matrix library
Posted by Reynold Xin <rx...@cs.berkeley.edu>.
They are asking about dedicated matrix libraries.
Neither GraphX nor Giraph are matrix libraries. These are systems that
handle large scale graph processing, which could possibly be modeled as
matrix computations. Hama looks like a BSP framework, so I am not sure if
it has anything to do with matrix library either.
For very small matrices (3x3, 4x4), the cost of going through jni to do
native matrix operations will likely dominate the computation itself, so
you are probably better off with a simple unrolled for loop in Java.
I haven't looked into this myself, but I heard mahout-math is a decent
library.
--
Reynold Xin, AMPLab, UC Berkeley
http://rxin.org
On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> keep forgetting this: what is graphx release roadmap?
>
> On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org> wrote:
> > Would it be more logical to use GraphX ?
> > https://amplab.cs.berkeley.edu/publication/graphx-grades/
> >
> > Cos
> >
> > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
> >> Thanks Roman, I was thinking Giraph too (knew it supported graphs but
> >> wasn't sure it supported matrices). If Giraph supports matrices, big +1.
> >>
> >> Cheers,
> >> Chris
> >>
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Chris Mattmann, Ph.D.
> >> Senior Computer Scientist
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 171-266B, Mailstop: 171-246
> >> Email: chris.a.mattmann@nasa.gov
> >> WWW: http://sunset.usc.edu/~mattmann/
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Adjunct Assistant Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Roman Shaposhnik <rv...@apache.org>
> >> Date: Friday, September 6, 2013 2:00 PM
> >> To: <de...@spark.incubator.apache.org>
> >> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
> >> Subject: Re: Needs a matrix library
> >>
> >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
> >> ><ch...@jpl.nasa.gov> wrote:
> >> >> Hey Martin,
> >> >>
> >> >> We may seriously consider using either Apache Hama here (which will
> >> >> bring in Hadoop):
> >> >
> >> >On that note I'd highly recommend taking a look at Apache Giraph
> >> >as well: http://giraph.apache.org/
> >> >
> >> >Thanks,
> >> >Roman.
> >> >
> >>
>
Re: Needs a matrix library
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
keep forgetting this: what is graphx release roadmap?
On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <co...@apache.org> wrote:
> Would it be more logical to use GraphX ?
> https://amplab.cs.berkeley.edu/publication/graphx-grades/
>
> Cos
>
> On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
>> Thanks Roman, I was thinking Giraph too (knew it supported graphs but
>> wasn't sure it supported matrices). If Giraph supports matrices, big +1.
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Roman Shaposhnik <rv...@apache.org>
>> Date: Friday, September 6, 2013 2:00 PM
>> To: <de...@spark.incubator.apache.org>
>> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
>> Subject: Re: Needs a matrix library
>>
>> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
>> ><ch...@jpl.nasa.gov> wrote:
>> >> Hey Martin,
>> >>
>> >> We may seriously consider using either Apache Hama here (which will
>> >> bring in Hadoop):
>> >
>> >On that note I'd highly recommend taking a look at Apache Giraph
>> >as well: http://giraph.apache.org/
>> >
>> >Thanks,
>> >Roman.
>> >
>>
Re: Needs a matrix library
Posted by Konstantin Boudnik <co...@apache.org>.
Would it be more logical to use GraphX ?
https://amplab.cs.berkeley.edu/publication/graphx-grades/
Cos
On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
> Thanks Roman, I was thinking Giraph too (knew it supported graphs but
> wasn't sure it supported matrices). If Giraph supports matrices, big +1.
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Roman Shaposhnik <rv...@apache.org>
> Date: Friday, September 6, 2013 2:00 PM
> To: <de...@spark.incubator.apache.org>
> Cc: "dev@sis.apache.org" <de...@sis.apache.org>
> Subject: Re: Needs a matrix library
>
> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
> ><ch...@jpl.nasa.gov> wrote:
> >> Hey Martin,
> >>
> >> We may seriously consider using either Apache Hama here (which will
> >> bring in Hadoop):
> >
> >On that note I'd highly recommend taking a look at Apache Giraph
> >as well: http://giraph.apache.org/
> >
> >Thanks,
> >Roman.
> >
>
Re: Needs a matrix library
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Thanks Roman, I was thinking Giraph too (knew it supported graphs but
wasn't sure it supported matrices). If Giraph supports matrices, big +1.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Roman Shaposhnik <rv...@apache.org>
Date: Friday, September 6, 2013 2:00 PM
To: <de...@spark.incubator.apache.org>
Cc: "dev@sis.apache.org" <de...@sis.apache.org>
Subject: Re: Needs a matrix library
>On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
><ch...@jpl.nasa.gov> wrote:
>> Hey Martin,
>>
>> We may seriously consider using either Apache Hama here (which will
>> bring in Hadoop):
>
>On that note I'd highly recommend taking a look at Apache Giraph
>as well: http://giraph.apache.org/
>
>Thanks,
>Roman.
>
Re: Needs a matrix library
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Thanks Roman, I was thinking Giraph too (knew it supported graphs but
wasn't sure it supported matrices). If Giraph supports matrices, big +1.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Roman Shaposhnik <rv...@apache.org>
Date: Friday, September 6, 2013 2:00 PM
To: <de...@spark.incubator.apache.org>
Cc: "dev@sis.apache.org" <de...@sis.apache.org>
Subject: Re: Needs a matrix library
>On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
><ch...@jpl.nasa.gov> wrote:
>> Hey Martin,
>>
>> We may seriously consider using either Apache Hama here (which will
>> bring in Hadoop):
>
>On that note I'd highly recommend taking a look at Apache Giraph
>as well: http://giraph.apache.org/
>
>Thanks,
>Roman.
>
Re: Needs a matrix library
Posted by Roman Shaposhnik <rv...@apache.org>.
On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
<ch...@jpl.nasa.gov> wrote:
> Hey Martin,
>
> We may seriously consider using either Apache Hama here (which will
> bring in Hadoop):
On that note I'd highly recommend taking a look at Apache Giraph
as well: http://giraph.apache.org/
Thanks,
Roman.
Re: Needs a matrix library
Posted by Roman Shaposhnik <rv...@apache.org>.
On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
<ch...@jpl.nasa.gov> wrote:
> Hey Martin,
>
> We may seriously consider using either Apache Hama here (which will
> bring in Hadoop):
On that note I'd highly recommend taking a look at Apache Giraph
as well: http://giraph.apache.org/
Thanks,
Roman.
Re: Needs a matrix library
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Sounds great, let's keep chatting. Spark + MLbase may also be a good
idea..
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Martin Desruisseaux <ma...@geomatys.fr>
Organization: Geomatys
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Friday, September 6, 2013 1:49 PM
To: "dev@sis.apache.org" <de...@sis.apache.org>
Subject: Re: Needs a matrix library
>Wow, that was a fast answer!
>
>Le 06/09/13 22:33, Mattmann, Chris A (398J) a écrit :
>> We may seriously consider using either Apache Hama here (which will
>> bring in Hadoop):
>>
>> http://hama.apache.org/
>
>Thanks for the link! By reading the description, I got the feeling that
>Hama is designed for very large matrices, to be distributed on a network
>of computers if possible. Maybe using Hadoop for inverting a single 4x4
>matrix would seems surprising to users :-). But I definitively keep Hama
>in mind for other tasks to come later (e.g. operations on Grid Coverages).
>
>Maybe we can continue to collect ideas and revisit in a few days?
>
> Martin
>
>
>> Or alternatively, think about some Apache Spark based library:
>>
>> http://spark.incubator.apache.org/
>>
>>
>>http://stackoverflow.com/questions/18453359/scala-spark-matrix-operations
>>
>>
>> It will refer to you to MLBase, which we could base it on.
>>
>> I'm CC'ing the Apache Spark list here to connect the dots.
>>
>> Cheers,
>> Chris
>
Re: Needs a matrix library
Posted by Martin Desruisseaux <ma...@geomatys.fr>.
Wow, that was a fast answer!
Le 06/09/13 22:33, Mattmann, Chris A (398J) a écrit :
> We may seriously consider using either Apache Hama here (which will
> bring in Hadoop):
>
> http://hama.apache.org/
Thanks for the link! By reading the description, I got the feeling that
Hama is designed for very large matrices, to be distributed on a network
of computers if possible. Maybe using Hadoop for inverting a single 4x4
matrix would seems surprising to users :-). But I definitively keep Hama
in mind for other tasks to come later (e.g. operations on Grid Coverages).
Maybe we can continue to collect ideas and revisit in a few days?
Martin
> Or alternatively, think about some Apache Spark based library:
>
> http://spark.incubator.apache.org/
>
> http://stackoverflow.com/questions/18453359/scala-spark-matrix-operations
>
>
> It will refer to you to MLBase, which we could base it on.
>
> I'm CC'ing the Apache Spark list here to connect the dots.
>
> Cheers,
> Chris
Re: Needs a matrix library
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Hey Martin,
We may seriously consider using either Apache Hama here (which will
bring in Hadoop):
http://hama.apache.org/
Or alternatively, think about some Apache Spark based library:
http://spark.incubator.apache.org/
http://stackoverflow.com/questions/18453359/scala-spark-matrix-operations
It will refer to you to MLBase, which we could base it on.
I'm CC'ing the Apache Spark list here to connect the dots.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Martin Desruisseaux <ma...@geomatys.fr>
Organization: Geomatys
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Friday, September 6, 2013 1:29 PM
To: Apache SIS <de...@sis.apache.org>
Subject: Needs a matrix library
>Hello all
>
>I have hit a point in Apache SIS where I need some matrix
>implementations. Additions and multiplications are easy to implement,
>but matrix inversions are more difficult. In Geotk I was using the
>"vecmath" package (a derivative of legacy Sun Java3D library). However
>vecmath is licensed under LGPL 2 [1], which in my understanding can not
>work with Apache projects.
>
>Does anyone would recommend a small library implementing Matrix objects
>with basic operations? Note that I'm not looking for a full linear
>algebra package. In particular, I would like a library with optimized
>matrix implementation for the 3x3 and 4x4 cases - matrices of those
>sizes will occur very often, and dedicated implementations make a real
>difference both in performance and accuracy.
>
> Martin
>
>[1]
>https://java.net/projects/vecmath/sources/svn/content/trunk/LICENSE.txt
>
Re: Needs a matrix library
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Hey Martin,
We may seriously consider using either Apache Hama here (which will
bring in Hadoop):
http://hama.apache.org/
Or alternatively, think about some Apache Spark based library:
http://spark.incubator.apache.org/
http://stackoverflow.com/questions/18453359/scala-spark-matrix-operations
It will refer to you to MLBase, which we could base it on.
I'm CC'ing the Apache Spark list here to connect the dots.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Martin Desruisseaux <ma...@geomatys.fr>
Organization: Geomatys
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Friday, September 6, 2013 1:29 PM
To: Apache SIS <de...@sis.apache.org>
Subject: Needs a matrix library
>Hello all
>
>I have hit a point in Apache SIS where I need some matrix
>implementations. Additions and multiplications are easy to implement,
>but matrix inversions are more difficult. In Geotk I was using the
>"vecmath" package (a derivative of legacy Sun Java3D library). However
>vecmath is licensed under LGPL 2 [1], which in my understanding can not
>work with Apache projects.
>
>Does anyone would recommend a small library implementing Matrix objects
>with basic operations? Note that I'm not looking for a full linear
>algebra package. In particular, I would like a library with optimized
>matrix implementation for the 3x3 and 4x4 cases - matrices of those
>sizes will occur very often, and dedicated implementations make a real
>difference both in performance and accuracy.
>
> Martin
>
>[1]
>https://java.net/projects/vecmath/sources/svn/content/trunk/LICENSE.txt
>