You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Trevor Grant <tr...@gmail.com> on 2016/02/18 17:14:01 UTC
Dense matricies in FlinkML
Is there a specific reason vectors are imported from Breeze and matrices
aren't?
Specifically I need to take the dot product of a matrix and a vector and a
matrix and a matrix. Was wondering if there is a theoretical reason or it
just wasn't needed at the time. Looks like some of this got lifted from
Spark (per the comments in the code, not trying to troll).
I need dot products of matrices and vectors which isn't implemented.
Thoughts?
tg
Trevor Grant
Data Scientist
https://github.com/rawkintrevo
http://stackexchange.com/users/3002022/rawkintrevo
http://trevorgrant.org
*"Fortunate is he, who is able to know the causes of things." -Virgil*
Re: Dense matricies in FlinkML
Posted by Theodore Vasiloudis <th...@gmail.com>.
Yes, what I meant by BLAS calls is to have bindings to the relevant Breeze
or netlib-java implementation .
--
Sent from a mobile device. May contain autocorrect errors.
On Feb 19, 2016 3:06 PM, "Trevor Grant" <tr...@gmail.com> wrote:
> That makes sense, a more accurate question would be, why does Vector.scala
> provide BLAS methods (functions, routines, whatever you call them), and
> Matrix doesn't?
>
> I assume they are there for speed (?) so does Matrix need them?
>
>
>
> Trevor Grant
> Data Scientist
> https://github.com/rawkintrevo
> http://stackexchange.com/users/3002022/rawkintrevo
> http://trevorgrant.org
>
> *"Fortunate is he, who is able to know the causes of things." -Virgil*
>
>
> On Fri, Feb 19, 2016 at 6:29 AM, Till Rohrmann <tr...@apache.org>
> wrote:
>
> > The idea was actually to leverage existing linear algebra libraries such
> as
> > breeze instead of building another blas implementation which will never
> be
> > as good as the ones out there.
> >
> > Cheers,
> > Till
> >
> > On Fri, Feb 19, 2016 at 9:48 AM, Theodore Vasiloudis <
> > theodoros.vasiloudis@gmail.com> wrote:
> >
> > > Just to note: This should be a separate PR if you plan on contributing
> > > this.
> > >
> > > On Thu, Feb 18, 2016 at 7:54 PM, Márton Balassi <
> > balassi.marton@gmail.com>
> > > wrote:
> > >
> > > > Hi guys,
> > > >
> > > > They are at least already registered for serialization [1], so there
> > > should
> > > > be no intentional conflict as Theo has suggested.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/common/FlinkMLTools.scala#L67-L73
> > > >
> > > > Best,
> > > >
> > > > Marton
> > > >
> > > > On Thu, Feb 18, 2016 at 7:47 PM, Theodore Vasiloudis <
> > > > theodoros.vasiloudis@gmail.com> wrote:
> > > >
> > > > > Hello Trevor,
> > > > >
> > > > > IIRC it was mostly that they weren't needed at the time. Feel free
> to
> > > > add,
> > > > > along with BLAS ops.
> > > > >
> > > > > Cheers,
> > > > > Theo
> > > > >
> > > > > On Thu, Feb 18, 2016 at 5:14 PM, Trevor Grant <
> > > trevor.d.grant@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Is there a specific reason vectors are imported from Breeze and
> > > > matrices
> > > > > > aren't?
> > > > > >
> > > > > > Specifically I need to take the dot product of a matrix and a
> > vector
> > > > and
> > > > > a
> > > > > > matrix and a matrix. Was wondering if there is a theoretical
> reason
> > > or
> > > > it
> > > > > > just wasn't needed at the time. Looks like some of this got
> lifted
> > > > from
> > > > > > Spark (per the comments in the code, not trying to troll).
> > > > > >
> > > > > > I need dot products of matrices and vectors which isn't
> > implemented.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > tg
> > > > > >
> > > > > >
> > > > > >
> > > > > > Trevor Grant
> > > > > > Data Scientist
> > > > > > https://github.com/rawkintrevo
> > > > > > http://stackexchange.com/users/3002022/rawkintrevo
> > > > > > http://trevorgrant.org
> > > > > >
> > > > > > *"Fortunate is he, who is able to know the causes of things."
> > > -Virgil*
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Dense matricies in FlinkML
Posted by Trevor Grant <tr...@gmail.com>.
That makes sense, a more accurate question would be, why does Vector.scala
provide BLAS methods (functions, routines, whatever you call them), and
Matrix doesn't?
I assume they are there for speed (?) so does Matrix need them?
Trevor Grant
Data Scientist
https://github.com/rawkintrevo
http://stackexchange.com/users/3002022/rawkintrevo
http://trevorgrant.org
*"Fortunate is he, who is able to know the causes of things." -Virgil*
On Fri, Feb 19, 2016 at 6:29 AM, Till Rohrmann <tr...@apache.org> wrote:
> The idea was actually to leverage existing linear algebra libraries such as
> breeze instead of building another blas implementation which will never be
> as good as the ones out there.
>
> Cheers,
> Till
>
> On Fri, Feb 19, 2016 at 9:48 AM, Theodore Vasiloudis <
> theodoros.vasiloudis@gmail.com> wrote:
>
> > Just to note: This should be a separate PR if you plan on contributing
> > this.
> >
> > On Thu, Feb 18, 2016 at 7:54 PM, Márton Balassi <
> balassi.marton@gmail.com>
> > wrote:
> >
> > > Hi guys,
> > >
> > > They are at least already registered for serialization [1], so there
> > should
> > > be no intentional conflict as Theo has suggested.
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/common/FlinkMLTools.scala#L67-L73
> > >
> > > Best,
> > >
> > > Marton
> > >
> > > On Thu, Feb 18, 2016 at 7:47 PM, Theodore Vasiloudis <
> > > theodoros.vasiloudis@gmail.com> wrote:
> > >
> > > > Hello Trevor,
> > > >
> > > > IIRC it was mostly that they weren't needed at the time. Feel free to
> > > add,
> > > > along with BLAS ops.
> > > >
> > > > Cheers,
> > > > Theo
> > > >
> > > > On Thu, Feb 18, 2016 at 5:14 PM, Trevor Grant <
> > trevor.d.grant@gmail.com>
> > > > wrote:
> > > >
> > > > > Is there a specific reason vectors are imported from Breeze and
> > > matrices
> > > > > aren't?
> > > > >
> > > > > Specifically I need to take the dot product of a matrix and a
> vector
> > > and
> > > > a
> > > > > matrix and a matrix. Was wondering if there is a theoretical reason
> > or
> > > it
> > > > > just wasn't needed at the time. Looks like some of this got lifted
> > > from
> > > > > Spark (per the comments in the code, not trying to troll).
> > > > >
> > > > > I need dot products of matrices and vectors which isn't
> implemented.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > tg
> > > > >
> > > > >
> > > > >
> > > > > Trevor Grant
> > > > > Data Scientist
> > > > > https://github.com/rawkintrevo
> > > > > http://stackexchange.com/users/3002022/rawkintrevo
> > > > > http://trevorgrant.org
> > > > >
> > > > > *"Fortunate is he, who is able to know the causes of things."
> > -Virgil*
> > > > >
> > > >
> > >
> >
>
Re: Dense matricies in FlinkML
Posted by Till Rohrmann <tr...@apache.org>.
The idea was actually to leverage existing linear algebra libraries such as
breeze instead of building another blas implementation which will never be
as good as the ones out there.
Cheers,
Till
On Fri, Feb 19, 2016 at 9:48 AM, Theodore Vasiloudis <
theodoros.vasiloudis@gmail.com> wrote:
> Just to note: This should be a separate PR if you plan on contributing
> this.
>
> On Thu, Feb 18, 2016 at 7:54 PM, Márton Balassi <ba...@gmail.com>
> wrote:
>
> > Hi guys,
> >
> > They are at least already registered for serialization [1], so there
> should
> > be no intentional conflict as Theo has suggested.
> >
> > [1]
> >
> >
> https://github.com/apache/flink/blob/master/flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/common/FlinkMLTools.scala#L67-L73
> >
> > Best,
> >
> > Marton
> >
> > On Thu, Feb 18, 2016 at 7:47 PM, Theodore Vasiloudis <
> > theodoros.vasiloudis@gmail.com> wrote:
> >
> > > Hello Trevor,
> > >
> > > IIRC it was mostly that they weren't needed at the time. Feel free to
> > add,
> > > along with BLAS ops.
> > >
> > > Cheers,
> > > Theo
> > >
> > > On Thu, Feb 18, 2016 at 5:14 PM, Trevor Grant <
> trevor.d.grant@gmail.com>
> > > wrote:
> > >
> > > > Is there a specific reason vectors are imported from Breeze and
> > matrices
> > > > aren't?
> > > >
> > > > Specifically I need to take the dot product of a matrix and a vector
> > and
> > > a
> > > > matrix and a matrix. Was wondering if there is a theoretical reason
> or
> > it
> > > > just wasn't needed at the time. Looks like some of this got lifted
> > from
> > > > Spark (per the comments in the code, not trying to troll).
> > > >
> > > > I need dot products of matrices and vectors which isn't implemented.
> > > >
> > > > Thoughts?
> > > >
> > > > tg
> > > >
> > > >
> > > >
> > > > Trevor Grant
> > > > Data Scientist
> > > > https://github.com/rawkintrevo
> > > > http://stackexchange.com/users/3002022/rawkintrevo
> > > > http://trevorgrant.org
> > > >
> > > > *"Fortunate is he, who is able to know the causes of things."
> -Virgil*
> > > >
> > >
> >
>
Re: Dense matricies in FlinkML
Posted by Trevor Grant <tr...@gmail.com>.
Agree. Since the current matrices will support what I'm trying to do, I'm
going to drive on with the Neural nets. I think the BLAS should speed up
matrix computations ( someone more familiar with Java math things can
correct me) so I'll make that its own issue.
Trevor Grant
Data Scientist
https://github.com/rawkintrevo
http://stackexchange.com/users/3002022/rawkintrevo
http://trevorgrant.org
*"Fortunate is he, who is able to know the causes of things." -Virgil*
On Fri, Feb 19, 2016 at 2:48 AM, Theodore Vasiloudis <
theodoros.vasiloudis@gmail.com> wrote:
> Just to note: This should be a separate PR if you plan on contributing
> this.
>
> On Thu, Feb 18, 2016 at 7:54 PM, Márton Balassi <ba...@gmail.com>
> wrote:
>
> > Hi guys,
> >
> > They are at least already registered for serialization [1], so there
> should
> > be no intentional conflict as Theo has suggested.
> >
> > [1]
> >
> >
> https://github.com/apache/flink/blob/master/flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/common/FlinkMLTools.scala#L67-L73
> >
> > Best,
> >
> > Marton
> >
> > On Thu, Feb 18, 2016 at 7:47 PM, Theodore Vasiloudis <
> > theodoros.vasiloudis@gmail.com> wrote:
> >
> > > Hello Trevor,
> > >
> > > IIRC it was mostly that they weren't needed at the time. Feel free to
> > add,
> > > along with BLAS ops.
> > >
> > > Cheers,
> > > Theo
> > >
> > > On Thu, Feb 18, 2016 at 5:14 PM, Trevor Grant <
> trevor.d.grant@gmail.com>
> > > wrote:
> > >
> > > > Is there a specific reason vectors are imported from Breeze and
> > matrices
> > > > aren't?
> > > >
> > > > Specifically I need to take the dot product of a matrix and a vector
> > and
> > > a
> > > > matrix and a matrix. Was wondering if there is a theoretical reason
> or
> > it
> > > > just wasn't needed at the time. Looks like some of this got lifted
> > from
> > > > Spark (per the comments in the code, not trying to troll).
> > > >
> > > > I need dot products of matrices and vectors which isn't implemented.
> > > >
> > > > Thoughts?
> > > >
> > > > tg
> > > >
> > > >
> > > >
> > > > Trevor Grant
> > > > Data Scientist
> > > > https://github.com/rawkintrevo
> > > > http://stackexchange.com/users/3002022/rawkintrevo
> > > > http://trevorgrant.org
> > > >
> > > > *"Fortunate is he, who is able to know the causes of things."
> -Virgil*
> > > >
> > >
> >
>
Re: Dense matricies in FlinkML
Posted by Theodore Vasiloudis <th...@gmail.com>.
Just to note: This should be a separate PR if you plan on contributing this.
On Thu, Feb 18, 2016 at 7:54 PM, Márton Balassi <ba...@gmail.com>
wrote:
> Hi guys,
>
> They are at least already registered for serialization [1], so there should
> be no intentional conflict as Theo has suggested.
>
> [1]
>
> https://github.com/apache/flink/blob/master/flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/common/FlinkMLTools.scala#L67-L73
>
> Best,
>
> Marton
>
> On Thu, Feb 18, 2016 at 7:47 PM, Theodore Vasiloudis <
> theodoros.vasiloudis@gmail.com> wrote:
>
> > Hello Trevor,
> >
> > IIRC it was mostly that they weren't needed at the time. Feel free to
> add,
> > along with BLAS ops.
> >
> > Cheers,
> > Theo
> >
> > On Thu, Feb 18, 2016 at 5:14 PM, Trevor Grant <tr...@gmail.com>
> > wrote:
> >
> > > Is there a specific reason vectors are imported from Breeze and
> matrices
> > > aren't?
> > >
> > > Specifically I need to take the dot product of a matrix and a vector
> and
> > a
> > > matrix and a matrix. Was wondering if there is a theoretical reason or
> it
> > > just wasn't needed at the time. Looks like some of this got lifted
> from
> > > Spark (per the comments in the code, not trying to troll).
> > >
> > > I need dot products of matrices and vectors which isn't implemented.
> > >
> > > Thoughts?
> > >
> > > tg
> > >
> > >
> > >
> > > Trevor Grant
> > > Data Scientist
> > > https://github.com/rawkintrevo
> > > http://stackexchange.com/users/3002022/rawkintrevo
> > > http://trevorgrant.org
> > >
> > > *"Fortunate is he, who is able to know the causes of things." -Virgil*
> > >
> >
>
Re: Dense matricies in FlinkML
Posted by Márton Balassi <ba...@gmail.com>.
Hi guys,
They are at least already registered for serialization [1], so there should
be no intentional conflict as Theo has suggested.
[1]
https://github.com/apache/flink/blob/master/flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/common/FlinkMLTools.scala#L67-L73
Best,
Marton
On Thu, Feb 18, 2016 at 7:47 PM, Theodore Vasiloudis <
theodoros.vasiloudis@gmail.com> wrote:
> Hello Trevor,
>
> IIRC it was mostly that they weren't needed at the time. Feel free to add,
> along with BLAS ops.
>
> Cheers,
> Theo
>
> On Thu, Feb 18, 2016 at 5:14 PM, Trevor Grant <tr...@gmail.com>
> wrote:
>
> > Is there a specific reason vectors are imported from Breeze and matrices
> > aren't?
> >
> > Specifically I need to take the dot product of a matrix and a vector and
> a
> > matrix and a matrix. Was wondering if there is a theoretical reason or it
> > just wasn't needed at the time. Looks like some of this got lifted from
> > Spark (per the comments in the code, not trying to troll).
> >
> > I need dot products of matrices and vectors which isn't implemented.
> >
> > Thoughts?
> >
> > tg
> >
> >
> >
> > Trevor Grant
> > Data Scientist
> > https://github.com/rawkintrevo
> > http://stackexchange.com/users/3002022/rawkintrevo
> > http://trevorgrant.org
> >
> > *"Fortunate is he, who is able to know the causes of things." -Virgil*
> >
>
Re: Dense matricies in FlinkML
Posted by Theodore Vasiloudis <th...@gmail.com>.
Hello Trevor,
IIRC it was mostly that they weren't needed at the time. Feel free to add,
along with BLAS ops.
Cheers,
Theo
On Thu, Feb 18, 2016 at 5:14 PM, Trevor Grant <tr...@gmail.com>
wrote:
> Is there a specific reason vectors are imported from Breeze and matrices
> aren't?
>
> Specifically I need to take the dot product of a matrix and a vector and a
> matrix and a matrix. Was wondering if there is a theoretical reason or it
> just wasn't needed at the time. Looks like some of this got lifted from
> Spark (per the comments in the code, not trying to troll).
>
> I need dot products of matrices and vectors which isn't implemented.
>
> Thoughts?
>
> tg
>
>
>
> Trevor Grant
> Data Scientist
> https://github.com/rawkintrevo
> http://stackexchange.com/users/3002022/rawkintrevo
> http://trevorgrant.org
>
> *"Fortunate is he, who is able to know the causes of things." -Virgil*
>