You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Vasiliki Kalavri <va...@gmail.com> on 2014/11/27 13:59:31 UTC
Samoa on Flink
Hello everyone,
Last week at Strata, I attended a talk about a streaming machine learning
framework from Yahoo labs, SAMOA: https://github.com/yahoo/samoa. The
project is going through apache incubation application and currently runs
on Storm, S4 and Samza.
After the talk, I spoke with the guy that presented it and we both thought
that it'd be cool to port Samoa on Flink.
I'm writing about this here because I want to know your thoughts on this
and also because I'm not very familiar with the Streaming-API, so I will
need your input :)
I have already discussed with Paris and there is interest to pick this up
from some students at KTH as well.
As far as I understand, Samoa expects an API similar to that of Storm and
from there the integration should be pretty straight-forward.
So, do you think it would be feasible and -most importantly- worth it?
Cheers,
Vasia.
Re: Samoa on Flink
Posted by Vasiliki Kalavri <va...@gmail.com>.
FYI, we'll be working on this here:
https://github.com/senorcarbone/samoa/tree/flink
Feel free to provide input :))
Cheers,
V.
On 3 December 2014 at 11:56, Paris Carbone <pa...@kth.se> wrote:
> Great news! Good to be aligned :)
>
> Paris
>
> > On 03 Dec 2014, at 10:24, Fabian Hueske <fh...@apache.org> wrote:
> >
> > FYI, Samoa was just proposed for Apache Incubation :-)
> >
> > 2014-11-27 15:19 GMT+01:00 Márton Balassi <ba...@gmail.com>:
> >
> >> Hey,
> >>
> >> Thanks Vasia, great idea.
> >> When we've started the streaming development the first sketch API we
> had is
> >> was actually really similar to Storm's API a couple of versions back.
> I've
> >> talked to Gyula about the issue when we've first heard about SAMOA ago
> and
> >> we are confident that porting Samoa to Flink streaming shoul be
> definitely
> >> managable and really useful as Paris mentions.
> >>
> >> Luckily they have ASF2 license so the legal cooperation is clear. The
> rest
> >> is on us and fortunately Paris might have a candidate already to kick
> off
> >> the development. :)
> >>
> >> Cheers,
> >>
> >> Marton
> >>
> >> On Thu, Nov 27, 2014 at 2:05 PM, Paris Carbone <pa...@kth.se> wrote:
> >>
> >>> Hey Vasia,
> >>>
> >>> I really like the idea and it seems we have some volunteers from our
> side
> >>> already!
> >>> It is in my opinion a starter task since it requires to create a simple
> >>> adapter for flink as well as mapping abstractions to our streaming
> >> topology
> >>> API in a similar fashion (in its simplest form just using several
> >> flatmaps).
> >>>
> >>> It will serve as a suitable common layer to make some comparisons
> between
> >>> flink-streaming to storm, s4 and samoa on incremental machine learning
> >>> tasks.
> >>>
> >>> Paris
> >>>
> >>>> On 27 Nov 2014, at 13:59, Vasiliki Kalavri <vasilikikalavri@gmail.com
> >
> >>> wrote:
> >>>>
> >>>> Hello everyone,
> >>>>
> >>>> Last week at Strata, I attended a talk about a streaming machine
> >> learning
> >>>> framework from Yahoo labs, SAMOA: https://github.com/yahoo/samoa. The
> >>>> project is going through apache incubation application and currently
> >> runs
> >>>> on Storm, S4 and Samza.
> >>>>
> >>>> After the talk, I spoke with the guy that presented it and we both
> >>> thought
> >>>> that it'd be cool to port Samoa on Flink.
> >>>>
> >>>> I'm writing about this here because I want to know your thoughts on
> >> this
> >>>> and also because I'm not very familiar with the Streaming-API, so I
> >> will
> >>>> need your input :)
> >>>> I have already discussed with Paris and there is interest to pick this
> >> up
> >>>> from some students at KTH as well.
> >>>> As far as I understand, Samoa expects an API similar to that of Storm
> >> and
> >>>> from there the integration should be pretty straight-forward.
> >>>>
> >>>> So, do you think it would be feasible and -most importantly- worth it?
> >>>>
> >>>> Cheers,
> >>>> Vasia.
> >>>
> >>>
> >>
>
>
Re: Samoa on Flink
Posted by Paris Carbone <pa...@kth.se>.
Great news! Good to be aligned :)
Paris
> On 03 Dec 2014, at 10:24, Fabian Hueske <fh...@apache.org> wrote:
>
> FYI, Samoa was just proposed for Apache Incubation :-)
>
> 2014-11-27 15:19 GMT+01:00 Márton Balassi <ba...@gmail.com>:
>
>> Hey,
>>
>> Thanks Vasia, great idea.
>> When we've started the streaming development the first sketch API we had is
>> was actually really similar to Storm's API a couple of versions back. I've
>> talked to Gyula about the issue when we've first heard about SAMOA ago and
>> we are confident that porting Samoa to Flink streaming shoul be definitely
>> managable and really useful as Paris mentions.
>>
>> Luckily they have ASF2 license so the legal cooperation is clear. The rest
>> is on us and fortunately Paris might have a candidate already to kick off
>> the development. :)
>>
>> Cheers,
>>
>> Marton
>>
>> On Thu, Nov 27, 2014 at 2:05 PM, Paris Carbone <pa...@kth.se> wrote:
>>
>>> Hey Vasia,
>>>
>>> I really like the idea and it seems we have some volunteers from our side
>>> already!
>>> It is in my opinion a starter task since it requires to create a simple
>>> adapter for flink as well as mapping abstractions to our streaming
>> topology
>>> API in a similar fashion (in its simplest form just using several
>> flatmaps).
>>>
>>> It will serve as a suitable common layer to make some comparisons between
>>> flink-streaming to storm, s4 and samoa on incremental machine learning
>>> tasks.
>>>
>>> Paris
>>>
>>>> On 27 Nov 2014, at 13:59, Vasiliki Kalavri <va...@gmail.com>
>>> wrote:
>>>>
>>>> Hello everyone,
>>>>
>>>> Last week at Strata, I attended a talk about a streaming machine
>> learning
>>>> framework from Yahoo labs, SAMOA: https://github.com/yahoo/samoa. The
>>>> project is going through apache incubation application and currently
>> runs
>>>> on Storm, S4 and Samza.
>>>>
>>>> After the talk, I spoke with the guy that presented it and we both
>>> thought
>>>> that it'd be cool to port Samoa on Flink.
>>>>
>>>> I'm writing about this here because I want to know your thoughts on
>> this
>>>> and also because I'm not very familiar with the Streaming-API, so I
>> will
>>>> need your input :)
>>>> I have already discussed with Paris and there is interest to pick this
>> up
>>>> from some students at KTH as well.
>>>> As far as I understand, Samoa expects an API similar to that of Storm
>> and
>>>> from there the integration should be pretty straight-forward.
>>>>
>>>> So, do you think it would be feasible and -most importantly- worth it?
>>>>
>>>> Cheers,
>>>> Vasia.
>>>
>>>
>>
Re: Samoa on Flink
Posted by Fabian Hueske <fh...@apache.org>.
FYI, Samoa was just proposed for Apache Incubation :-)
2014-11-27 15:19 GMT+01:00 Márton Balassi <ba...@gmail.com>:
> Hey,
>
> Thanks Vasia, great idea.
> When we've started the streaming development the first sketch API we had is
> was actually really similar to Storm's API a couple of versions back. I've
> talked to Gyula about the issue when we've first heard about SAMOA ago and
> we are confident that porting Samoa to Flink streaming shoul be definitely
> managable and really useful as Paris mentions.
>
> Luckily they have ASF2 license so the legal cooperation is clear. The rest
> is on us and fortunately Paris might have a candidate already to kick off
> the development. :)
>
> Cheers,
>
> Marton
>
> On Thu, Nov 27, 2014 at 2:05 PM, Paris Carbone <pa...@kth.se> wrote:
>
> > Hey Vasia,
> >
> > I really like the idea and it seems we have some volunteers from our side
> > already!
> > It is in my opinion a starter task since it requires to create a simple
> > adapter for flink as well as mapping abstractions to our streaming
> topology
> > API in a similar fashion (in its simplest form just using several
> flatmaps).
> >
> > It will serve as a suitable common layer to make some comparisons between
> > flink-streaming to storm, s4 and samoa on incremental machine learning
> > tasks.
> >
> > Paris
> >
> > > On 27 Nov 2014, at 13:59, Vasiliki Kalavri <va...@gmail.com>
> > wrote:
> > >
> > > Hello everyone,
> > >
> > > Last week at Strata, I attended a talk about a streaming machine
> learning
> > > framework from Yahoo labs, SAMOA: https://github.com/yahoo/samoa. The
> > > project is going through apache incubation application and currently
> runs
> > > on Storm, S4 and Samza.
> > >
> > > After the talk, I spoke with the guy that presented it and we both
> > thought
> > > that it'd be cool to port Samoa on Flink.
> > >
> > > I'm writing about this here because I want to know your thoughts on
> this
> > > and also because I'm not very familiar with the Streaming-API, so I
> will
> > > need your input :)
> > > I have already discussed with Paris and there is interest to pick this
> up
> > > from some students at KTH as well.
> > > As far as I understand, Samoa expects an API similar to that of Storm
> and
> > > from there the integration should be pretty straight-forward.
> > >
> > > So, do you think it would be feasible and -most importantly- worth it?
> > >
> > > Cheers,
> > > Vasia.
> >
> >
>
Re: Samoa on Flink
Posted by Márton Balassi <ba...@gmail.com>.
Hey,
Thanks Vasia, great idea.
When we've started the streaming development the first sketch API we had is
was actually really similar to Storm's API a couple of versions back. I've
talked to Gyula about the issue when we've first heard about SAMOA ago and
we are confident that porting Samoa to Flink streaming shoul be definitely
managable and really useful as Paris mentions.
Luckily they have ASF2 license so the legal cooperation is clear. The rest
is on us and fortunately Paris might have a candidate already to kick off
the development. :)
Cheers,
Marton
On Thu, Nov 27, 2014 at 2:05 PM, Paris Carbone <pa...@kth.se> wrote:
> Hey Vasia,
>
> I really like the idea and it seems we have some volunteers from our side
> already!
> It is in my opinion a starter task since it requires to create a simple
> adapter for flink as well as mapping abstractions to our streaming topology
> API in a similar fashion (in its simplest form just using several flatmaps).
>
> It will serve as a suitable common layer to make some comparisons between
> flink-streaming to storm, s4 and samoa on incremental machine learning
> tasks.
>
> Paris
>
> > On 27 Nov 2014, at 13:59, Vasiliki Kalavri <va...@gmail.com>
> wrote:
> >
> > Hello everyone,
> >
> > Last week at Strata, I attended a talk about a streaming machine learning
> > framework from Yahoo labs, SAMOA: https://github.com/yahoo/samoa. The
> > project is going through apache incubation application and currently runs
> > on Storm, S4 and Samza.
> >
> > After the talk, I spoke with the guy that presented it and we both
> thought
> > that it'd be cool to port Samoa on Flink.
> >
> > I'm writing about this here because I want to know your thoughts on this
> > and also because I'm not very familiar with the Streaming-API, so I will
> > need your input :)
> > I have already discussed with Paris and there is interest to pick this up
> > from some students at KTH as well.
> > As far as I understand, Samoa expects an API similar to that of Storm and
> > from there the integration should be pretty straight-forward.
> >
> > So, do you think it would be feasible and -most importantly- worth it?
> >
> > Cheers,
> > Vasia.
>
>
Re: Samoa on Flink
Posted by Paris Carbone <pa...@kth.se>.
Hey Vasia,
I really like the idea and it seems we have some volunteers from our side already!
It is in my opinion a starter task since it requires to create a simple adapter for flink as well as mapping abstractions to our streaming topology API in a similar fashion (in its simplest form just using several flatmaps).
It will serve as a suitable common layer to make some comparisons between flink-streaming to storm, s4 and samoa on incremental machine learning tasks.
Paris
> On 27 Nov 2014, at 13:59, Vasiliki Kalavri <va...@gmail.com> wrote:
>
> Hello everyone,
>
> Last week at Strata, I attended a talk about a streaming machine learning
> framework from Yahoo labs, SAMOA: https://github.com/yahoo/samoa. The
> project is going through apache incubation application and currently runs
> on Storm, S4 and Samza.
>
> After the talk, I spoke with the guy that presented it and we both thought
> that it'd be cool to port Samoa on Flink.
>
> I'm writing about this here because I want to know your thoughts on this
> and also because I'm not very familiar with the Streaming-API, so I will
> need your input :)
> I have already discussed with Paris and there is interest to pick this up
> from some students at KTH as well.
> As far as I understand, Samoa expects an API similar to that of Storm and
> from there the integration should be pretty straight-forward.
>
> So, do you think it would be feasible and -most importantly- worth it?
>
> Cheers,
> Vasia.