You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Stephan Ewen <se...@apache.org> on 2014/08/06 17:18:32 UTC
Re: Adding the streaming project to the main repository
The ICLA's and the SGA are cleared as far as I know.
I think we should merge the code into the current master (not the 0.6
release, but the successive one).
Re: Adding the streaming project to the main repository
Posted by Henry Saputra <he...@gmail.com>.
W00t!
- Henry
On Mon, Aug 18, 2014 at 10:55 AM, Stephan Ewen <se...@apache.org> wrote:
> After the Apache Secretary confirmed that the SGA has arrived and the ICLAs
> are filed, I have merged the streaming code into the master for the next
> release.
>
> A whole bunch of code that was!
>
> Great work, all of you. Looking forward to what this blossoms into... It's
> a good day, today :-)
>
>
>
> On Wed, Aug 6, 2014 at 7:17 PM, Robert Metzger <rm...@apache.org> wrote:
>
>> Cool. I if we have the confirmation by the secretary (or for foundation
>> members:
>> https://svn.apache.org/repos/private/foundation/officers/iclas.txt),
>> I vote for adding the code to the "master" branch.
>>
>>
>> On Wed, Aug 6, 2014 at 5:18 PM, Stephan Ewen <se...@apache.org> wrote:
>>
>> > The ICLA's and the SGA are cleared as far as I know.
>> >
>> > I think we should merge the code into the current master (not the 0.6
>> > release, but the successive one).
>> >
>>
Re: Adding the streaming project to the main repository
Posted by Henry Saputra <he...@gmail.com>.
Thanks, I just assign you the issue =)
- Henry
On Mon, Aug 18, 2014 at 11:38 PM, Márton Balassi
<ba...@gmail.com> wrote:
> My username is mbalassi.
> I've started watching the issue to give you a link. :)
>
>
> On Tue, Aug 19, 2014 at 8:07 AM, Henry Saputra <he...@gmail.com>
> wrote:
>
>> Hi Marton,
>>
>> I created the JIRA ticket to track the streaming documentation:
>> https://issues.apache.org/jira/browse/FLINK-1058
>> Somehow I could not find your ASF JIRA username. Could you tell me
>> what is your ASF JIRA username?
>>
>> - Henry
>>
>> On Mon, Aug 18, 2014 at 10:29 PM, Márton Balassi
>> <ba...@gmail.com> wrote:
>> > Sure, please assign it to me.
>> >
>> > On Aug 19, 2014 2:44 AM, "Henry Saputra" <he...@gmail.com>
>> wrote:
>> >
>> >> Thanks Stephan. If no one object I will create JIRA ticket as reminder
>> >> to add formal documentation for the streaming feature.
>> >>
>> >> - Henry
>> >>
>> >> On Mon, Aug 18, 2014 at 11:53 AM, Stephan Ewen <se...@apache.org>
>> wrote:
>> >> > The streaming code is in "flink-addons", for new/experimental code.
>> >> >
>> >> > Documents should come over the next days/weeks, definitely before we
>> make
>> >> > this part of the core.
>> >> >
>> >> > Right now, I would suggest to have a look at some of the examples, to
>> >> get a
>> >> > feeling for the addon, check for example this here:
>> >> >
>> >>
>> https://github.com/apache/incubator-flink/tree/master/flink-addons/flink-streaming/flink-streaming-examples/src/main/java/org/apache/flink/streaming/examples/wordcount
>> >> >
>> >> > (The example reads a file for simplicity, but the project also
>> provides
>> >> > connectors for Kafka, RabbitMQ, ...)
>> >>
>>
Re: Adding the streaming project to the main repository
Posted by Márton Balassi <ba...@gmail.com>.
My username is mbalassi.
I've started watching the issue to give you a link. :)
On Tue, Aug 19, 2014 at 8:07 AM, Henry Saputra <he...@gmail.com>
wrote:
> Hi Marton,
>
> I created the JIRA ticket to track the streaming documentation:
> https://issues.apache.org/jira/browse/FLINK-1058
> Somehow I could not find your ASF JIRA username. Could you tell me
> what is your ASF JIRA username?
>
> - Henry
>
> On Mon, Aug 18, 2014 at 10:29 PM, Márton Balassi
> <ba...@gmail.com> wrote:
> > Sure, please assign it to me.
> >
> > On Aug 19, 2014 2:44 AM, "Henry Saputra" <he...@gmail.com>
> wrote:
> >
> >> Thanks Stephan. If no one object I will create JIRA ticket as reminder
> >> to add formal documentation for the streaming feature.
> >>
> >> - Henry
> >>
> >> On Mon, Aug 18, 2014 at 11:53 AM, Stephan Ewen <se...@apache.org>
> wrote:
> >> > The streaming code is in "flink-addons", for new/experimental code.
> >> >
> >> > Documents should come over the next days/weeks, definitely before we
> make
> >> > this part of the core.
> >> >
> >> > Right now, I would suggest to have a look at some of the examples, to
> >> get a
> >> > feeling for the addon, check for example this here:
> >> >
> >>
> https://github.com/apache/incubator-flink/tree/master/flink-addons/flink-streaming/flink-streaming-examples/src/main/java/org/apache/flink/streaming/examples/wordcount
> >> >
> >> > (The example reads a file for simplicity, but the project also
> provides
> >> > connectors for Kafka, RabbitMQ, ...)
> >>
>
Re: Adding the streaming project to the main repository
Posted by Henry Saputra <he...@gmail.com>.
Hi Marton,
I created the JIRA ticket to track the streaming documentation:
https://issues.apache.org/jira/browse/FLINK-1058
Somehow I could not find your ASF JIRA username. Could you tell me
what is your ASF JIRA username?
- Henry
On Mon, Aug 18, 2014 at 10:29 PM, Márton Balassi
<ba...@gmail.com> wrote:
> Sure, please assign it to me.
>
> On Aug 19, 2014 2:44 AM, "Henry Saputra" <he...@gmail.com> wrote:
>
>> Thanks Stephan. If no one object I will create JIRA ticket as reminder
>> to add formal documentation for the streaming feature.
>>
>> - Henry
>>
>> On Mon, Aug 18, 2014 at 11:53 AM, Stephan Ewen <se...@apache.org> wrote:
>> > The streaming code is in "flink-addons", for new/experimental code.
>> >
>> > Documents should come over the next days/weeks, definitely before we make
>> > this part of the core.
>> >
>> > Right now, I would suggest to have a look at some of the examples, to
>> get a
>> > feeling for the addon, check for example this here:
>> >
>> https://github.com/apache/incubator-flink/tree/master/flink-addons/flink-streaming/flink-streaming-examples/src/main/java/org/apache/flink/streaming/examples/wordcount
>> >
>> > (The example reads a file for simplicity, but the project also provides
>> > connectors for Kafka, RabbitMQ, ...)
>>
Re: Adding the streaming project to the main repository
Posted by Márton Balassi <ba...@gmail.com>.
Sure, please assign it to me.
On Aug 19, 2014 2:44 AM, "Henry Saputra" <he...@gmail.com> wrote:
> Thanks Stephan. If no one object I will create JIRA ticket as reminder
> to add formal documentation for the streaming feature.
>
> - Henry
>
> On Mon, Aug 18, 2014 at 11:53 AM, Stephan Ewen <se...@apache.org> wrote:
> > The streaming code is in "flink-addons", for new/experimental code.
> >
> > Documents should come over the next days/weeks, definitely before we make
> > this part of the core.
> >
> > Right now, I would suggest to have a look at some of the examples, to
> get a
> > feeling for the addon, check for example this here:
> >
> https://github.com/apache/incubator-flink/tree/master/flink-addons/flink-streaming/flink-streaming-examples/src/main/java/org/apache/flink/streaming/examples/wordcount
> >
> > (The example reads a file for simplicity, but the project also provides
> > connectors for Kafka, RabbitMQ, ...)
>
Re: Adding the streaming project to the main repository
Posted by Henry Saputra <he...@gmail.com>.
Thanks Stephan. If no one object I will create JIRA ticket as reminder
to add formal documentation for the streaming feature.
- Henry
On Mon, Aug 18, 2014 at 11:53 AM, Stephan Ewen <se...@apache.org> wrote:
> The streaming code is in "flink-addons", for new/experimental code.
>
> Documents should come over the next days/weeks, definitely before we make
> this part of the core.
>
> Right now, I would suggest to have a look at some of the examples, to get a
> feeling for the addon, check for example this here:
> https://github.com/apache/incubator-flink/tree/master/flink-addons/flink-streaming/flink-streaming-examples/src/main/java/org/apache/flink/streaming/examples/wordcount
>
> (The example reads a file for simplicity, but the project also provides
> connectors for Kafka, RabbitMQ, ...)
Re: Adding the streaming project to the main repository
Posted by Fabian Hueske <fh...@apache.org>.
Thanks for the explanation!
Very nice set of features. Looking forward to check it out myself :-)
2014-08-18 21:38 GMT+02:00 Gyula Fóra <gy...@gmail.com>:
> Hey,
>
> The simple reduce is like what you said yes. But there are also grouped
> reduce which you can use by calling .groupBy(keyposition) and then reduce.
>
> Also there is reduce for windows: batchReduce and windowReduce batch gives
> you a sliding window over a predefined number of records, and window reduce
> gices you the same but by time. (also there are grouped versions of these)
>
> Cheers,
> Gyula
>
>
> On Mon, Aug 18, 2014 at 9:19 PM, Fabian Hueske <fh...@apache.org> wrote:
>
> > Hi folks,
> >
> > great work!
> >
> > Looking at the example I have a quick question. What's the semantics of
> the
> > Reduce operator? I guess its not a window reduce.
> > Is it backed by a hash table and every input tuple updates the hash table
> > and returns the updated value?
> >
> > Cheers, Fabian
> >
> >
> > 2014-08-18 20:53 GMT+02:00 Stephan Ewen <se...@apache.org>:
> >
> > > The streaming code is in "flink-addons", for new/experimental code.
> > >
> > > Documents should come over the next days/weeks, definitely before we
> make
> > > this part of the core.
> > >
> > > Right now, I would suggest to have a look at some of the examples, to
> > get a
> > > feeling for the addon, check for example this here:
> > >
> > >
> >
> https://github.com/apache/incubator-flink/tree/master/flink-addons/flink-streaming/flink-streaming-examples/src/main/java/org/apache/flink/streaming/examples/wordcount
> > >
> > > (The example reads a file for simplicity, but the project also provides
> > > connectors for Kafka, RabbitMQ, ...)
> > >
> >
>
Re: Adding the streaming project to the main repository
Posted by Gyula Fóra <gy...@gmail.com>.
Hey,
The simple reduce is like what you said yes. But there are also grouped
reduce which you can use by calling .groupBy(keyposition) and then reduce.
Also there is reduce for windows: batchReduce and windowReduce batch gives
you a sliding window over a predefined number of records, and window reduce
gices you the same but by time. (also there are grouped versions of these)
Cheers,
Gyula
On Mon, Aug 18, 2014 at 9:19 PM, Fabian Hueske <fh...@apache.org> wrote:
> Hi folks,
>
> great work!
>
> Looking at the example I have a quick question. What's the semantics of the
> Reduce operator? I guess its not a window reduce.
> Is it backed by a hash table and every input tuple updates the hash table
> and returns the updated value?
>
> Cheers, Fabian
>
>
> 2014-08-18 20:53 GMT+02:00 Stephan Ewen <se...@apache.org>:
>
> > The streaming code is in "flink-addons", for new/experimental code.
> >
> > Documents should come over the next days/weeks, definitely before we make
> > this part of the core.
> >
> > Right now, I would suggest to have a look at some of the examples, to
> get a
> > feeling for the addon, check for example this here:
> >
> >
> https://github.com/apache/incubator-flink/tree/master/flink-addons/flink-streaming/flink-streaming-examples/src/main/java/org/apache/flink/streaming/examples/wordcount
> >
> > (The example reads a file for simplicity, but the project also provides
> > connectors for Kafka, RabbitMQ, ...)
> >
>
Re: Adding the streaming project to the main repository
Posted by Fabian Hueske <fh...@apache.org>.
Hi folks,
great work!
Looking at the example I have a quick question. What's the semantics of the
Reduce operator? I guess its not a window reduce.
Is it backed by a hash table and every input tuple updates the hash table
and returns the updated value?
Cheers, Fabian
2014-08-18 20:53 GMT+02:00 Stephan Ewen <se...@apache.org>:
> The streaming code is in "flink-addons", for new/experimental code.
>
> Documents should come over the next days/weeks, definitely before we make
> this part of the core.
>
> Right now, I would suggest to have a look at some of the examples, to get a
> feeling for the addon, check for example this here:
>
> https://github.com/apache/incubator-flink/tree/master/flink-addons/flink-streaming/flink-streaming-examples/src/main/java/org/apache/flink/streaming/examples/wordcount
>
> (The example reads a file for simplicity, but the project also provides
> connectors for Kafka, RabbitMQ, ...)
>
Re: Adding the streaming project to the main repository
Posted by Stephan Ewen <se...@apache.org>.
The streaming code is in "flink-addons", for new/experimental code.
Documents should come over the next days/weeks, definitely before we make
this part of the core.
Right now, I would suggest to have a look at some of the examples, to get a
feeling for the addon, check for example this here:
https://github.com/apache/incubator-flink/tree/master/flink-addons/flink-streaming/flink-streaming-examples/src/main/java/org/apache/flink/streaming/examples/wordcount
(The example reads a file for simplicity, but the project also provides
connectors for Kafka, RabbitMQ, ...)
Re: Adding the streaming project to the main repository
Posted by Henry Saputra <he...@gmail.com>.
Hmm, quick question, I could not find any documentation about the
streaming support. Is it part of the source code or will there be
additional doc included?
- Henry
On Mon, Aug 18, 2014 at 10:55 AM, Stephan Ewen <se...@apache.org> wrote:
> After the Apache Secretary confirmed that the SGA has arrived and the ICLAs
> are filed, I have merged the streaming code into the master for the next
> release.
>
> A whole bunch of code that was!
>
> Great work, all of you. Looking forward to what this blossoms into... It's
> a good day, today :-)
>
>
>
> On Wed, Aug 6, 2014 at 7:17 PM, Robert Metzger <rm...@apache.org> wrote:
>
>> Cool. I if we have the confirmation by the secretary (or for foundation
>> members:
>> https://svn.apache.org/repos/private/foundation/officers/iclas.txt),
>> I vote for adding the code to the "master" branch.
>>
>>
>> On Wed, Aug 6, 2014 at 5:18 PM, Stephan Ewen <se...@apache.org> wrote:
>>
>> > The ICLA's and the SGA are cleared as far as I know.
>> >
>> > I think we should merge the code into the current master (not the 0.6
>> > release, but the successive one).
>> >
>>
Re: Adding the streaming project to the main repository
Posted by Stephan Ewen <se...@apache.org>.
After the Apache Secretary confirmed that the SGA has arrived and the ICLAs
are filed, I have merged the streaming code into the master for the next
release.
A whole bunch of code that was!
Great work, all of you. Looking forward to what this blossoms into... It's
a good day, today :-)
On Wed, Aug 6, 2014 at 7:17 PM, Robert Metzger <rm...@apache.org> wrote:
> Cool. I if we have the confirmation by the secretary (or for foundation
> members:
> https://svn.apache.org/repos/private/foundation/officers/iclas.txt),
> I vote for adding the code to the "master" branch.
>
>
> On Wed, Aug 6, 2014 at 5:18 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > The ICLA's and the SGA are cleared as far as I know.
> >
> > I think we should merge the code into the current master (not the 0.6
> > release, but the successive one).
> >
>
Re: Adding the streaming project to the main repository
Posted by Robert Metzger <rm...@apache.org>.
Cool. I if we have the confirmation by the secretary (or for foundation
members: https://svn.apache.org/repos/private/foundation/officers/iclas.txt),
I vote for adding the code to the "master" branch.
On Wed, Aug 6, 2014 at 5:18 PM, Stephan Ewen <se...@apache.org> wrote:
> The ICLA's and the SGA are cleared as far as I know.
>
> I think we should merge the code into the current master (not the 0.6
> release, but the successive one).
>