You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Robert Metzger <rm...@apache.org> on 2016/02/02 19:12:20 UTC
Re: Release Flink 1.0.0
Hi,
I think that getting the stop signal in would be very nice.
I would like to postpone the feature freeze till end of this week and
create the first RC on Monday. There are many open pull requests with fixes
that need to go in (stop signal, rocksdb state backend, interface
annotations, streaming api fixes)
On Mon, Jan 25, 2016 at 9:02 AM, Matthias J. Sax <mj...@apache.org> wrote:
> Hi,
>
> I also would like to get the STOP signal in. But I do not have time to
> work in it this week... According to Till's comments, this will be the
> last round of reviewing required. So I should be able to finish it till
> 3rd Feb, but not sure.
>
> What do you think about it?
>
> -Matthias
>
> On 01/25/2016 04:29 PM, Aljoscha Krettek wrote:
> > Hi,
> > I think the refactoring of Partitioned State and the WindowOperator on
> state work is almost ready. I also have the RocksDB state backend working.
> I’m running some tests now on the cluster and should be able to open a PR
> tomorrow.
> >
> >
> >> On 25 Jan 2016, at 15:36, Stephan Ewen <se...@apache.org> wrote:
> >>
> >> I agree, with Gyula, one out-of-core state backend should be in. We are
> >> pretty close to that. Aljoscha has done good work on extending test
> >> coverage for state backends, so we should be pretty comfortable that it
> >> works as well, once we integrate new state backends with the tests.
> >>
> >> There is a bit of work do do around extending the interface of the
> >> key/value state. I would like to start a separate thread on that today
> or
> >> tomorrow...
> >>
> >>
> >>
> >> On Mon, Jan 25, 2016 at 12:16 PM, Gyula Fóra <gy...@apache.org> wrote:
> >>
> >>> Hi,
> >>>
> >>> I agree that getting Flink 1.0.0 out soon would be great as Flink is
> in a
> >>> pretty solid state right now.
> >>>
> >>> I wonder whether it would make sense to include an out-of-core state
> >>> backend in streaming core that can be used with partitioned/window
> states.
> >>> I think if we are releasing 1.0.0 we should have a solid feature set
> for
> >>> our strong steaming use-cases (in this case stateful, and windowed
> >>> computations) and this should be a part of that.
> >>>
> >>> I know that Aljoscha is working on a solution for this which will
> probably
> >>> involve a heavy refactor of the State backend interfaces, and I am also
> >>> working on a similar solution. Maybe it would be good to get at least
> one
> >>> good robust solution for this in and definitely Aljoscha's refactor
> for the
> >>> interfaces.
> >>>
> >>> If we decide to do this, I think this needs 1-2 extra weeks of proper
> >>> testing so this might delay the schedule a little bit.
> >>>
> >>> What do you think?
> >>>
> >>> Gyula
> >>>
> >>>
> >>>
> >>> Robert Metzger <rm...@apache.org> ezt írta (időpont: 2016. jan.
> 25., H,
> >>> 11:54):
> >>>
> >>>> Hi,
> >>>>
> >>>> I would like to release 1.0.0 in the next weeks.
> >>>> Looking at the JIRAs, I think we are going to close a lot of blocking
> >>>> issues soon. How about we do a first release candidate on Wednesday,
> 3.
> >>>> February?
> >>>>
> >>>> The first release candidate is most likely not going to pass the vote,
> >>> the
> >>>> primary goal will be collecting a list of issues we need to address.
> >>>>
> >>>> There is also a Wiki page for the 1.0 release:
> >>>> https://cwiki.apache.org/confluence/display/FLINK/1.0+Release
> >>>>
> >>>> Please -1 to this message if 3. February is too soon for the first RC
> (it
> >>>> also means that we'll do a feature freeze around that time).
> >>>>
> >>>
> >
>
>
Re: Release Flink 1.0.0
Posted by Robert Metzger <rm...@apache.org>.
Okay, I didn't check the individual PRs closely. I agree that we should not
merge big core changes if we are not certain about their stability.
On Mon, Feb 8, 2016 at 12:04 PM, Fabian Hueske <fh...@gmail.com> wrote:
> Regarding FLINK-2237 (hash-based combiner), I think we need a bit more
> time.
> It is a fairly large contribution and touches/adds core functionality.
>
> I started reviewing the PR and will suggest a few changes in the next days.
>
> 2016-02-08 11:48 GMT+01:00 Robert Metzger <rm...@apache.org>:
>
> > There are still some 8 open blockers for the 1.0 release:
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC
> >
> > I also think that there are some pull requests which are almost ready to
> > merge and should go in:
> >
> > - [FLINK-3341] Make 'auto.offset.reset' compatible with Kafka 0.8 and 0.9
> > - [FLINK-3336] Add Rescale Data Shipping for DataStream
> > - FLINK-2213 Makes the number of vcores per YARN container configurable
> > - [FLINK-2021] Rework examples to use ParameterTool
> > - [FLINK-3310] [runtime, runtime-web] Add back pressure statistics
> > - [FLINK-3296] Remove 'flushing' behavior of the OutputFormat in
> DataStream
> > API
> > - [FLINK-3270] Add Kafka example
> > - FLINK-2380: allow to specify the default filesystem scheme in the flink
> > configuration file.
> > - [FLINK-2237] [runtime] Add hash-based combiner.
> > - [FLINK-3187] Introduce RestartStrategy to decouple restarting behaviour
> > from ExecutionGraph
> >
> > I try to get these PRs in and push the owners of the blocking issues for
> > resolutions.
> >
> >
> >
> > On Tue, Feb 2, 2016 at 7:12 PM, Robert Metzger <rm...@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > I think that getting the stop signal in would be very nice.
> > >
> > > I would like to postpone the feature freeze till end of this week and
> > > create the first RC on Monday. There are many open pull requests with
> > fixes
> > > that need to go in (stop signal, rocksdb state backend, interface
> > > annotations, streaming api fixes)
> > >
> > >
> > >
> > > On Mon, Jan 25, 2016 at 9:02 AM, Matthias J. Sax <mj...@apache.org>
> > wrote:
> > >
> > >> Hi,
> > >>
> > >> I also would like to get the STOP signal in. But I do not have time to
> > >> work in it this week... According to Till's comments, this will be the
> > >> last round of reviewing required. So I should be able to finish it
> till
> > >> 3rd Feb, but not sure.
> > >>
> > >> What do you think about it?
> > >>
> > >> -Matthias
> > >>
> > >> On 01/25/2016 04:29 PM, Aljoscha Krettek wrote:
> > >> > Hi,
> > >> > I think the refactoring of Partitioned State and the WindowOperator
> on
> > >> state work is almost ready. I also have the RocksDB state backend
> > working.
> > >> I’m running some tests now on the cluster and should be able to open a
> > PR
> > >> tomorrow.
> > >> >
> > >> >
> > >> >> On 25 Jan 2016, at 15:36, Stephan Ewen <se...@apache.org> wrote:
> > >> >>
> > >> >> I agree, with Gyula, one out-of-core state backend should be in. We
> > are
> > >> >> pretty close to that. Aljoscha has done good work on extending test
> > >> >> coverage for state backends, so we should be pretty comfortable
> that
> > it
> > >> >> works as well, once we integrate new state backends with the tests.
> > >> >>
> > >> >> There is a bit of work do do around extending the interface of the
> > >> >> key/value state. I would like to start a separate thread on that
> > today
> > >> or
> > >> >> tomorrow...
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Mon, Jan 25, 2016 at 12:16 PM, Gyula Fóra <gy...@apache.org>
> > >> wrote:
> > >> >>
> > >> >>> Hi,
> > >> >>>
> > >> >>> I agree that getting Flink 1.0.0 out soon would be great as Flink
> is
> > >> in a
> > >> >>> pretty solid state right now.
> > >> >>>
> > >> >>> I wonder whether it would make sense to include an out-of-core
> state
> > >> >>> backend in streaming core that can be used with partitioned/window
> > >> states.
> > >> >>> I think if we are releasing 1.0.0 we should have a solid feature
> set
> > >> for
> > >> >>> our strong steaming use-cases (in this case stateful, and
> windowed
> > >> >>> computations) and this should be a part of that.
> > >> >>>
> > >> >>> I know that Aljoscha is working on a solution for this which will
> > >> probably
> > >> >>> involve a heavy refactor of the State backend interfaces, and I am
> > >> also
> > >> >>> working on a similar solution. Maybe it would be good to get at
> > least
> > >> one
> > >> >>> good robust solution for this in and definitely Aljoscha's
> refactor
> > >> for the
> > >> >>> interfaces.
> > >> >>>
> > >> >>> If we decide to do this, I think this needs 1-2 extra weeks of
> > proper
> > >> >>> testing so this might delay the schedule a little bit.
> > >> >>>
> > >> >>> What do you think?
> > >> >>>
> > >> >>> Gyula
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> Robert Metzger <rm...@apache.org> ezt írta (időpont: 2016.
> jan.
> > >> 25., H,
> > >> >>> 11:54):
> > >> >>>
> > >> >>>> Hi,
> > >> >>>>
> > >> >>>> I would like to release 1.0.0 in the next weeks.
> > >> >>>> Looking at the JIRAs, I think we are going to close a lot of
> > blocking
> > >> >>>> issues soon. How about we do a first release candidate on
> > Wednesday,
> > >> 3.
> > >> >>>> February?
> > >> >>>>
> > >> >>>> The first release candidate is most likely not going to pass the
> > >> vote,
> > >> >>> the
> > >> >>>> primary goal will be collecting a list of issues we need to
> > address.
> > >> >>>>
> > >> >>>> There is also a Wiki page for the 1.0 release:
> > >> >>>> https://cwiki.apache.org/confluence/display/FLINK/1.0+Release
> > >> >>>>
> > >> >>>> Please -1 to this message if 3. February is too soon for the
> first
> > >> RC (it
> > >> >>>> also means that we'll do a feature freeze around that time).
> > >> >>>>
> > >> >>>
> > >> >
> > >>
> > >>
> > >
> >
>
Re: Release Flink 1.0.0
Posted by Fabian Hueske <fh...@gmail.com>.
Regarding FLINK-2237 (hash-based combiner), I think we need a bit more
time.
It is a fairly large contribution and touches/adds core functionality.
I started reviewing the PR and will suggest a few changes in the next days.
2016-02-08 11:48 GMT+01:00 Robert Metzger <rm...@apache.org>:
> There are still some 8 open blockers for the 1.0 release:
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC
>
> I also think that there are some pull requests which are almost ready to
> merge and should go in:
>
> - [FLINK-3341] Make 'auto.offset.reset' compatible with Kafka 0.8 and 0.9
> - [FLINK-3336] Add Rescale Data Shipping for DataStream
> - FLINK-2213 Makes the number of vcores per YARN container configurable
> - [FLINK-2021] Rework examples to use ParameterTool
> - [FLINK-3310] [runtime, runtime-web] Add back pressure statistics
> - [FLINK-3296] Remove 'flushing' behavior of the OutputFormat in DataStream
> API
> - [FLINK-3270] Add Kafka example
> - FLINK-2380: allow to specify the default filesystem scheme in the flink
> configuration file.
> - [FLINK-2237] [runtime] Add hash-based combiner.
> - [FLINK-3187] Introduce RestartStrategy to decouple restarting behaviour
> from ExecutionGraph
>
> I try to get these PRs in and push the owners of the blocking issues for
> resolutions.
>
>
>
> On Tue, Feb 2, 2016 at 7:12 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
> > Hi,
> >
> > I think that getting the stop signal in would be very nice.
> >
> > I would like to postpone the feature freeze till end of this week and
> > create the first RC on Monday. There are many open pull requests with
> fixes
> > that need to go in (stop signal, rocksdb state backend, interface
> > annotations, streaming api fixes)
> >
> >
> >
> > On Mon, Jan 25, 2016 at 9:02 AM, Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> >> Hi,
> >>
> >> I also would like to get the STOP signal in. But I do not have time to
> >> work in it this week... According to Till's comments, this will be the
> >> last round of reviewing required. So I should be able to finish it till
> >> 3rd Feb, but not sure.
> >>
> >> What do you think about it?
> >>
> >> -Matthias
> >>
> >> On 01/25/2016 04:29 PM, Aljoscha Krettek wrote:
> >> > Hi,
> >> > I think the refactoring of Partitioned State and the WindowOperator on
> >> state work is almost ready. I also have the RocksDB state backend
> working.
> >> I’m running some tests now on the cluster and should be able to open a
> PR
> >> tomorrow.
> >> >
> >> >
> >> >> On 25 Jan 2016, at 15:36, Stephan Ewen <se...@apache.org> wrote:
> >> >>
> >> >> I agree, with Gyula, one out-of-core state backend should be in. We
> are
> >> >> pretty close to that. Aljoscha has done good work on extending test
> >> >> coverage for state backends, so we should be pretty comfortable that
> it
> >> >> works as well, once we integrate new state backends with the tests.
> >> >>
> >> >> There is a bit of work do do around extending the interface of the
> >> >> key/value state. I would like to start a separate thread on that
> today
> >> or
> >> >> tomorrow...
> >> >>
> >> >>
> >> >>
> >> >> On Mon, Jan 25, 2016 at 12:16 PM, Gyula Fóra <gy...@apache.org>
> >> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> I agree that getting Flink 1.0.0 out soon would be great as Flink is
> >> in a
> >> >>> pretty solid state right now.
> >> >>>
> >> >>> I wonder whether it would make sense to include an out-of-core state
> >> >>> backend in streaming core that can be used with partitioned/window
> >> states.
> >> >>> I think if we are releasing 1.0.0 we should have a solid feature set
> >> for
> >> >>> our strong steaming use-cases (in this case stateful, and windowed
> >> >>> computations) and this should be a part of that.
> >> >>>
> >> >>> I know that Aljoscha is working on a solution for this which will
> >> probably
> >> >>> involve a heavy refactor of the State backend interfaces, and I am
> >> also
> >> >>> working on a similar solution. Maybe it would be good to get at
> least
> >> one
> >> >>> good robust solution for this in and definitely Aljoscha's refactor
> >> for the
> >> >>> interfaces.
> >> >>>
> >> >>> If we decide to do this, I think this needs 1-2 extra weeks of
> proper
> >> >>> testing so this might delay the schedule a little bit.
> >> >>>
> >> >>> What do you think?
> >> >>>
> >> >>> Gyula
> >> >>>
> >> >>>
> >> >>>
> >> >>> Robert Metzger <rm...@apache.org> ezt írta (időpont: 2016. jan.
> >> 25., H,
> >> >>> 11:54):
> >> >>>
> >> >>>> Hi,
> >> >>>>
> >> >>>> I would like to release 1.0.0 in the next weeks.
> >> >>>> Looking at the JIRAs, I think we are going to close a lot of
> blocking
> >> >>>> issues soon. How about we do a first release candidate on
> Wednesday,
> >> 3.
> >> >>>> February?
> >> >>>>
> >> >>>> The first release candidate is most likely not going to pass the
> >> vote,
> >> >>> the
> >> >>>> primary goal will be collecting a list of issues we need to
> address.
> >> >>>>
> >> >>>> There is also a Wiki page for the 1.0 release:
> >> >>>> https://cwiki.apache.org/confluence/display/FLINK/1.0+Release
> >> >>>>
> >> >>>> Please -1 to this message if 3. February is too soon for the first
> >> RC (it
> >> >>>> also means that we'll do a feature freeze around that time).
> >> >>>>
> >> >>>
> >> >
> >>
> >>
> >
>
Re: Release Flink 1.0.0
Posted by Robert Metzger <rm...@apache.org>.
There are still some 8 open blockers for the 1.0 release:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC
I also think that there are some pull requests which are almost ready to
merge and should go in:
- [FLINK-3341] Make 'auto.offset.reset' compatible with Kafka 0.8 and 0.9
- [FLINK-3336] Add Rescale Data Shipping for DataStream
- FLINK-2213 Makes the number of vcores per YARN container configurable
- [FLINK-2021] Rework examples to use ParameterTool
- [FLINK-3310] [runtime, runtime-web] Add back pressure statistics
- [FLINK-3296] Remove 'flushing' behavior of the OutputFormat in DataStream
API
- [FLINK-3270] Add Kafka example
- FLINK-2380: allow to specify the default filesystem scheme in the flink
configuration file.
- [FLINK-2237] [runtime] Add hash-based combiner.
- [FLINK-3187] Introduce RestartStrategy to decouple restarting behaviour
from ExecutionGraph
I try to get these PRs in and push the owners of the blocking issues for
resolutions.
On Tue, Feb 2, 2016 at 7:12 PM, Robert Metzger <rm...@apache.org> wrote:
> Hi,
>
> I think that getting the stop signal in would be very nice.
>
> I would like to postpone the feature freeze till end of this week and
> create the first RC on Monday. There are many open pull requests with fixes
> that need to go in (stop signal, rocksdb state backend, interface
> annotations, streaming api fixes)
>
>
>
> On Mon, Jan 25, 2016 at 9:02 AM, Matthias J. Sax <mj...@apache.org> wrote:
>
>> Hi,
>>
>> I also would like to get the STOP signal in. But I do not have time to
>> work in it this week... According to Till's comments, this will be the
>> last round of reviewing required. So I should be able to finish it till
>> 3rd Feb, but not sure.
>>
>> What do you think about it?
>>
>> -Matthias
>>
>> On 01/25/2016 04:29 PM, Aljoscha Krettek wrote:
>> > Hi,
>> > I think the refactoring of Partitioned State and the WindowOperator on
>> state work is almost ready. I also have the RocksDB state backend working.
>> I’m running some tests now on the cluster and should be able to open a PR
>> tomorrow.
>> >
>> >
>> >> On 25 Jan 2016, at 15:36, Stephan Ewen <se...@apache.org> wrote:
>> >>
>> >> I agree, with Gyula, one out-of-core state backend should be in. We are
>> >> pretty close to that. Aljoscha has done good work on extending test
>> >> coverage for state backends, so we should be pretty comfortable that it
>> >> works as well, once we integrate new state backends with the tests.
>> >>
>> >> There is a bit of work do do around extending the interface of the
>> >> key/value state. I would like to start a separate thread on that today
>> or
>> >> tomorrow...
>> >>
>> >>
>> >>
>> >> On Mon, Jan 25, 2016 at 12:16 PM, Gyula Fóra <gy...@apache.org>
>> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> I agree that getting Flink 1.0.0 out soon would be great as Flink is
>> in a
>> >>> pretty solid state right now.
>> >>>
>> >>> I wonder whether it would make sense to include an out-of-core state
>> >>> backend in streaming core that can be used with partitioned/window
>> states.
>> >>> I think if we are releasing 1.0.0 we should have a solid feature set
>> for
>> >>> our strong steaming use-cases (in this case stateful, and windowed
>> >>> computations) and this should be a part of that.
>> >>>
>> >>> I know that Aljoscha is working on a solution for this which will
>> probably
>> >>> involve a heavy refactor of the State backend interfaces, and I am
>> also
>> >>> working on a similar solution. Maybe it would be good to get at least
>> one
>> >>> good robust solution for this in and definitely Aljoscha's refactor
>> for the
>> >>> interfaces.
>> >>>
>> >>> If we decide to do this, I think this needs 1-2 extra weeks of proper
>> >>> testing so this might delay the schedule a little bit.
>> >>>
>> >>> What do you think?
>> >>>
>> >>> Gyula
>> >>>
>> >>>
>> >>>
>> >>> Robert Metzger <rm...@apache.org> ezt írta (időpont: 2016. jan.
>> 25., H,
>> >>> 11:54):
>> >>>
>> >>>> Hi,
>> >>>>
>> >>>> I would like to release 1.0.0 in the next weeks.
>> >>>> Looking at the JIRAs, I think we are going to close a lot of blocking
>> >>>> issues soon. How about we do a first release candidate on Wednesday,
>> 3.
>> >>>> February?
>> >>>>
>> >>>> The first release candidate is most likely not going to pass the
>> vote,
>> >>> the
>> >>>> primary goal will be collecting a list of issues we need to address.
>> >>>>
>> >>>> There is also a Wiki page for the 1.0 release:
>> >>>> https://cwiki.apache.org/confluence/display/FLINK/1.0+Release
>> >>>>
>> >>>> Please -1 to this message if 3. February is too soon for the first
>> RC (it
>> >>>> also means that we'll do a feature freeze around that time).
>> >>>>
>> >>>
>> >
>>
>>
>