[DISCUSS] Releasing Flink 1.1.0

[Robert Metzger <rm...@apache.org>]

Hi,

Flink 1.0.0 was released early March, so three months have passed and I
think we should start discussing the scope of the next major release
(1.1.0).

From a high level point of view, we've added the following new features:
in master:
- Table API Refactoring, SQL, StreamSQL
- The metrics system
- Kinesis Connector
- Persistent file sources for streaming

Half merged:
- Resource manager refactoring

Unmerged features:
- Cassandra connector
- Key groups ("rescaling from savepoints")
- Queryable state

I'm pretty sure I forgot many other features / pull requests, please post
them to this thread. I'll collect them and create a Wiki page out of it.

Some immediate TODOs for us:
- Which of the unmerged features are we going to add to the release?
- Which blockers do we need to address before releasing?
- Are there any volunteers for the release manager?


Regards,
Robert

Re: [DISCUSS] Releasing Flink 1.1.0

[Ufuk Celebi <uc...@apache.org>]

Great that we are all on the same page :-) Thanks for pointing out the
two issues Aljoscha and Till. I agree with you and I've updated them
to blockers ;-)

The FsStateBackend looks like it will be done soon. @Till: do you have
time to look into FLINK-4150 this week? I can also do it after I've
addressed your comments for the savepoint PRs today.

On Tue, Jul 5, 2016 at 9:55 AM, Till Rohrmann <tr...@apache.org> wrote:
> Stefan Richter found the following problem with HA:
> https://issues.apache.org/jira/browse/FLINK-4150
>
> I think we should fix it for the 1.1 release.
>
> On Mon, Jul 4, 2016 at 9:05 PM, Robert Metzger <rm...@apache.org> wrote:
>
>> +1 to do a RC0 this week, but the master-forking with RC1. I would like to
>> reduce the time we need to apply patches to multiple branches.
>>
>> @Aljoscha: I was running into the same issue on EMR when I used Flink w/
>> RocksDB recently, so I agree ;)
>>
>>
>>
>> On Mon, Jul 4, 2016 at 3:35 PM, Aljoscha Krettek <al...@apache.org>
>> wrote:
>>
>> > IMHO, the fix for this should also go in:
>> > https://issues.apache.org/jira/browse/FLINK-4115. This is blocking for
>> > users that want to use the RocksDB backend or FsStateBackend on Amazon
>> EMR
>> > with S3.
>> >
>> > There is already an open PR that I'm hoping to get in this week.
>> >
>> > On Mon, 4 Jul 2016 at 13:48 Ufuk Celebi <uc...@apache.org> wrote:
>> >
>> > > Thanks for the feedback. I would like to create a preview RC0 this
>> > > week like we did for the last releases, too. In past major releases,
>> > > we needed to create many release candidates, often for fixing just
>> > > some small issues. I would like to speed up the release process by
>> > > collecting as many issues as possible now with the RC0. Once these
>> > > issues are resolved, we can start voting with the RC1. This
>> > > essentially means that we have a feature freeze. I would create the
>> > > release-1.1 branch with RC1.
>> > >
>> > > Regarding the open issues:
>> > >
>> > > - The savepoint fixes are not yet in. There is a PR for the savepoint
>> > > headers (#2194) and the savepoint disposal PR needs addressing of
>> > > comments.
>> > >
>> > > - The Cassandra connector fixes are either merged or very close to be
>> > > merged.
>> > >
>> > > - Metrics docs are not a blocker since the online docs are updated
>> > > after the release. Regarding the renaming, we should decide soon. Any
>> > > opinions here?
>> > >
>> > > - The YARN issues have been resolved.
>> > >
>> > > I will also make a pass over JIRA and our PRs to check whether we've
>> > > missed something.
>> > >
>> > > @Greg: You are right, the hash-based combine PR has been extensively
>> > > reviewed. Unfortunately, I'm not familiar with the code as I didn't
>> > > look into it and cannot tell whether we should merge it now. Is the
>> > > hash-based combine strategy activated by default or does the user has
>> > > to activate it manually? The later case would make me feel more
>> > > comfortable merging it for the release.
>> > >
>> > >
>> > > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mx...@apache.org>
>> > wrote:
>> > > > Yet another one for the release: FLINK-4144
>> > > > https://github.com/apache/flink/pull/2191
>> > > >
>> > > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mx...@apache.org>
>> > > wrote:
>> > > >> There is also FLINK-4141. We just found this during testing. PR is
>> > > >> waiting to be merged here:
>> https://github.com/apache/flink/pull/2190
>> > > >>
>> > > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mx...@apache.org>
>> > > wrote:
>> > > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this
>> release
>> > > >>> and actually a bit tricky to fix. I've addressed the other issues
>> and
>> > > >>> merged all pending pull requests. Good to go from my side.
>> > > >>>
>> > > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <
>> > chesnay@apache.org>
>> > > wrote:
>> > > >>>> There are also 2 fixes for Cassandra that  should be included:
>> > > >>>> https://github.com/apache/flink/pull/2167
>> > > >>>> https://github.com/apache/flink/pull/2183
>> > > >>>>
>> > > >>>> We should also include the documentation for the metrics stuff
>> > > (hopefully
>> > > >>>> merged today)
>> > > >>>> https://github.com/apache/flink/pull/2158
>> > > >>>>
>> > > >>>> In regards to metrics: To add a counter metric a user currently
>> has
>> > > to call
>> > > >>>> "counter(...)" on
>> > > >>>> a MetricGroup. The point was raised in the documentation PR that
>> we
>> > > may want
>> > > >>>> to give
>> > > >>>> them a more descriptive name like "addCounter(...)".
>> > > >>>>
>> > > >>>> I would be in favor of changing them but would like others to
>> weigh
>> > > in on
>> > > >>>> this. IMO we
>> > > >>>> should nail this down before 1.1 .
>> > > >>>>
>> > > >>>> Regards,
>> > > >>>> Chesnay
>> > > >>>>
>> > > >>>>
>> > > >>>> On 30.06.2016 22:59, Greg Hogan wrote:
>> > > >>>>>
>> > > >>>>> It would be great if hash-based combine (FLINK-3477) could make
>> it
>> > > in to
>> > > >>>>> be
>> > > >>>>> tested for this release. We've seen impressive improvements in
>> > > performance
>> > > >>>>> (though, admittedly, some sort-based enhancements are yet to be
>> > > worked
>> > > >>>>> on).
>> > > >>>>> This PR looks to be ripe.
>> > > >>>>>
>> > > >>>>> Also, as we tidy up a few things with Gelly and documentation,
>> what
>> > > is the
>> > > >>>>> schedule for a feature freeze and creating a 1.1 branch off
>> master?
>> > > >>>>>
>> > > >>>>> Thanks,
>> > > >>>>> Greg
>> > > >>>>>
>> > > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <
>> > rmetzger@apache.org
>> > > >
>> > > >>>>> wrote:
>> > > >>>>>
>> > > >>>>>> Sure Ufuk! Thanks a lot for taking care of the release
>> management.
>> > > >>>>>> I'll be on vacation in three weeks, for three weeks and I'm not
>> > > sure if
>> > > >>>>>> we
>> > > >>>>>> get the release done until then.
>> > > >>>>>>
>> > > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org>
>> > > wrote:
>> > > >>>>>>
>> > > >>>>>>> I would like to do it if that's OK with you Robert. I would
>> > follow
>> > > >>>>>>> your suggestion and wait a few days until the following
>> important
>> > > >>>>>>> fixes are in:
>> > > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
>> > > >>>>>>> https://github.com/apache/flink/pull/2083)
>> > > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
>> > > >>>>>>> - Table API time support (
>> > > https://github.com/apache/flink/pull/2150)
>> > > >>>>>>> - Kafka at-least-once Producer (
>> > > >>>>>>
>> > > >>>>>> https://github.com/apache/flink/pull/2108)
>> > > >>>>>>>
>> > > >>>>>>> - Cassandra connector fixes (
>> > > https://github.com/apache/flink/pull/2163)
>> > > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there
>> > > something
>> > > >>>>>>
>> > > >>>>>> else?)
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <
>> > > rmetzger@apache.org>
>> > > >>>>>>> wrote:
>> > > >>>>>>>>
>> > > >>>>>>>> Hi,
>> > > >>>>>>>> it doesn't seem that there are volunteers for the RM, so I'll
>> > > probably
>> > > >>>>>>
>> > > >>>>>> do
>> > > >>>>>>>>
>> > > >>>>>>>> it.
>> > > >>>>>>>>
>> > > >>>>>>>> I try to do the first release candidate (mostly for testing)
>> > next
>> > > week
>> > > >>>>>>>
>> > > >>>>>>> (it
>> > > >>>>>>>>
>> > > >>>>>>>> depends on the JIRAs fixed by then)
>> > > >>>>>>>>
>> > > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
>> > > >>>>>>
>> > > >>>>>> henry.saputra@gmail.com
>> > > >>>>>>>>
>> > > >>>>>>>> wrote:
>> > > >>>>>>>>
>> > > >>>>>>>>> Thanks for the reply, @Max. I was not aware it was about
>> > dynamic
>> > > >>>>>>>
>> > > >>>>>>> scaling,
>> > > >>>>>>>>>
>> > > >>>>>>>>> which I think also asked for YARN support.
>> > > >>>>>>>>> I agree to list all related half merge JIRA for the
>> > > ResourceManager.
>> > > >>>>>>>>>
>> > > >>>>>>>>> Looking forward for the Apache Mesos integration design for
>> > sure
>> > > =)
>> > > >>>>>>>>>
>> > > >>>>>>>>> - Henry
>> > > >>>>>>>>>
>> > > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <
>> > > mxm@apache.org>
>> > > >>>>>>>>> wrote:
>> > > >>>>>>>>>
>> > > >>>>>>>>>> Hi Robert, hi Henry,
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features
>> that
>> > > >>>>>>
>> > > >>>>>> justify
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> a major release.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> @Henry We have plans to extend the ResourceManager to
>> interact
>> > > with
>> > > >>>>>>>>>> the Scheduler which will be a prerequisite for dynamic
>> > scaling.
>> > > I
>> > > >>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos
>> > > integration
>> > > >>>>>>>>>> won't require additional refactoring of the ResourceManager.
>> > > >>>>>>
>> > > >>>>>> Instead,
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> we will create a new "Dispatcher" component that takes care
>> of
>> > > >>>>>>>>>> bootstrapping the initial node with the
>> > > JobManager/ResourceManager.
>> > > >>>>>>>>>>  From there on, everything will be handled by the Mesos
>> > > >>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC)
>> who
>> > > came
>> > > >>>>>>
>> > > >>>>>> up
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> with this design and he plans to publish it to the mailing
>> > list
>> > > >>>>>>
>> > > >>>>>> soon.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged"
>> is
>> > > kind
>> > > >>>>>>
>> > > >>>>>> of
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> hard to get for people who are not involved in the different
>> > > >>>>>>>>>> components.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> The Cassandra adapter seems like a pretty important thing to
>> > > have
>> > > >>>>>>
>> > > >>>>>> for
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> the next release. In addition, I would like to merge
>> > FLINK-3667
>> > > and
>> > > >>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those
>> > are
>> > > a)
>> > > >>>>>>>>>> refactoring of the command-line and client classes b) adding
>> > > >>>>>>>>>> capability to resume cluster programmatically.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Then we should also have a look at any other critical/major
>> > bugs
>> > > >>>>>>>
>> > > >>>>>>> listed
>> > > >>>>>>>>>
>> > > >>>>>>>>> in
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> JIRA.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Cheers,
>> > > >>>>>>>>>> Max
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
>> > > >>>>>>>
>> > > >>>>>>> henry.saputra@gmail.com
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> wrote:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Hi Robert,
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Thanks for staying the discussion.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Do you know if there any open tasks for the Resource
>> Manager
>> > > left?
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> That is probably needed for Mesos integration?
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> - Henry
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
>> > > >>>>>>>
>> > > >>>>>>> rmetzger@apache.org>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> Hi,
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Flink 1.0.0 was released early March, so three months have
>> > > passed
>> > > >>>>>>>
>> > > >>>>>>> and
>> > > >>>>>>>>>
>> > > >>>>>>>>> I
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> think we should start discussing the scope of the next
>> major
>> > > >>>>>>>
>> > > >>>>>>> release
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> (1.1.0).
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>  From a high level point of view, we've added the
>> following
>> > > new
>> > > >>>>>>>>>
>> > > >>>>>>>>> features:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> in master:
>> > > >>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
>> > > >>>>>>>>>>>> - The metrics system
>> > > >>>>>>>>>>>> - Kinesis Connector
>> > > >>>>>>>>>>>> - Persistent file sources for streaming
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Half merged:
>> > > >>>>>>>>>>>> - Resource manager refactoring
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Unmerged features:
>> > > >>>>>>>>>>>> - Cassandra connector
>> > > >>>>>>>>>>>> - Key groups ("rescaling from savepoints")
>> > > >>>>>>>>>>>> - Queryable state
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> I'm pretty sure I forgot many other features / pull
>> > requests,
>> > > >>>>>>>
>> > > >>>>>>> please
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> post
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki
>> > page
>> > > out
>> > > >>>>>>>
>> > > >>>>>>> of
>> > > >>>>>>>>>
>> > > >>>>>>>>> it.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Some immediate TODOs for us:
>> > > >>>>>>>>>>>> - Which of the unmerged features are we going to add to
>> the
>> > > >>>>>>>
>> > > >>>>>>> release?
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> - Which blockers do we need to address before releasing?
>> > > >>>>>>>>>>>> - Are there any volunteers for the release manager?
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Regards,
>> > > >>>>>>>>>>>> Robert
>> > > >>>>>>>>>>>>
>> > > >>>>
>> > >
>> >
>>

Re: [DISCUSS] Releasing Flink 1.1.0

[Till Rohrmann <tr...@apache.org>]

Stefan Richter found the following problem with HA:
https://issues.apache.org/jira/browse/FLINK-4150

I think we should fix it for the 1.1 release.

On Mon, Jul 4, 2016 at 9:05 PM, Robert Metzger <rm...@apache.org> wrote:

> +1 to do a RC0 this week, but the master-forking with RC1. I would like to
> reduce the time we need to apply patches to multiple branches.
>
> @Aljoscha: I was running into the same issue on EMR when I used Flink w/
> RocksDB recently, so I agree ;)
>
>
>
> On Mon, Jul 4, 2016 at 3:35 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
> > IMHO, the fix for this should also go in:
> > https://issues.apache.org/jira/browse/FLINK-4115. This is blocking for
> > users that want to use the RocksDB backend or FsStateBackend on Amazon
> EMR
> > with S3.
> >
> > There is already an open PR that I'm hoping to get in this week.
> >
> > On Mon, 4 Jul 2016 at 13:48 Ufuk Celebi <uc...@apache.org> wrote:
> >
> > > Thanks for the feedback. I would like to create a preview RC0 this
> > > week like we did for the last releases, too. In past major releases,
> > > we needed to create many release candidates, often for fixing just
> > > some small issues. I would like to speed up the release process by
> > > collecting as many issues as possible now with the RC0. Once these
> > > issues are resolved, we can start voting with the RC1. This
> > > essentially means that we have a feature freeze. I would create the
> > > release-1.1 branch with RC1.
> > >
> > > Regarding the open issues:
> > >
> > > - The savepoint fixes are not yet in. There is a PR for the savepoint
> > > headers (#2194) and the savepoint disposal PR needs addressing of
> > > comments.
> > >
> > > - The Cassandra connector fixes are either merged or very close to be
> > > merged.
> > >
> > > - Metrics docs are not a blocker since the online docs are updated
> > > after the release. Regarding the renaming, we should decide soon. Any
> > > opinions here?
> > >
> > > - The YARN issues have been resolved.
> > >
> > > I will also make a pass over JIRA and our PRs to check whether we've
> > > missed something.
> > >
> > > @Greg: You are right, the hash-based combine PR has been extensively
> > > reviewed. Unfortunately, I'm not familiar with the code as I didn't
> > > look into it and cannot tell whether we should merge it now. Is the
> > > hash-based combine strategy activated by default or does the user has
> > > to activate it manually? The later case would make me feel more
> > > comfortable merging it for the release.
> > >
> > >
> > > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mx...@apache.org>
> > wrote:
> > > > Yet another one for the release: FLINK-4144
> > > > https://github.com/apache/flink/pull/2191
> > > >
> > > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mx...@apache.org>
> > > wrote:
> > > >> There is also FLINK-4141. We just found this during testing. PR is
> > > >> waiting to be merged here:
> https://github.com/apache/flink/pull/2190
> > > >>
> > > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mx...@apache.org>
> > > wrote:
> > > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this
> release
> > > >>> and actually a bit tricky to fix. I've addressed the other issues
> and
> > > >>> merged all pending pull requests. Good to go from my side.
> > > >>>
> > > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <
> > chesnay@apache.org>
> > > wrote:
> > > >>>> There are also 2 fixes for Cassandra that  should be included:
> > > >>>> https://github.com/apache/flink/pull/2167
> > > >>>> https://github.com/apache/flink/pull/2183
> > > >>>>
> > > >>>> We should also include the documentation for the metrics stuff
> > > (hopefully
> > > >>>> merged today)
> > > >>>> https://github.com/apache/flink/pull/2158
> > > >>>>
> > > >>>> In regards to metrics: To add a counter metric a user currently
> has
> > > to call
> > > >>>> "counter(...)" on
> > > >>>> a MetricGroup. The point was raised in the documentation PR that
> we
> > > may want
> > > >>>> to give
> > > >>>> them a more descriptive name like "addCounter(...)".
> > > >>>>
> > > >>>> I would be in favor of changing them but would like others to
> weigh
> > > in on
> > > >>>> this. IMO we
> > > >>>> should nail this down before 1.1 .
> > > >>>>
> > > >>>> Regards,
> > > >>>> Chesnay
> > > >>>>
> > > >>>>
> > > >>>> On 30.06.2016 22:59, Greg Hogan wrote:
> > > >>>>>
> > > >>>>> It would be great if hash-based combine (FLINK-3477) could make
> it
> > > in to
> > > >>>>> be
> > > >>>>> tested for this release. We've seen impressive improvements in
> > > performance
> > > >>>>> (though, admittedly, some sort-based enhancements are yet to be
> > > worked
> > > >>>>> on).
> > > >>>>> This PR looks to be ripe.
> > > >>>>>
> > > >>>>> Also, as we tidy up a few things with Gelly and documentation,
> what
> > > is the
> > > >>>>> schedule for a feature freeze and creating a 1.1 branch off
> master?
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Greg
> > > >>>>>
> > > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <
> > rmetzger@apache.org
> > > >
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Sure Ufuk! Thanks a lot for taking care of the release
> management.
> > > >>>>>> I'll be on vacation in three weeks, for three weeks and I'm not
> > > sure if
> > > >>>>>> we
> > > >>>>>> get the release done until then.
> > > >>>>>>
> > > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org>
> > > wrote:
> > > >>>>>>
> > > >>>>>>> I would like to do it if that's OK with you Robert. I would
> > follow
> > > >>>>>>> your suggestion and wait a few days until the following
> important
> > > >>>>>>> fixes are in:
> > > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
> > > >>>>>>> https://github.com/apache/flink/pull/2083)
> > > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
> > > >>>>>>> - Table API time support (
> > > https://github.com/apache/flink/pull/2150)
> > > >>>>>>> - Kafka at-least-once Producer (
> > > >>>>>>
> > > >>>>>> https://github.com/apache/flink/pull/2108)
> > > >>>>>>>
> > > >>>>>>> - Cassandra connector fixes (
> > > https://github.com/apache/flink/pull/2163)
> > > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there
> > > something
> > > >>>>>>
> > > >>>>>> else?)
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <
> > > rmetzger@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> Hi,
> > > >>>>>>>> it doesn't seem that there are volunteers for the RM, so I'll
> > > probably
> > > >>>>>>
> > > >>>>>> do
> > > >>>>>>>>
> > > >>>>>>>> it.
> > > >>>>>>>>
> > > >>>>>>>> I try to do the first release candidate (mostly for testing)
> > next
> > > week
> > > >>>>>>>
> > > >>>>>>> (it
> > > >>>>>>>>
> > > >>>>>>>> depends on the JIRAs fixed by then)
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
> > > >>>>>>
> > > >>>>>> henry.saputra@gmail.com
> > > >>>>>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Thanks for the reply, @Max. I was not aware it was about
> > dynamic
> > > >>>>>>>
> > > >>>>>>> scaling,
> > > >>>>>>>>>
> > > >>>>>>>>> which I think also asked for YARN support.
> > > >>>>>>>>> I agree to list all related half merge JIRA for the
> > > ResourceManager.
> > > >>>>>>>>>
> > > >>>>>>>>> Looking forward for the Apache Mesos integration design for
> > sure
> > > =)
> > > >>>>>>>>>
> > > >>>>>>>>> - Henry
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <
> > > mxm@apache.org>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi Robert, hi Henry,
> > > >>>>>>>>>>
> > > >>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features
> that
> > > >>>>>>
> > > >>>>>> justify
> > > >>>>>>>>>>
> > > >>>>>>>>>> a major release.
> > > >>>>>>>>>>
> > > >>>>>>>>>> @Henry We have plans to extend the ResourceManager to
> interact
> > > with
> > > >>>>>>>>>> the Scheduler which will be a prerequisite for dynamic
> > scaling.
> > > I
> > > >>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos
> > > integration
> > > >>>>>>>>>> won't require additional refactoring of the ResourceManager.
> > > >>>>>>
> > > >>>>>> Instead,
> > > >>>>>>>>>>
> > > >>>>>>>>>> we will create a new "Dispatcher" component that takes care
> of
> > > >>>>>>>>>> bootstrapping the initial node with the
> > > JobManager/ResourceManager.
> > > >>>>>>>>>>  From there on, everything will be handled by the Mesos
> > > >>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC)
> who
> > > came
> > > >>>>>>
> > > >>>>>> up
> > > >>>>>>>>>>
> > > >>>>>>>>>> with this design and he plans to publish it to the mailing
> > list
> > > >>>>>>
> > > >>>>>> soon.
> > > >>>>>>>>>>
> > > >>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged"
> is
> > > kind
> > > >>>>>>
> > > >>>>>> of
> > > >>>>>>>>>>
> > > >>>>>>>>>> hard to get for people who are not involved in the different
> > > >>>>>>>>>> components.
> > > >>>>>>>>>>
> > > >>>>>>>>>> The Cassandra adapter seems like a pretty important thing to
> > > have
> > > >>>>>>
> > > >>>>>> for
> > > >>>>>>>>>>
> > > >>>>>>>>>> the next release. In addition, I would like to merge
> > FLINK-3667
> > > and
> > > >>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those
> > are
> > > a)
> > > >>>>>>>>>> refactoring of the command-line and client classes b) adding
> > > >>>>>>>>>> capability to resume cluster programmatically.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Then we should also have a look at any other critical/major
> > bugs
> > > >>>>>>>
> > > >>>>>>> listed
> > > >>>>>>>>>
> > > >>>>>>>>> in
> > > >>>>>>>>>>
> > > >>>>>>>>>> JIRA.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Cheers,
> > > >>>>>>>>>> Max
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> > > >>>>>>>
> > > >>>>>>> henry.saputra@gmail.com
> > > >>>>>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Hi Robert,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks for staying the discussion.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Do you know if there any open tasks for the Resource
> Manager
> > > left?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> That is probably needed for Mesos integration?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Henry
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> > > >>>>>>>
> > > >>>>>>> rmetzger@apache.org>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Flink 1.0.0 was released early March, so three months have
> > > passed
> > > >>>>>>>
> > > >>>>>>> and
> > > >>>>>>>>>
> > > >>>>>>>>> I
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> think we should start discussing the scope of the next
> major
> > > >>>>>>>
> > > >>>>>>> release
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> (1.1.0).
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>  From a high level point of view, we've added the
> following
> > > new
> > > >>>>>>>>>
> > > >>>>>>>>> features:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> in master:
> > > >>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
> > > >>>>>>>>>>>> - The metrics system
> > > >>>>>>>>>>>> - Kinesis Connector
> > > >>>>>>>>>>>> - Persistent file sources for streaming
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Half merged:
> > > >>>>>>>>>>>> - Resource manager refactoring
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Unmerged features:
> > > >>>>>>>>>>>> - Cassandra connector
> > > >>>>>>>>>>>> - Key groups ("rescaling from savepoints")
> > > >>>>>>>>>>>> - Queryable state
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I'm pretty sure I forgot many other features / pull
> > requests,
> > > >>>>>>>
> > > >>>>>>> please
> > > >>>>>>>>>>
> > > >>>>>>>>>> post
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki
> > page
> > > out
> > > >>>>>>>
> > > >>>>>>> of
> > > >>>>>>>>>
> > > >>>>>>>>> it.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Some immediate TODOs for us:
> > > >>>>>>>>>>>> - Which of the unmerged features are we going to add to
> the
> > > >>>>>>>
> > > >>>>>>> release?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> - Which blockers do we need to address before releasing?
> > > >>>>>>>>>>>> - Are there any volunteers for the release manager?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>> Robert
> > > >>>>>>>>>>>>
> > > >>>>
> > >
> >
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Robert Metzger <rm...@apache.org>]

+1 to do a RC0 this week, but the master-forking with RC1. I would like to
reduce the time we need to apply patches to multiple branches.

@Aljoscha: I was running into the same issue on EMR when I used Flink w/
RocksDB recently, so I agree ;)



On Mon, Jul 4, 2016 at 3:35 PM, Aljoscha Krettek <al...@apache.org>
wrote:

> IMHO, the fix for this should also go in:
> https://issues.apache.org/jira/browse/FLINK-4115. This is blocking for
> users that want to use the RocksDB backend or FsStateBackend on Amazon EMR
> with S3.
>
> There is already an open PR that I'm hoping to get in this week.
>
> On Mon, 4 Jul 2016 at 13:48 Ufuk Celebi <uc...@apache.org> wrote:
>
> > Thanks for the feedback. I would like to create a preview RC0 this
> > week like we did for the last releases, too. In past major releases,
> > we needed to create many release candidates, often for fixing just
> > some small issues. I would like to speed up the release process by
> > collecting as many issues as possible now with the RC0. Once these
> > issues are resolved, we can start voting with the RC1. This
> > essentially means that we have a feature freeze. I would create the
> > release-1.1 branch with RC1.
> >
> > Regarding the open issues:
> >
> > - The savepoint fixes are not yet in. There is a PR for the savepoint
> > headers (#2194) and the savepoint disposal PR needs addressing of
> > comments.
> >
> > - The Cassandra connector fixes are either merged or very close to be
> > merged.
> >
> > - Metrics docs are not a blocker since the online docs are updated
> > after the release. Regarding the renaming, we should decide soon. Any
> > opinions here?
> >
> > - The YARN issues have been resolved.
> >
> > I will also make a pass over JIRA and our PRs to check whether we've
> > missed something.
> >
> > @Greg: You are right, the hash-based combine PR has been extensively
> > reviewed. Unfortunately, I'm not familiar with the code as I didn't
> > look into it and cannot tell whether we should merge it now. Is the
> > hash-based combine strategy activated by default or does the user has
> > to activate it manually? The later case would make me feel more
> > comfortable merging it for the release.
> >
> >
> > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mx...@apache.org>
> wrote:
> > > Yet another one for the release: FLINK-4144
> > > https://github.com/apache/flink/pull/2191
> > >
> > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mx...@apache.org>
> > wrote:
> > >> There is also FLINK-4141. We just found this during testing. PR is
> > >> waiting to be merged here: https://github.com/apache/flink/pull/2190
> > >>
> > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mx...@apache.org>
> > wrote:
> > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this release
> > >>> and actually a bit tricky to fix. I've addressed the other issues and
> > >>> merged all pending pull requests. Good to go from my side.
> > >>>
> > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <
> chesnay@apache.org>
> > wrote:
> > >>>> There are also 2 fixes for Cassandra that  should be included:
> > >>>> https://github.com/apache/flink/pull/2167
> > >>>> https://github.com/apache/flink/pull/2183
> > >>>>
> > >>>> We should also include the documentation for the metrics stuff
> > (hopefully
> > >>>> merged today)
> > >>>> https://github.com/apache/flink/pull/2158
> > >>>>
> > >>>> In regards to metrics: To add a counter metric a user currently has
> > to call
> > >>>> "counter(...)" on
> > >>>> a MetricGroup. The point was raised in the documentation PR that we
> > may want
> > >>>> to give
> > >>>> them a more descriptive name like "addCounter(...)".
> > >>>>
> > >>>> I would be in favor of changing them but would like others to weigh
> > in on
> > >>>> this. IMO we
> > >>>> should nail this down before 1.1 .
> > >>>>
> > >>>> Regards,
> > >>>> Chesnay
> > >>>>
> > >>>>
> > >>>> On 30.06.2016 22:59, Greg Hogan wrote:
> > >>>>>
> > >>>>> It would be great if hash-based combine (FLINK-3477) could make it
> > in to
> > >>>>> be
> > >>>>> tested for this release. We've seen impressive improvements in
> > performance
> > >>>>> (though, admittedly, some sort-based enhancements are yet to be
> > worked
> > >>>>> on).
> > >>>>> This PR looks to be ripe.
> > >>>>>
> > >>>>> Also, as we tidy up a few things with Gelly and documentation, what
> > is the
> > >>>>> schedule for a feature freeze and creating a 1.1 branch off master?
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Greg
> > >>>>>
> > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <
> rmetzger@apache.org
> > >
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Sure Ufuk! Thanks a lot for taking care of the release management.
> > >>>>>> I'll be on vacation in three weeks, for three weeks and I'm not
> > sure if
> > >>>>>> we
> > >>>>>> get the release done until then.
> > >>>>>>
> > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org>
> > wrote:
> > >>>>>>
> > >>>>>>> I would like to do it if that's OK with you Robert. I would
> follow
> > >>>>>>> your suggestion and wait a few days until the following important
> > >>>>>>> fixes are in:
> > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
> > >>>>>>> https://github.com/apache/flink/pull/2083)
> > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
> > >>>>>>> - Table API time support (
> > https://github.com/apache/flink/pull/2150)
> > >>>>>>> - Kafka at-least-once Producer (
> > >>>>>>
> > >>>>>> https://github.com/apache/flink/pull/2108)
> > >>>>>>>
> > >>>>>>> - Cassandra connector fixes (
> > https://github.com/apache/flink/pull/2163)
> > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there
> > something
> > >>>>>>
> > >>>>>> else?)
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <
> > rmetzger@apache.org>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi,
> > >>>>>>>> it doesn't seem that there are volunteers for the RM, so I'll
> > probably
> > >>>>>>
> > >>>>>> do
> > >>>>>>>>
> > >>>>>>>> it.
> > >>>>>>>>
> > >>>>>>>> I try to do the first release candidate (mostly for testing)
> next
> > week
> > >>>>>>>
> > >>>>>>> (it
> > >>>>>>>>
> > >>>>>>>> depends on the JIRAs fixed by then)
> > >>>>>>>>
> > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
> > >>>>>>
> > >>>>>> henry.saputra@gmail.com
> > >>>>>>>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Thanks for the reply, @Max. I was not aware it was about
> dynamic
> > >>>>>>>
> > >>>>>>> scaling,
> > >>>>>>>>>
> > >>>>>>>>> which I think also asked for YARN support.
> > >>>>>>>>> I agree to list all related half merge JIRA for the
> > ResourceManager.
> > >>>>>>>>>
> > >>>>>>>>> Looking forward for the Apache Mesos integration design for
> sure
> > =)
> > >>>>>>>>>
> > >>>>>>>>> - Henry
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <
> > mxm@apache.org>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Robert, hi Henry,
> > >>>>>>>>>>
> > >>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features that
> > >>>>>>
> > >>>>>> justify
> > >>>>>>>>>>
> > >>>>>>>>>> a major release.
> > >>>>>>>>>>
> > >>>>>>>>>> @Henry We have plans to extend the ResourceManager to interact
> > with
> > >>>>>>>>>> the Scheduler which will be a prerequisite for dynamic
> scaling.
> > I
> > >>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos
> > integration
> > >>>>>>>>>> won't require additional refactoring of the ResourceManager.
> > >>>>>>
> > >>>>>> Instead,
> > >>>>>>>>>>
> > >>>>>>>>>> we will create a new "Dispatcher" component that takes care of
> > >>>>>>>>>> bootstrapping the initial node with the
> > JobManager/ResourceManager.
> > >>>>>>>>>>  From there on, everything will be handled by the Mesos
> > >>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC) who
> > came
> > >>>>>>
> > >>>>>> up
> > >>>>>>>>>>
> > >>>>>>>>>> with this design and he plans to publish it to the mailing
> list
> > >>>>>>
> > >>>>>> soon.
> > >>>>>>>>>>
> > >>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged" is
> > kind
> > >>>>>>
> > >>>>>> of
> > >>>>>>>>>>
> > >>>>>>>>>> hard to get for people who are not involved in the different
> > >>>>>>>>>> components.
> > >>>>>>>>>>
> > >>>>>>>>>> The Cassandra adapter seems like a pretty important thing to
> > have
> > >>>>>>
> > >>>>>> for
> > >>>>>>>>>>
> > >>>>>>>>>> the next release. In addition, I would like to merge
> FLINK-3667
> > and
> > >>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those
> are
> > a)
> > >>>>>>>>>> refactoring of the command-line and client classes b) adding
> > >>>>>>>>>> capability to resume cluster programmatically.
> > >>>>>>>>>>
> > >>>>>>>>>> Then we should also have a look at any other critical/major
> bugs
> > >>>>>>>
> > >>>>>>> listed
> > >>>>>>>>>
> > >>>>>>>>> in
> > >>>>>>>>>>
> > >>>>>>>>>> JIRA.
> > >>>>>>>>>>
> > >>>>>>>>>> Cheers,
> > >>>>>>>>>> Max
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> > >>>>>>>
> > >>>>>>> henry.saputra@gmail.com
> > >>>>>>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi Robert,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for staying the discussion.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Do you know if there any open tasks for the Resource Manager
> > left?
> > >>>>>>>>>>>
> > >>>>>>>>>>> That is probably needed for Mesos integration?
> > >>>>>>>>>>>
> > >>>>>>>>>>> - Henry
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> > >>>>>>>
> > >>>>>>> rmetzger@apache.org>
> > >>>>>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Flink 1.0.0 was released early March, so three months have
> > passed
> > >>>>>>>
> > >>>>>>> and
> > >>>>>>>>>
> > >>>>>>>>> I
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> think we should start discussing the scope of the next major
> > >>>>>>>
> > >>>>>>> release
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> (1.1.0).
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>  From a high level point of view, we've added the following
> > new
> > >>>>>>>>>
> > >>>>>>>>> features:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> in master:
> > >>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
> > >>>>>>>>>>>> - The metrics system
> > >>>>>>>>>>>> - Kinesis Connector
> > >>>>>>>>>>>> - Persistent file sources for streaming
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Half merged:
> > >>>>>>>>>>>> - Resource manager refactoring
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Unmerged features:
> > >>>>>>>>>>>> - Cassandra connector
> > >>>>>>>>>>>> - Key groups ("rescaling from savepoints")
> > >>>>>>>>>>>> - Queryable state
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I'm pretty sure I forgot many other features / pull
> requests,
> > >>>>>>>
> > >>>>>>> please
> > >>>>>>>>>>
> > >>>>>>>>>> post
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki
> page
> > out
> > >>>>>>>
> > >>>>>>> of
> > >>>>>>>>>
> > >>>>>>>>> it.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Some immediate TODOs for us:
> > >>>>>>>>>>>> - Which of the unmerged features are we going to add to the
> > >>>>>>>
> > >>>>>>> release?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> - Which blockers do we need to address before releasing?
> > >>>>>>>>>>>> - Are there any volunteers for the release manager?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Regards,
> > >>>>>>>>>>>> Robert
> > >>>>>>>>>>>>
> > >>>>
> >
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Aljoscha Krettek <al...@apache.org>]

IMHO, the fix for this should also go in:
https://issues.apache.org/jira/browse/FLINK-4115. This is blocking for
users that want to use the RocksDB backend or FsStateBackend on Amazon EMR
with S3.

There is already an open PR that I'm hoping to get in this week.

On Mon, 4 Jul 2016 at 13:48 Ufuk Celebi <uc...@apache.org> wrote:

> Thanks for the feedback. I would like to create a preview RC0 this
> week like we did for the last releases, too. In past major releases,
> we needed to create many release candidates, often for fixing just
> some small issues. I would like to speed up the release process by
> collecting as many issues as possible now with the RC0. Once these
> issues are resolved, we can start voting with the RC1. This
> essentially means that we have a feature freeze. I would create the
> release-1.1 branch with RC1.
>
> Regarding the open issues:
>
> - The savepoint fixes are not yet in. There is a PR for the savepoint
> headers (#2194) and the savepoint disposal PR needs addressing of
> comments.
>
> - The Cassandra connector fixes are either merged or very close to be
> merged.
>
> - Metrics docs are not a blocker since the online docs are updated
> after the release. Regarding the renaming, we should decide soon. Any
> opinions here?
>
> - The YARN issues have been resolved.
>
> I will also make a pass over JIRA and our PRs to check whether we've
> missed something.
>
> @Greg: You are right, the hash-based combine PR has been extensively
> reviewed. Unfortunately, I'm not familiar with the code as I didn't
> look into it and cannot tell whether we should merge it now. Is the
> hash-based combine strategy activated by default or does the user has
> to activate it manually? The later case would make me feel more
> comfortable merging it for the release.
>
>
> On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mx...@apache.org> wrote:
> > Yet another one for the release: FLINK-4144
> > https://github.com/apache/flink/pull/2191
> >
> > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mx...@apache.org>
> wrote:
> >> There is also FLINK-4141. We just found this during testing. PR is
> >> waiting to be merged here: https://github.com/apache/flink/pull/2190
> >>
> >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mx...@apache.org>
> wrote:
> >>> FLINK-3904 is not Yarn related. Not pressing to fix for this release
> >>> and actually a bit tricky to fix. I've addressed the other issues and
> >>> merged all pending pull requests. Good to go from my side.
> >>>
> >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <ch...@apache.org>
> wrote:
> >>>> There are also 2 fixes for Cassandra that  should be included:
> >>>> https://github.com/apache/flink/pull/2167
> >>>> https://github.com/apache/flink/pull/2183
> >>>>
> >>>> We should also include the documentation for the metrics stuff
> (hopefully
> >>>> merged today)
> >>>> https://github.com/apache/flink/pull/2158
> >>>>
> >>>> In regards to metrics: To add a counter metric a user currently has
> to call
> >>>> "counter(...)" on
> >>>> a MetricGroup. The point was raised in the documentation PR that we
> may want
> >>>> to give
> >>>> them a more descriptive name like "addCounter(...)".
> >>>>
> >>>> I would be in favor of changing them but would like others to weigh
> in on
> >>>> this. IMO we
> >>>> should nail this down before 1.1 .
> >>>>
> >>>> Regards,
> >>>> Chesnay
> >>>>
> >>>>
> >>>> On 30.06.2016 22:59, Greg Hogan wrote:
> >>>>>
> >>>>> It would be great if hash-based combine (FLINK-3477) could make it
> in to
> >>>>> be
> >>>>> tested for this release. We've seen impressive improvements in
> performance
> >>>>> (though, admittedly, some sort-based enhancements are yet to be
> worked
> >>>>> on).
> >>>>> This PR looks to be ripe.
> >>>>>
> >>>>> Also, as we tidy up a few things with Gelly and documentation, what
> is the
> >>>>> schedule for a feature freeze and creating a 1.1 branch off master?
> >>>>>
> >>>>> Thanks,
> >>>>> Greg
> >>>>>
> >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <rmetzger@apache.org
> >
> >>>>> wrote:
> >>>>>
> >>>>>> Sure Ufuk! Thanks a lot for taking care of the release management.
> >>>>>> I'll be on vacation in three weeks, for three weeks and I'm not
> sure if
> >>>>>> we
> >>>>>> get the release done until then.
> >>>>>>
> >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org>
> wrote:
> >>>>>>
> >>>>>>> I would like to do it if that's OK with you Robert. I would follow
> >>>>>>> your suggestion and wait a few days until the following important
> >>>>>>> fixes are in:
> >>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
> >>>>>>> https://github.com/apache/flink/pull/2083)
> >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
> >>>>>>> - Table API time support (
> https://github.com/apache/flink/pull/2150)
> >>>>>>> - Kafka at-least-once Producer (
> >>>>>>
> >>>>>> https://github.com/apache/flink/pull/2108)
> >>>>>>>
> >>>>>>> - Cassandra connector fixes (
> https://github.com/apache/flink/pull/2163)
> >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there
> something
> >>>>>>
> >>>>>> else?)
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <
> rmetzger@apache.org>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>> it doesn't seem that there are volunteers for the RM, so I'll
> probably
> >>>>>>
> >>>>>> do
> >>>>>>>>
> >>>>>>>> it.
> >>>>>>>>
> >>>>>>>> I try to do the first release candidate (mostly for testing) next
> week
> >>>>>>>
> >>>>>>> (it
> >>>>>>>>
> >>>>>>>> depends on the JIRAs fixed by then)
> >>>>>>>>
> >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
> >>>>>>
> >>>>>> henry.saputra@gmail.com
> >>>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Thanks for the reply, @Max. I was not aware it was about dynamic
> >>>>>>>
> >>>>>>> scaling,
> >>>>>>>>>
> >>>>>>>>> which I think also asked for YARN support.
> >>>>>>>>> I agree to list all related half merge JIRA for the
> ResourceManager.
> >>>>>>>>>
> >>>>>>>>> Looking forward for the Apache Mesos integration design for sure
> =)
> >>>>>>>>>
> >>>>>>>>> - Henry
> >>>>>>>>>
> >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <
> mxm@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Robert, hi Henry,
> >>>>>>>>>>
> >>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features that
> >>>>>>
> >>>>>> justify
> >>>>>>>>>>
> >>>>>>>>>> a major release.
> >>>>>>>>>>
> >>>>>>>>>> @Henry We have plans to extend the ResourceManager to interact
> with
> >>>>>>>>>> the Scheduler which will be a prerequisite for dynamic scaling.
> I
> >>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos
> integration
> >>>>>>>>>> won't require additional refactoring of the ResourceManager.
> >>>>>>
> >>>>>> Instead,
> >>>>>>>>>>
> >>>>>>>>>> we will create a new "Dispatcher" component that takes care of
> >>>>>>>>>> bootstrapping the initial node with the
> JobManager/ResourceManager.
> >>>>>>>>>>  From there on, everything will be handled by the Mesos
> >>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC) who
> came
> >>>>>>
> >>>>>> up
> >>>>>>>>>>
> >>>>>>>>>> with this design and he plans to publish it to the mailing list
> >>>>>>
> >>>>>> soon.
> >>>>>>>>>>
> >>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged" is
> kind
> >>>>>>
> >>>>>> of
> >>>>>>>>>>
> >>>>>>>>>> hard to get for people who are not involved in the different
> >>>>>>>>>> components.
> >>>>>>>>>>
> >>>>>>>>>> The Cassandra adapter seems like a pretty important thing to
> have
> >>>>>>
> >>>>>> for
> >>>>>>>>>>
> >>>>>>>>>> the next release. In addition, I would like to merge FLINK-3667
> and
> >>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those are
> a)
> >>>>>>>>>> refactoring of the command-line and client classes b) adding
> >>>>>>>>>> capability to resume cluster programmatically.
> >>>>>>>>>>
> >>>>>>>>>> Then we should also have a look at any other critical/major bugs
> >>>>>>>
> >>>>>>> listed
> >>>>>>>>>
> >>>>>>>>> in
> >>>>>>>>>>
> >>>>>>>>>> JIRA.
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> Max
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> >>>>>>>
> >>>>>>> henry.saputra@gmail.com
> >>>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Robert,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for staying the discussion.
> >>>>>>>>>>>
> >>>>>>>>>>> Do you know if there any open tasks for the Resource Manager
> left?
> >>>>>>>>>>>
> >>>>>>>>>>> That is probably needed for Mesos integration?
> >>>>>>>>>>>
> >>>>>>>>>>> - Henry
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> >>>>>>>
> >>>>>>> rmetzger@apache.org>
> >>>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Flink 1.0.0 was released early March, so three months have
> passed
> >>>>>>>
> >>>>>>> and
> >>>>>>>>>
> >>>>>>>>> I
> >>>>>>>>>>>>
> >>>>>>>>>>>> think we should start discussing the scope of the next major
> >>>>>>>
> >>>>>>> release
> >>>>>>>>>>>>
> >>>>>>>>>>>> (1.1.0).
> >>>>>>>>>>>>
> >>>>>>>>>>>>  From a high level point of view, we've added the following
> new
> >>>>>>>>>
> >>>>>>>>> features:
> >>>>>>>>>>>>
> >>>>>>>>>>>> in master:
> >>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
> >>>>>>>>>>>> - The metrics system
> >>>>>>>>>>>> - Kinesis Connector
> >>>>>>>>>>>> - Persistent file sources for streaming
> >>>>>>>>>>>>
> >>>>>>>>>>>> Half merged:
> >>>>>>>>>>>> - Resource manager refactoring
> >>>>>>>>>>>>
> >>>>>>>>>>>> Unmerged features:
> >>>>>>>>>>>> - Cassandra connector
> >>>>>>>>>>>> - Key groups ("rescaling from savepoints")
> >>>>>>>>>>>> - Queryable state
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm pretty sure I forgot many other features / pull requests,
> >>>>>>>
> >>>>>>> please
> >>>>>>>>>>
> >>>>>>>>>> post
> >>>>>>>>>>>>
> >>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki page
> out
> >>>>>>>
> >>>>>>> of
> >>>>>>>>>
> >>>>>>>>> it.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Some immediate TODOs for us:
> >>>>>>>>>>>> - Which of the unmerged features are we going to add to the
> >>>>>>>
> >>>>>>> release?
> >>>>>>>>>>>>
> >>>>>>>>>>>> - Which blockers do we need to address before releasing?
> >>>>>>>>>>>> - Are there any volunteers for the release manager?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Regards,
> >>>>>>>>>>>> Robert
> >>>>>>>>>>>>
> >>>>
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Aljoscha Krettek <al...@apache.org>]

Btw, another blocking issue, IMHO:
https://issues.apache.org/jira/browse/FLINK-4149

I'm working on a fix.


On Tue, 5 Jul 2016 at 17:08 Till Rohrmann <tr...@apache.org> wrote:

> I found another critical issue [1]. The murmur hash correction introduced
> between Flink 1.0 and 1.1 breaks the backwards compatibility with respect
> to savepoints. I think we have to fix this for the release.
>
> @Ufuk, I'm not sure whether I find time this week to work on FLINK-4150. I
> could make it a stretch goal for this sprint. Or we schedule it in next
> week's sprint.
>
> [1] https://issues.apache.org/jira/browse/FLINK-4154
>
>
>
> On Tue, Jul 5, 2016 at 3:56 PM, Greg Hogan <co...@greghogan.com> wrote:
>
> > Hi Ufuk,
> >
> > The old sort-based combine is still the default. The user calls
> > .setCombineHint(CombineHint) to make a selection (I think this was
> > originally overloaded on DataSet and it looks like the pr1517
> documentation
> > update does not reflect the new usage).
> >
> > I'd be glad to merge this in but I didn't want to preempt Gabor or
> Fabian.
> >
> > Greg
> >
> > On Mon, Jul 4, 2016 at 7:47 AM, Ufuk Celebi <uc...@apache.org> wrote:
> >
> > > Thanks for the feedback. I would like to create a preview RC0 this
> > > week like we did for the last releases, too. In past major releases,
> > > we needed to create many release candidates, often for fixing just
> > > some small issues. I would like to speed up the release process by
> > > collecting as many issues as possible now with the RC0. Once these
> > > issues are resolved, we can start voting with the RC1. This
> > > essentially means that we have a feature freeze. I would create the
> > > release-1.1 branch with RC1.
> > >
> > > Regarding the open issues:
> > >
> > > - The savepoint fixes are not yet in. There is a PR for the savepoint
> > > headers (#2194) and the savepoint disposal PR needs addressing of
> > > comments.
> > >
> > > - The Cassandra connector fixes are either merged or very close to be
> > > merged.
> > >
> > > - Metrics docs are not a blocker since the online docs are updated
> > > after the release. Regarding the renaming, we should decide soon. Any
> > > opinions here?
> > >
> > > - The YARN issues have been resolved.
> > >
> > > I will also make a pass over JIRA and our PRs to check whether we've
> > > missed something.
> > >
> > > @Greg: You are right, the hash-based combine PR has been extensively
> > > reviewed. Unfortunately, I'm not familiar with the code as I didn't
> > > look into it and cannot tell whether we should merge it now. Is the
> > > hash-based combine strategy activated by default or does the user has
> > > to activate it manually? The later case would make me feel more
> > > comfortable merging it for the release.
> > >
> > >
> > > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mx...@apache.org>
> > wrote:
> > > > Yet another one for the release: FLINK-4144
> > > > https://github.com/apache/flink/pull/2191
> > > >
> > > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mx...@apache.org>
> > > wrote:
> > > >> There is also FLINK-4141. We just found this during testing. PR is
> > > >> waiting to be merged here:
> https://github.com/apache/flink/pull/2190
> > > >>
> > > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mx...@apache.org>
> > > wrote:
> > > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this
> release
> > > >>> and actually a bit tricky to fix. I've addressed the other issues
> and
> > > >>> merged all pending pull requests. Good to go from my side.
> > > >>>
> > > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <
> > chesnay@apache.org>
> > > wrote:
> > > >>>> There are also 2 fixes for Cassandra that  should be included:
> > > >>>> https://github.com/apache/flink/pull/2167
> > > >>>> https://github.com/apache/flink/pull/2183
> > > >>>>
> > > >>>> We should also include the documentation for the metrics stuff
> > > (hopefully
> > > >>>> merged today)
> > > >>>> https://github.com/apache/flink/pull/2158
> > > >>>>
> > > >>>> In regards to metrics: To add a counter metric a user currently
> has
> > > to call
> > > >>>> "counter(...)" on
> > > >>>> a MetricGroup. The point was raised in the documentation PR that
> we
> > > may want
> > > >>>> to give
> > > >>>> them a more descriptive name like "addCounter(...)".
> > > >>>>
> > > >>>> I would be in favor of changing them but would like others to
> weigh
> > > in on
> > > >>>> this. IMO we
> > > >>>> should nail this down before 1.1 .
> > > >>>>
> > > >>>> Regards,
> > > >>>> Chesnay
> > > >>>>
> > > >>>>
> > > >>>> On 30.06.2016 22:59, Greg Hogan wrote:
> > > >>>>>
> > > >>>>> It would be great if hash-based combine (FLINK-3477) could make
> it
> > > in to
> > > >>>>> be
> > > >>>>> tested for this release. We've seen impressive improvements in
> > > performance
> > > >>>>> (though, admittedly, some sort-based enhancements are yet to be
> > > worked
> > > >>>>> on).
> > > >>>>> This PR looks to be ripe.
> > > >>>>>
> > > >>>>> Also, as we tidy up a few things with Gelly and documentation,
> what
> > > is the
> > > >>>>> schedule for a feature freeze and creating a 1.1 branch off
> master?
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Greg
> > > >>>>>
> > > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <
> > rmetzger@apache.org
> > > >
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Sure Ufuk! Thanks a lot for taking care of the release
> management.
> > > >>>>>> I'll be on vacation in three weeks, for three weeks and I'm not
> > > sure if
> > > >>>>>> we
> > > >>>>>> get the release done until then.
> > > >>>>>>
> > > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org>
> > > wrote:
> > > >>>>>>
> > > >>>>>>> I would like to do it if that's OK with you Robert. I would
> > follow
> > > >>>>>>> your suggestion and wait a few days until the following
> important
> > > >>>>>>> fixes are in:
> > > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
> > > >>>>>>> https://github.com/apache/flink/pull/2083)
> > > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
> > > >>>>>>> - Table API time support (
> > > https://github.com/apache/flink/pull/2150)
> > > >>>>>>> - Kafka at-least-once Producer (
> > > >>>>>>
> > > >>>>>> https://github.com/apache/flink/pull/2108)
> > > >>>>>>>
> > > >>>>>>> - Cassandra connector fixes (
> > > https://github.com/apache/flink/pull/2163)
> > > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there
> > > something
> > > >>>>>>
> > > >>>>>> else?)
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <
> > > rmetzger@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> Hi,
> > > >>>>>>>> it doesn't seem that there are volunteers for the RM, so I'll
> > > probably
> > > >>>>>>
> > > >>>>>> do
> > > >>>>>>>>
> > > >>>>>>>> it.
> > > >>>>>>>>
> > > >>>>>>>> I try to do the first release candidate (mostly for testing)
> > next
> > > week
> > > >>>>>>>
> > > >>>>>>> (it
> > > >>>>>>>>
> > > >>>>>>>> depends on the JIRAs fixed by then)
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
> > > >>>>>>
> > > >>>>>> henry.saputra@gmail.com
> > > >>>>>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Thanks for the reply, @Max. I was not aware it was about
> > dynamic
> > > >>>>>>>
> > > >>>>>>> scaling,
> > > >>>>>>>>>
> > > >>>>>>>>> which I think also asked for YARN support.
> > > >>>>>>>>> I agree to list all related half merge JIRA for the
> > > ResourceManager.
> > > >>>>>>>>>
> > > >>>>>>>>> Looking forward for the Apache Mesos integration design for
> > sure
> > > =)
> > > >>>>>>>>>
> > > >>>>>>>>> - Henry
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <
> > > mxm@apache.org>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi Robert, hi Henry,
> > > >>>>>>>>>>
> > > >>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features
> that
> > > >>>>>>
> > > >>>>>> justify
> > > >>>>>>>>>>
> > > >>>>>>>>>> a major release.
> > > >>>>>>>>>>
> > > >>>>>>>>>> @Henry We have plans to extend the ResourceManager to
> interact
> > > with
> > > >>>>>>>>>> the Scheduler which will be a prerequisite for dynamic
> > scaling.
> > > I
> > > >>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos
> > > integration
> > > >>>>>>>>>> won't require additional refactoring of the ResourceManager.
> > > >>>>>>
> > > >>>>>> Instead,
> > > >>>>>>>>>>
> > > >>>>>>>>>> we will create a new "Dispatcher" component that takes care
> of
> > > >>>>>>>>>> bootstrapping the initial node with the
> > > JobManager/ResourceManager.
> > > >>>>>>>>>>  From there on, everything will be handled by the Mesos
> > > >>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC)
> who
> > > came
> > > >>>>>>
> > > >>>>>> up
> > > >>>>>>>>>>
> > > >>>>>>>>>> with this design and he plans to publish it to the mailing
> > list
> > > >>>>>>
> > > >>>>>> soon.
> > > >>>>>>>>>>
> > > >>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged"
> is
> > > kind
> > > >>>>>>
> > > >>>>>> of
> > > >>>>>>>>>>
> > > >>>>>>>>>> hard to get for people who are not involved in the different
> > > >>>>>>>>>> components.
> > > >>>>>>>>>>
> > > >>>>>>>>>> The Cassandra adapter seems like a pretty important thing to
> > > have
> > > >>>>>>
> > > >>>>>> for
> > > >>>>>>>>>>
> > > >>>>>>>>>> the next release. In addition, I would like to merge
> > FLINK-3667
> > > and
> > > >>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those
> > are
> > > a)
> > > >>>>>>>>>> refactoring of the command-line and client classes b) adding
> > > >>>>>>>>>> capability to resume cluster programmatically.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Then we should also have a look at any other critical/major
> > bugs
> > > >>>>>>>
> > > >>>>>>> listed
> > > >>>>>>>>>
> > > >>>>>>>>> in
> > > >>>>>>>>>>
> > > >>>>>>>>>> JIRA.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Cheers,
> > > >>>>>>>>>> Max
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> > > >>>>>>>
> > > >>>>>>> henry.saputra@gmail.com
> > > >>>>>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Hi Robert,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks for staying the discussion.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Do you know if there any open tasks for the Resource
> Manager
> > > left?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> That is probably needed for Mesos integration?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Henry
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> > > >>>>>>>
> > > >>>>>>> rmetzger@apache.org>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Flink 1.0.0 was released early March, so three months have
> > > passed
> > > >>>>>>>
> > > >>>>>>> and
> > > >>>>>>>>>
> > > >>>>>>>>> I
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> think we should start discussing the scope of the next
> major
> > > >>>>>>>
> > > >>>>>>> release
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> (1.1.0).
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>  From a high level point of view, we've added the
> following
> > > new
> > > >>>>>>>>>
> > > >>>>>>>>> features:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> in master:
> > > >>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
> > > >>>>>>>>>>>> - The metrics system
> > > >>>>>>>>>>>> - Kinesis Connector
> > > >>>>>>>>>>>> - Persistent file sources for streaming
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Half merged:
> > > >>>>>>>>>>>> - Resource manager refactoring
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Unmerged features:
> > > >>>>>>>>>>>> - Cassandra connector
> > > >>>>>>>>>>>> - Key groups ("rescaling from savepoints")
> > > >>>>>>>>>>>> - Queryable state
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I'm pretty sure I forgot many other features / pull
> > requests,
> > > >>>>>>>
> > > >>>>>>> please
> > > >>>>>>>>>>
> > > >>>>>>>>>> post
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki
> > page
> > > out
> > > >>>>>>>
> > > >>>>>>> of
> > > >>>>>>>>>
> > > >>>>>>>>> it.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Some immediate TODOs for us:
> > > >>>>>>>>>>>> - Which of the unmerged features are we going to add to
> the
> > > >>>>>>>
> > > >>>>>>> release?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> - Which blockers do we need to address before releasing?
> > > >>>>>>>>>>>> - Are there any volunteers for the release manager?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>> Robert
> > > >>>>>>>>>>>>
> > > >>>>
> > >
> >
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Ufuk Celebi <uc...@apache.org>]

I've now created a preview RC0 in a separate thread. As soon as we
resolve the remaining open issues and anything we the community
discovers with RC0, we can start with the first official RC1.

– Ufuk


On Tue, Jul 5, 2016 at 6:18 PM, Flavio Pompermaier <po...@okkam.it> wrote:
> it's just because Fabian said that it's better to not mix java and scala
> (as you can see in the comments of that PR)
> On 5 Jul 2016 18:53, "Aljoscha Krettek" <al...@apache.org> wrote:
>
>> I think it's not strictly required that all code be in Scala. There is
>> already some Java code in there so we shouldn't force people to write Scala
>> code if they make a valuable contribution in Java.
>>
>> On Tue, 5 Jul 2016 at 17:33 Flavio Pompermaier <po...@okkam.it>
>> wrote:
>>
>> > Hi to all,
>> > if Flink 1.1 will introduce ufficially the Table API, do you think
>> someone
>> > could take care of rewriting in scala the necessary java code of my PR
>> > about reading CSV as Rows instead of tuples[1]?
>> >
>> > For our use cases, and many new users approaching to Flink IMHO, that
>> will
>> > be definitely useful because you can read csv files with more than 25
>> > columns and properly handle null values. What  do you think?
>> >
>> > [1] https://github.com/apache/flink/pull/1989
>> >
>> > Best,
>> > Flavio
>> > On 5 Jul 2016 18:08, "Till Rohrmann" <tr...@apache.org> wrote:
>> >
>> > I found another critical issue [1]. The murmur hash correction introduced
>> > between Flink 1.0 and 1.1 breaks the backwards compatibility with respect
>> > to savepoints. I think we have to fix this for the release.
>> >
>> > @Ufuk, I'm not sure whether I find time this week to work on FLINK-4150.
>> I
>> > could make it a stretch goal for this sprint. Or we schedule it in next
>> > week's sprint.
>> >
>> > [1] https://issues.apache.org/jira/browse/FLINK-4154
>> >
>> >
>> >
>> > On Tue, Jul 5, 2016 at 3:56 PM, Greg Hogan <co...@greghogan.com> wrote:
>> >
>> > > Hi Ufuk,
>> > >
>> > > The old sort-based combine is still the default. The user calls
>> > > .setCombineHint(CombineHint) to make a selection (I think this was
>> > > originally overloaded on DataSet and it looks like the pr1517
>> > documentation
>> > > update does not reflect the new usage).
>> > >
>> > > I'd be glad to merge this in but I didn't want to preempt Gabor or
>> > Fabian.
>> > >
>> > > Greg
>> > >
>> > > On Mon, Jul 4, 2016 at 7:47 AM, Ufuk Celebi <uc...@apache.org> wrote:
>> > >
>> > > > Thanks for the feedback. I would like to create a preview RC0 this
>> > > > week like we did for the last releases, too. In past major releases,
>> > > > we needed to create many release candidates, often for fixing just
>> > > > some small issues. I would like to speed up the release process by
>> > > > collecting as many issues as possible now with the RC0. Once these
>> > > > issues are resolved, we can start voting with the RC1. This
>> > > > essentially means that we have a feature freeze. I would create the
>> > > > release-1.1 branch with RC1.
>> > > >
>> > > > Regarding the open issues:
>> > > >
>> > > > - The savepoint fixes are not yet in. There is a PR for the savepoint
>> > > > headers (#2194) and the savepoint disposal PR needs addressing of
>> > > > comments.
>> > > >
>> > > > - The Cassandra connector fixes are either merged or very close to be
>> > > > merged.
>> > > >
>> > > > - Metrics docs are not a blocker since the online docs are updated
>> > > > after the release. Regarding the renaming, we should decide soon. Any
>> > > > opinions here?
>> > > >
>> > > > - The YARN issues have been resolved.
>> > > >
>> > > > I will also make a pass over JIRA and our PRs to check whether we've
>> > > > missed something.
>> > > >
>> > > > @Greg: You are right, the hash-based combine PR has been extensively
>> > > > reviewed. Unfortunately, I'm not familiar with the code as I didn't
>> > > > look into it and cannot tell whether we should merge it now. Is the
>> > > > hash-based combine strategy activated by default or does the user has
>> > > > to activate it manually? The later case would make me feel more
>> > > > comfortable merging it for the release.
>> > > >
>> > > >
>> > > > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mx...@apache.org>
>> > > wrote:
>> > > > > Yet another one for the release: FLINK-4144
>> > > > > https://github.com/apache/flink/pull/2191
>> > > > >
>> > > > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mxm@apache.org
>> >
>> > > > wrote:
>> > > > >> There is also FLINK-4141. We just found this during testing. PR is
>> > > > >> waiting to be merged here:
>> > https://github.com/apache/flink/pull/2190
>> > > > >>
>> > > > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <
>> mxm@apache.org>
>> > > > wrote:
>> > > > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this
>> > release
>> > > > >>> and actually a bit tricky to fix. I've addressed the other issues
>> > and
>> > > > >>> merged all pending pull requests. Good to go from my side.
>> > > > >>>
>> > > > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <
>> > > chesnay@apache.org>
>> > > > wrote:
>> > > > >>>> There are also 2 fixes for Cassandra that  should be included:
>> > > > >>>> https://github.com/apache/flink/pull/2167
>> > > > >>>> https://github.com/apache/flink/pull/2183
>> > > > >>>>
>> > > > >>>> We should also include the documentation for the metrics stuff
>> > > > (hopefully
>> > > > >>>> merged today)
>> > > > >>>> https://github.com/apache/flink/pull/2158
>> > > > >>>>
>> > > > >>>> In regards to metrics: To add a counter metric a user currently
>> > has
>> > > > to call
>> > > > >>>> "counter(...)" on
>> > > > >>>> a MetricGroup. The point was raised in the documentation PR that
>> > we
>> > > > may want
>> > > > >>>> to give
>> > > > >>>> them a more descriptive name like "addCounter(...)".
>> > > > >>>>
>> > > > >>>> I would be in favor of changing them but would like others to
>> > weigh
>> > > > in on
>> > > > >>>> this. IMO we
>> > > > >>>> should nail this down before 1.1 .
>> > > > >>>>
>> > > > >>>> Regards,
>> > > > >>>> Chesnay
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> On 30.06.2016 22:59, Greg Hogan wrote:
>> > > > >>>>>
>> > > > >>>>> It would be great if hash-based combine (FLINK-3477) could make
>> > it
>> > > > in to
>> > > > >>>>> be
>> > > > >>>>> tested for this release. We've seen impressive improvements in
>> > > > performance
>> > > > >>>>> (though, admittedly, some sort-based enhancements are yet to be
>> > > > worked
>> > > > >>>>> on).
>> > > > >>>>> This PR looks to be ripe.
>> > > > >>>>>
>> > > > >>>>> Also, as we tidy up a few things with Gelly and documentation,
>> > what
>> > > > is the
>> > > > >>>>> schedule for a feature freeze and creating a 1.1 branch off
>> > master?
>> > > > >>>>>
>> > > > >>>>> Thanks,
>> > > > >>>>> Greg
>> > > > >>>>>
>> > > > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <
>> > > rmetzger@apache.org
>> > > > >
>> > > > >>>>> wrote:
>> > > > >>>>>
>> > > > >>>>>> Sure Ufuk! Thanks a lot for taking care of the release
>> > management.
>> > > > >>>>>> I'll be on vacation in three weeks, for three weeks and I'm
>> not
>> > > > sure if
>> > > > >>>>>> we
>> > > > >>>>>> get the release done until then.
>> > > > >>>>>>
>> > > > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uce@apache.org
>> >
>> > > > wrote:
>> > > > >>>>>>
>> > > > >>>>>>> I would like to do it if that's OK with you Robert. I would
>> > > follow
>> > > > >>>>>>> your suggestion and wait a few days until the following
>> > important
>> > > > >>>>>>> fixes are in:
>> > > > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
>> > > > >>>>>>> https://github.com/apache/flink/pull/2083)
>> > > > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
>> > > > >>>>>>> - Table API time support (
>> > > > https://github.com/apache/flink/pull/2150)
>> > > > >>>>>>> - Kafka at-least-once Producer (
>> > > > >>>>>>
>> > > > >>>>>> https://github.com/apache/flink/pull/2108)
>> > > > >>>>>>>
>> > > > >>>>>>> - Cassandra connector fixes (
>> > > > https://github.com/apache/flink/pull/2163)
>> > > > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there
>> > > > something
>> > > > >>>>>>
>> > > > >>>>>> else?)
>> > > > >>>>>>>
>> > > > >>>>>>>
>> > > > >>>>>>>
>> > > > >>>>>>>
>> > > > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <
>> > > > rmetzger@apache.org>
>> > > > >>>>>>> wrote:
>> > > > >>>>>>>>
>> > > > >>>>>>>> Hi,
>> > > > >>>>>>>> it doesn't seem that there are volunteers for the RM, so
>> I'll
>> > > > probably
>> > > > >>>>>>
>> > > > >>>>>> do
>> > > > >>>>>>>>
>> > > > >>>>>>>> it.
>> > > > >>>>>>>>
>> > > > >>>>>>>> I try to do the first release candidate (mostly for testing)
>> > > next
>> > > > week
>> > > > >>>>>>>
>> > > > >>>>>>> (it
>> > > > >>>>>>>>
>> > > > >>>>>>>> depends on the JIRAs fixed by then)
>> > > > >>>>>>>>
>> > > > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
>> > > > >>>>>>
>> > > > >>>>>> henry.saputra@gmail.com
>> > > > >>>>>>>>
>> > > > >>>>>>>> wrote:
>> > > > >>>>>>>>
>> > > > >>>>>>>>> Thanks for the reply, @Max. I was not aware it was about
>> > > dynamic
>> > > > >>>>>>>
>> > > > >>>>>>> scaling,
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> which I think also asked for YARN support.
>> > > > >>>>>>>>> I agree to list all related half merge JIRA for the
>> > > > ResourceManager.
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> Looking forward for the Apache Mesos integration design for
>> > > sure
>> > > > =)
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> - Henry
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <
>> > > > mxm@apache.org>
>> > > > >>>>>>>>> wrote:
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> Hi Robert, hi Henry,
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features
>> > that
>> > > > >>>>>>
>> > > > >>>>>> justify
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> a major release.
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> @Henry We have plans to extend the ResourceManager to
>> > interact
>> > > > with
>> > > > >>>>>>>>>> the Scheduler which will be a prerequisite for dynamic
>> > > scaling.
>> > > > I
>> > > > >>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos
>> > > > integration
>> > > > >>>>>>>>>> won't require additional refactoring of the
>> ResourceManager.
>> > > > >>>>>>
>> > > > >>>>>> Instead,
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> we will create a new "Dispatcher" component that takes
>> care
>> > of
>> > > > >>>>>>>>>> bootstrapping the initial node with the
>> > > > JobManager/ResourceManager.
>> > > > >>>>>>>>>>  From there on, everything will be handled by the Mesos
>> > > > >>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC)
>> > who
>> > > > came
>> > > > >>>>>>
>> > > > >>>>>> up
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> with this design and he plans to publish it to the mailing
>> > > list
>> > > > >>>>>>
>> > > > >>>>>> soon.
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged"
>> > is
>> > > > kind
>> > > > >>>>>>
>> > > > >>>>>> of
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> hard to get for people who are not involved in the
>> different
>> > > > >>>>>>>>>> components.
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> The Cassandra adapter seems like a pretty important thing
>> to
>> > > > have
>> > > > >>>>>>
>> > > > >>>>>> for
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> the next release. In addition, I would like to merge
>> > > FLINK-3667
>> > > > and
>> > > > >>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :)
>> Those
>> > > are
>> > > > a)
>> > > > >>>>>>>>>> refactoring of the command-line and client classes b)
>> adding
>> > > > >>>>>>>>>> capability to resume cluster programmatically.
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> Then we should also have a look at any other
>> critical/major
>> > > bugs
>> > > > >>>>>>>
>> > > > >>>>>>> listed
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> in
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> JIRA.
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> Cheers,
>> > > > >>>>>>>>>> Max
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
>> > > > >>>>>>>
>> > > > >>>>>>> henry.saputra@gmail.com
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> wrote:
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Hi Robert,
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Thanks for staying the discussion.
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Do you know if there any open tasks for the Resource
>> > Manager
>> > > > left?
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> That is probably needed for Mesos integration?
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> - Henry
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
>> > > > >>>>>>>
>> > > > >>>>>>> rmetzger@apache.org>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> wrote:
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>>> Hi,
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> Flink 1.0.0 was released early March, so three months
>> have
>> > > > passed
>> > > > >>>>>>>
>> > > > >>>>>>> and
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> I
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> think we should start discussing the scope of the next
>> > major
>> > > > >>>>>>>
>> > > > >>>>>>> release
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> (1.1.0).
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>>  From a high level point of view, we've added the
>> > following
>> > > > new
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> features:
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> in master:
>> > > > >>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
>> > > > >>>>>>>>>>>> - The metrics system
>> > > > >>>>>>>>>>>> - Kinesis Connector
>> > > > >>>>>>>>>>>> - Persistent file sources for streaming
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> Half merged:
>> > > > >>>>>>>>>>>> - Resource manager refactoring
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> Unmerged features:
>> > > > >>>>>>>>>>>> - Cassandra connector
>> > > > >>>>>>>>>>>> - Key groups ("rescaling from savepoints")
>> > > > >>>>>>>>>>>> - Queryable state
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> I'm pretty sure I forgot many other features / pull
>> > > requests,
>> > > > >>>>>>>
>> > > > >>>>>>> please
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> post
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki
>> > > page
>> > > > out
>> > > > >>>>>>>
>> > > > >>>>>>> of
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> it.
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> Some immediate TODOs for us:
>> > > > >>>>>>>>>>>> - Which of the unmerged features are we going to add to
>> > the
>> > > > >>>>>>>
>> > > > >>>>>>> release?
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> - Which blockers do we need to address before releasing?
>> > > > >>>>>>>>>>>> - Are there any volunteers for the release manager?
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> Regards,
>> > > > >>>>>>>>>>>> Robert
>> > > > >>>>>>>>>>>>
>> > > > >>>>
>> > > >
>> > >
>> >
>>

Re: [DISCUSS] Releasing Flink 1.1.0

[Flavio Pompermaier <po...@okkam.it>]

it's just because Fabian said that it's better to not mix java and scala
(as you can see in the comments of that PR)
On 5 Jul 2016 18:53, "Aljoscha Krettek" <al...@apache.org> wrote:

> I think it's not strictly required that all code be in Scala. There is
> already some Java code in there so we shouldn't force people to write Scala
> code if they make a valuable contribution in Java.
>
> On Tue, 5 Jul 2016 at 17:33 Flavio Pompermaier <po...@okkam.it>
> wrote:
>
> > Hi to all,
> > if Flink 1.1 will introduce ufficially the Table API, do you think
> someone
> > could take care of rewriting in scala the necessary java code of my PR
> > about reading CSV as Rows instead of tuples[1]?
> >
> > For our use cases, and many new users approaching to Flink IMHO, that
> will
> > be definitely useful because you can read csv files with more than 25
> > columns and properly handle null values. What  do you think?
> >
> > [1] https://github.com/apache/flink/pull/1989
> >
> > Best,
> > Flavio
> > On 5 Jul 2016 18:08, "Till Rohrmann" <tr...@apache.org> wrote:
> >
> > I found another critical issue [1]. The murmur hash correction introduced
> > between Flink 1.0 and 1.1 breaks the backwards compatibility with respect
> > to savepoints. I think we have to fix this for the release.
> >
> > @Ufuk, I'm not sure whether I find time this week to work on FLINK-4150.
> I
> > could make it a stretch goal for this sprint. Or we schedule it in next
> > week's sprint.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-4154
> >
> >
> >
> > On Tue, Jul 5, 2016 at 3:56 PM, Greg Hogan <co...@greghogan.com> wrote:
> >
> > > Hi Ufuk,
> > >
> > > The old sort-based combine is still the default. The user calls
> > > .setCombineHint(CombineHint) to make a selection (I think this was
> > > originally overloaded on DataSet and it looks like the pr1517
> > documentation
> > > update does not reflect the new usage).
> > >
> > > I'd be glad to merge this in but I didn't want to preempt Gabor or
> > Fabian.
> > >
> > > Greg
> > >
> > > On Mon, Jul 4, 2016 at 7:47 AM, Ufuk Celebi <uc...@apache.org> wrote:
> > >
> > > > Thanks for the feedback. I would like to create a preview RC0 this
> > > > week like we did for the last releases, too. In past major releases,
> > > > we needed to create many release candidates, often for fixing just
> > > > some small issues. I would like to speed up the release process by
> > > > collecting as many issues as possible now with the RC0. Once these
> > > > issues are resolved, we can start voting with the RC1. This
> > > > essentially means that we have a feature freeze. I would create the
> > > > release-1.1 branch with RC1.
> > > >
> > > > Regarding the open issues:
> > > >
> > > > - The savepoint fixes are not yet in. There is a PR for the savepoint
> > > > headers (#2194) and the savepoint disposal PR needs addressing of
> > > > comments.
> > > >
> > > > - The Cassandra connector fixes are either merged or very close to be
> > > > merged.
> > > >
> > > > - Metrics docs are not a blocker since the online docs are updated
> > > > after the release. Regarding the renaming, we should decide soon. Any
> > > > opinions here?
> > > >
> > > > - The YARN issues have been resolved.
> > > >
> > > > I will also make a pass over JIRA and our PRs to check whether we've
> > > > missed something.
> > > >
> > > > @Greg: You are right, the hash-based combine PR has been extensively
> > > > reviewed. Unfortunately, I'm not familiar with the code as I didn't
> > > > look into it and cannot tell whether we should merge it now. Is the
> > > > hash-based combine strategy activated by default or does the user has
> > > > to activate it manually? The later case would make me feel more
> > > > comfortable merging it for the release.
> > > >
> > > >
> > > > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mx...@apache.org>
> > > wrote:
> > > > > Yet another one for the release: FLINK-4144
> > > > > https://github.com/apache/flink/pull/2191
> > > > >
> > > > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mxm@apache.org
> >
> > > > wrote:
> > > > >> There is also FLINK-4141. We just found this during testing. PR is
> > > > >> waiting to be merged here:
> > https://github.com/apache/flink/pull/2190
> > > > >>
> > > > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <
> mxm@apache.org>
> > > > wrote:
> > > > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this
> > release
> > > > >>> and actually a bit tricky to fix. I've addressed the other issues
> > and
> > > > >>> merged all pending pull requests. Good to go from my side.
> > > > >>>
> > > > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <
> > > chesnay@apache.org>
> > > > wrote:
> > > > >>>> There are also 2 fixes for Cassandra that  should be included:
> > > > >>>> https://github.com/apache/flink/pull/2167
> > > > >>>> https://github.com/apache/flink/pull/2183
> > > > >>>>
> > > > >>>> We should also include the documentation for the metrics stuff
> > > > (hopefully
> > > > >>>> merged today)
> > > > >>>> https://github.com/apache/flink/pull/2158
> > > > >>>>
> > > > >>>> In regards to metrics: To add a counter metric a user currently
> > has
> > > > to call
> > > > >>>> "counter(...)" on
> > > > >>>> a MetricGroup. The point was raised in the documentation PR that
> > we
> > > > may want
> > > > >>>> to give
> > > > >>>> them a more descriptive name like "addCounter(...)".
> > > > >>>>
> > > > >>>> I would be in favor of changing them but would like others to
> > weigh
> > > > in on
> > > > >>>> this. IMO we
> > > > >>>> should nail this down before 1.1 .
> > > > >>>>
> > > > >>>> Regards,
> > > > >>>> Chesnay
> > > > >>>>
> > > > >>>>
> > > > >>>> On 30.06.2016 22:59, Greg Hogan wrote:
> > > > >>>>>
> > > > >>>>> It would be great if hash-based combine (FLINK-3477) could make
> > it
> > > > in to
> > > > >>>>> be
> > > > >>>>> tested for this release. We've seen impressive improvements in
> > > > performance
> > > > >>>>> (though, admittedly, some sort-based enhancements are yet to be
> > > > worked
> > > > >>>>> on).
> > > > >>>>> This PR looks to be ripe.
> > > > >>>>>
> > > > >>>>> Also, as we tidy up a few things with Gelly and documentation,
> > what
> > > > is the
> > > > >>>>> schedule for a feature freeze and creating a 1.1 branch off
> > master?
> > > > >>>>>
> > > > >>>>> Thanks,
> > > > >>>>> Greg
> > > > >>>>>
> > > > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <
> > > rmetzger@apache.org
> > > > >
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Sure Ufuk! Thanks a lot for taking care of the release
> > management.
> > > > >>>>>> I'll be on vacation in three weeks, for three weeks and I'm
> not
> > > > sure if
> > > > >>>>>> we
> > > > >>>>>> get the release done until then.
> > > > >>>>>>
> > > > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uce@apache.org
> >
> > > > wrote:
> > > > >>>>>>
> > > > >>>>>>> I would like to do it if that's OK with you Robert. I would
> > > follow
> > > > >>>>>>> your suggestion and wait a few days until the following
> > important
> > > > >>>>>>> fixes are in:
> > > > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
> > > > >>>>>>> https://github.com/apache/flink/pull/2083)
> > > > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
> > > > >>>>>>> - Table API time support (
> > > > https://github.com/apache/flink/pull/2150)
> > > > >>>>>>> - Kafka at-least-once Producer (
> > > > >>>>>>
> > > > >>>>>> https://github.com/apache/flink/pull/2108)
> > > > >>>>>>>
> > > > >>>>>>> - Cassandra connector fixes (
> > > > https://github.com/apache/flink/pull/2163)
> > > > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there
> > > > something
> > > > >>>>>>
> > > > >>>>>> else?)
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <
> > > > rmetzger@apache.org>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> Hi,
> > > > >>>>>>>> it doesn't seem that there are volunteers for the RM, so
> I'll
> > > > probably
> > > > >>>>>>
> > > > >>>>>> do
> > > > >>>>>>>>
> > > > >>>>>>>> it.
> > > > >>>>>>>>
> > > > >>>>>>>> I try to do the first release candidate (mostly for testing)
> > > next
> > > > week
> > > > >>>>>>>
> > > > >>>>>>> (it
> > > > >>>>>>>>
> > > > >>>>>>>> depends on the JIRAs fixed by then)
> > > > >>>>>>>>
> > > > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
> > > > >>>>>>
> > > > >>>>>> henry.saputra@gmail.com
> > > > >>>>>>>>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Thanks for the reply, @Max. I was not aware it was about
> > > dynamic
> > > > >>>>>>>
> > > > >>>>>>> scaling,
> > > > >>>>>>>>>
> > > > >>>>>>>>> which I think also asked for YARN support.
> > > > >>>>>>>>> I agree to list all related half merge JIRA for the
> > > > ResourceManager.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Looking forward for the Apache Mesos integration design for
> > > sure
> > > > =)
> > > > >>>>>>>>>
> > > > >>>>>>>>> - Henry
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <
> > > > mxm@apache.org>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Hi Robert, hi Henry,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features
> > that
> > > > >>>>>>
> > > > >>>>>> justify
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> a major release.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> @Henry We have plans to extend the ResourceManager to
> > interact
> > > > with
> > > > >>>>>>>>>> the Scheduler which will be a prerequisite for dynamic
> > > scaling.
> > > > I
> > > > >>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos
> > > > integration
> > > > >>>>>>>>>> won't require additional refactoring of the
> ResourceManager.
> > > > >>>>>>
> > > > >>>>>> Instead,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> we will create a new "Dispatcher" component that takes
> care
> > of
> > > > >>>>>>>>>> bootstrapping the initial node with the
> > > > JobManager/ResourceManager.
> > > > >>>>>>>>>>  From there on, everything will be handled by the Mesos
> > > > >>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC)
> > who
> > > > came
> > > > >>>>>>
> > > > >>>>>> up
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> with this design and he plans to publish it to the mailing
> > > list
> > > > >>>>>>
> > > > >>>>>> soon.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged"
> > is
> > > > kind
> > > > >>>>>>
> > > > >>>>>> of
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> hard to get for people who are not involved in the
> different
> > > > >>>>>>>>>> components.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> The Cassandra adapter seems like a pretty important thing
> to
> > > > have
> > > > >>>>>>
> > > > >>>>>> for
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> the next release. In addition, I would like to merge
> > > FLINK-3667
> > > > and
> > > > >>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :)
> Those
> > > are
> > > > a)
> > > > >>>>>>>>>> refactoring of the command-line and client classes b)
> adding
> > > > >>>>>>>>>> capability to resume cluster programmatically.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Then we should also have a look at any other
> critical/major
> > > bugs
> > > > >>>>>>>
> > > > >>>>>>> listed
> > > > >>>>>>>>>
> > > > >>>>>>>>> in
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> JIRA.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Cheers,
> > > > >>>>>>>>>> Max
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> > > > >>>>>>>
> > > > >>>>>>> henry.saputra@gmail.com
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Hi Robert,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Thanks for staying the discussion.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Do you know if there any open tasks for the Resource
> > Manager
> > > > left?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> That is probably needed for Mesos integration?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> - Henry
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> > > > >>>>>>>
> > > > >>>>>>> rmetzger@apache.org>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Hi,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Flink 1.0.0 was released early March, so three months
> have
> > > > passed
> > > > >>>>>>>
> > > > >>>>>>> and
> > > > >>>>>>>>>
> > > > >>>>>>>>> I
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> think we should start discussing the scope of the next
> > major
> > > > >>>>>>>
> > > > >>>>>>> release
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> (1.1.0).
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>  From a high level point of view, we've added the
> > following
> > > > new
> > > > >>>>>>>>>
> > > > >>>>>>>>> features:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> in master:
> > > > >>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
> > > > >>>>>>>>>>>> - The metrics system
> > > > >>>>>>>>>>>> - Kinesis Connector
> > > > >>>>>>>>>>>> - Persistent file sources for streaming
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Half merged:
> > > > >>>>>>>>>>>> - Resource manager refactoring
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Unmerged features:
> > > > >>>>>>>>>>>> - Cassandra connector
> > > > >>>>>>>>>>>> - Key groups ("rescaling from savepoints")
> > > > >>>>>>>>>>>> - Queryable state
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I'm pretty sure I forgot many other features / pull
> > > requests,
> > > > >>>>>>>
> > > > >>>>>>> please
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> post
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki
> > > page
> > > > out
> > > > >>>>>>>
> > > > >>>>>>> of
> > > > >>>>>>>>>
> > > > >>>>>>>>> it.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Some immediate TODOs for us:
> > > > >>>>>>>>>>>> - Which of the unmerged features are we going to add to
> > the
> > > > >>>>>>>
> > > > >>>>>>> release?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> - Which blockers do we need to address before releasing?
> > > > >>>>>>>>>>>> - Are there any volunteers for the release manager?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Regards,
> > > > >>>>>>>>>>>> Robert
> > > > >>>>>>>>>>>>
> > > > >>>>
> > > >
> > >
> >
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Aljoscha Krettek <al...@apache.org>]

I think it's not strictly required that all code be in Scala. There is
already some Java code in there so we shouldn't force people to write Scala
code if they make a valuable contribution in Java.

On Tue, 5 Jul 2016 at 17:33 Flavio Pompermaier <po...@okkam.it> wrote:

> Hi to all,
> if Flink 1.1 will introduce ufficially the Table API, do you think someone
> could take care of rewriting in scala the necessary java code of my PR
> about reading CSV as Rows instead of tuples[1]?
>
> For our use cases, and many new users approaching to Flink IMHO, that will
> be definitely useful because you can read csv files with more than 25
> columns and properly handle null values. What  do you think?
>
> [1] https://github.com/apache/flink/pull/1989
>
> Best,
> Flavio
> On 5 Jul 2016 18:08, "Till Rohrmann" <tr...@apache.org> wrote:
>
> I found another critical issue [1]. The murmur hash correction introduced
> between Flink 1.0 and 1.1 breaks the backwards compatibility with respect
> to savepoints. I think we have to fix this for the release.
>
> @Ufuk, I'm not sure whether I find time this week to work on FLINK-4150. I
> could make it a stretch goal for this sprint. Or we schedule it in next
> week's sprint.
>
> [1] https://issues.apache.org/jira/browse/FLINK-4154
>
>
>
> On Tue, Jul 5, 2016 at 3:56 PM, Greg Hogan <co...@greghogan.com> wrote:
>
> > Hi Ufuk,
> >
> > The old sort-based combine is still the default. The user calls
> > .setCombineHint(CombineHint) to make a selection (I think this was
> > originally overloaded on DataSet and it looks like the pr1517
> documentation
> > update does not reflect the new usage).
> >
> > I'd be glad to merge this in but I didn't want to preempt Gabor or
> Fabian.
> >
> > Greg
> >
> > On Mon, Jul 4, 2016 at 7:47 AM, Ufuk Celebi <uc...@apache.org> wrote:
> >
> > > Thanks for the feedback. I would like to create a preview RC0 this
> > > week like we did for the last releases, too. In past major releases,
> > > we needed to create many release candidates, often for fixing just
> > > some small issues. I would like to speed up the release process by
> > > collecting as many issues as possible now with the RC0. Once these
> > > issues are resolved, we can start voting with the RC1. This
> > > essentially means that we have a feature freeze. I would create the
> > > release-1.1 branch with RC1.
> > >
> > > Regarding the open issues:
> > >
> > > - The savepoint fixes are not yet in. There is a PR for the savepoint
> > > headers (#2194) and the savepoint disposal PR needs addressing of
> > > comments.
> > >
> > > - The Cassandra connector fixes are either merged or very close to be
> > > merged.
> > >
> > > - Metrics docs are not a blocker since the online docs are updated
> > > after the release. Regarding the renaming, we should decide soon. Any
> > > opinions here?
> > >
> > > - The YARN issues have been resolved.
> > >
> > > I will also make a pass over JIRA and our PRs to check whether we've
> > > missed something.
> > >
> > > @Greg: You are right, the hash-based combine PR has been extensively
> > > reviewed. Unfortunately, I'm not familiar with the code as I didn't
> > > look into it and cannot tell whether we should merge it now. Is the
> > > hash-based combine strategy activated by default or does the user has
> > > to activate it manually? The later case would make me feel more
> > > comfortable merging it for the release.
> > >
> > >
> > > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mx...@apache.org>
> > wrote:
> > > > Yet another one for the release: FLINK-4144
> > > > https://github.com/apache/flink/pull/2191
> > > >
> > > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mx...@apache.org>
> > > wrote:
> > > >> There is also FLINK-4141. We just found this during testing. PR is
> > > >> waiting to be merged here:
> https://github.com/apache/flink/pull/2190
> > > >>
> > > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mx...@apache.org>
> > > wrote:
> > > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this
> release
> > > >>> and actually a bit tricky to fix. I've addressed the other issues
> and
> > > >>> merged all pending pull requests. Good to go from my side.
> > > >>>
> > > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <
> > chesnay@apache.org>
> > > wrote:
> > > >>>> There are also 2 fixes for Cassandra that  should be included:
> > > >>>> https://github.com/apache/flink/pull/2167
> > > >>>> https://github.com/apache/flink/pull/2183
> > > >>>>
> > > >>>> We should also include the documentation for the metrics stuff
> > > (hopefully
> > > >>>> merged today)
> > > >>>> https://github.com/apache/flink/pull/2158
> > > >>>>
> > > >>>> In regards to metrics: To add a counter metric a user currently
> has
> > > to call
> > > >>>> "counter(...)" on
> > > >>>> a MetricGroup. The point was raised in the documentation PR that
> we
> > > may want
> > > >>>> to give
> > > >>>> them a more descriptive name like "addCounter(...)".
> > > >>>>
> > > >>>> I would be in favor of changing them but would like others to
> weigh
> > > in on
> > > >>>> this. IMO we
> > > >>>> should nail this down before 1.1 .
> > > >>>>
> > > >>>> Regards,
> > > >>>> Chesnay
> > > >>>>
> > > >>>>
> > > >>>> On 30.06.2016 22:59, Greg Hogan wrote:
> > > >>>>>
> > > >>>>> It would be great if hash-based combine (FLINK-3477) could make
> it
> > > in to
> > > >>>>> be
> > > >>>>> tested for this release. We've seen impressive improvements in
> > > performance
> > > >>>>> (though, admittedly, some sort-based enhancements are yet to be
> > > worked
> > > >>>>> on).
> > > >>>>> This PR looks to be ripe.
> > > >>>>>
> > > >>>>> Also, as we tidy up a few things with Gelly and documentation,
> what
> > > is the
> > > >>>>> schedule for a feature freeze and creating a 1.1 branch off
> master?
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Greg
> > > >>>>>
> > > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <
> > rmetzger@apache.org
> > > >
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Sure Ufuk! Thanks a lot for taking care of the release
> management.
> > > >>>>>> I'll be on vacation in three weeks, for three weeks and I'm not
> > > sure if
> > > >>>>>> we
> > > >>>>>> get the release done until then.
> > > >>>>>>
> > > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org>
> > > wrote:
> > > >>>>>>
> > > >>>>>>> I would like to do it if that's OK with you Robert. I would
> > follow
> > > >>>>>>> your suggestion and wait a few days until the following
> important
> > > >>>>>>> fixes are in:
> > > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
> > > >>>>>>> https://github.com/apache/flink/pull/2083)
> > > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
> > > >>>>>>> - Table API time support (
> > > https://github.com/apache/flink/pull/2150)
> > > >>>>>>> - Kafka at-least-once Producer (
> > > >>>>>>
> > > >>>>>> https://github.com/apache/flink/pull/2108)
> > > >>>>>>>
> > > >>>>>>> - Cassandra connector fixes (
> > > https://github.com/apache/flink/pull/2163)
> > > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there
> > > something
> > > >>>>>>
> > > >>>>>> else?)
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <
> > > rmetzger@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> Hi,
> > > >>>>>>>> it doesn't seem that there are volunteers for the RM, so I'll
> > > probably
> > > >>>>>>
> > > >>>>>> do
> > > >>>>>>>>
> > > >>>>>>>> it.
> > > >>>>>>>>
> > > >>>>>>>> I try to do the first release candidate (mostly for testing)
> > next
> > > week
> > > >>>>>>>
> > > >>>>>>> (it
> > > >>>>>>>>
> > > >>>>>>>> depends on the JIRAs fixed by then)
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
> > > >>>>>>
> > > >>>>>> henry.saputra@gmail.com
> > > >>>>>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Thanks for the reply, @Max. I was not aware it was about
> > dynamic
> > > >>>>>>>
> > > >>>>>>> scaling,
> > > >>>>>>>>>
> > > >>>>>>>>> which I think also asked for YARN support.
> > > >>>>>>>>> I agree to list all related half merge JIRA for the
> > > ResourceManager.
> > > >>>>>>>>>
> > > >>>>>>>>> Looking forward for the Apache Mesos integration design for
> > sure
> > > =)
> > > >>>>>>>>>
> > > >>>>>>>>> - Henry
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <
> > > mxm@apache.org>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi Robert, hi Henry,
> > > >>>>>>>>>>
> > > >>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features
> that
> > > >>>>>>
> > > >>>>>> justify
> > > >>>>>>>>>>
> > > >>>>>>>>>> a major release.
> > > >>>>>>>>>>
> > > >>>>>>>>>> @Henry We have plans to extend the ResourceManager to
> interact
> > > with
> > > >>>>>>>>>> the Scheduler which will be a prerequisite for dynamic
> > scaling.
> > > I
> > > >>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos
> > > integration
> > > >>>>>>>>>> won't require additional refactoring of the ResourceManager.
> > > >>>>>>
> > > >>>>>> Instead,
> > > >>>>>>>>>>
> > > >>>>>>>>>> we will create a new "Dispatcher" component that takes care
> of
> > > >>>>>>>>>> bootstrapping the initial node with the
> > > JobManager/ResourceManager.
> > > >>>>>>>>>>  From there on, everything will be handled by the Mesos
> > > >>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC)
> who
> > > came
> > > >>>>>>
> > > >>>>>> up
> > > >>>>>>>>>>
> > > >>>>>>>>>> with this design and he plans to publish it to the mailing
> > list
> > > >>>>>>
> > > >>>>>> soon.
> > > >>>>>>>>>>
> > > >>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged"
> is
> > > kind
> > > >>>>>>
> > > >>>>>> of
> > > >>>>>>>>>>
> > > >>>>>>>>>> hard to get for people who are not involved in the different
> > > >>>>>>>>>> components.
> > > >>>>>>>>>>
> > > >>>>>>>>>> The Cassandra adapter seems like a pretty important thing to
> > > have
> > > >>>>>>
> > > >>>>>> for
> > > >>>>>>>>>>
> > > >>>>>>>>>> the next release. In addition, I would like to merge
> > FLINK-3667
> > > and
> > > >>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those
> > are
> > > a)
> > > >>>>>>>>>> refactoring of the command-line and client classes b) adding
> > > >>>>>>>>>> capability to resume cluster programmatically.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Then we should also have a look at any other critical/major
> > bugs
> > > >>>>>>>
> > > >>>>>>> listed
> > > >>>>>>>>>
> > > >>>>>>>>> in
> > > >>>>>>>>>>
> > > >>>>>>>>>> JIRA.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Cheers,
> > > >>>>>>>>>> Max
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> > > >>>>>>>
> > > >>>>>>> henry.saputra@gmail.com
> > > >>>>>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Hi Robert,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks for staying the discussion.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Do you know if there any open tasks for the Resource
> Manager
> > > left?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> That is probably needed for Mesos integration?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Henry
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> > > >>>>>>>
> > > >>>>>>> rmetzger@apache.org>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Flink 1.0.0 was released early March, so three months have
> > > passed
> > > >>>>>>>
> > > >>>>>>> and
> > > >>>>>>>>>
> > > >>>>>>>>> I
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> think we should start discussing the scope of the next
> major
> > > >>>>>>>
> > > >>>>>>> release
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> (1.1.0).
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>  From a high level point of view, we've added the
> following
> > > new
> > > >>>>>>>>>
> > > >>>>>>>>> features:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> in master:
> > > >>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
> > > >>>>>>>>>>>> - The metrics system
> > > >>>>>>>>>>>> - Kinesis Connector
> > > >>>>>>>>>>>> - Persistent file sources for streaming
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Half merged:
> > > >>>>>>>>>>>> - Resource manager refactoring
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Unmerged features:
> > > >>>>>>>>>>>> - Cassandra connector
> > > >>>>>>>>>>>> - Key groups ("rescaling from savepoints")
> > > >>>>>>>>>>>> - Queryable state
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I'm pretty sure I forgot many other features / pull
> > requests,
> > > >>>>>>>
> > > >>>>>>> please
> > > >>>>>>>>>>
> > > >>>>>>>>>> post
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki
> > page
> > > out
> > > >>>>>>>
> > > >>>>>>> of
> > > >>>>>>>>>
> > > >>>>>>>>> it.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Some immediate TODOs for us:
> > > >>>>>>>>>>>> - Which of the unmerged features are we going to add to
> the
> > > >>>>>>>
> > > >>>>>>> release?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> - Which blockers do we need to address before releasing?
> > > >>>>>>>>>>>> - Are there any volunteers for the release manager?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>> Robert
> > > >>>>>>>>>>>>
> > > >>>>
> > >
> >
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Flavio Pompermaier <po...@okkam.it>]

Hi to all,
if Flink 1.1 will introduce ufficially the Table API, do you think someone
could take care of rewriting in scala the necessary java code of my PR
about reading CSV as Rows instead of tuples[1]?

For our use cases, and many new users approaching to Flink IMHO, that will
be definitely useful because you can read csv files with more than 25
columns and properly handle null values. What  do you think?

[1] https://github.com/apache/flink/pull/1989

Best,
Flavio
On 5 Jul 2016 18:08, "Till Rohrmann" <tr...@apache.org> wrote:

I found another critical issue [1]. The murmur hash correction introduced
between Flink 1.0 and 1.1 breaks the backwards compatibility with respect
to savepoints. I think we have to fix this for the release.

@Ufuk, I'm not sure whether I find time this week to work on FLINK-4150. I
could make it a stretch goal for this sprint. Or we schedule it in next
week's sprint.

[1] https://issues.apache.org/jira/browse/FLINK-4154



On Tue, Jul 5, 2016 at 3:56 PM, Greg Hogan <co...@greghogan.com> wrote:

> Hi Ufuk,
>
> The old sort-based combine is still the default. The user calls
> .setCombineHint(CombineHint) to make a selection (I think this was
> originally overloaded on DataSet and it looks like the pr1517
documentation
> update does not reflect the new usage).
>
> I'd be glad to merge this in but I didn't want to preempt Gabor or Fabian.
>
> Greg
>
> On Mon, Jul 4, 2016 at 7:47 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
> > Thanks for the feedback. I would like to create a preview RC0 this
> > week like we did for the last releases, too. In past major releases,
> > we needed to create many release candidates, often for fixing just
> > some small issues. I would like to speed up the release process by
> > collecting as many issues as possible now with the RC0. Once these
> > issues are resolved, we can start voting with the RC1. This
> > essentially means that we have a feature freeze. I would create the
> > release-1.1 branch with RC1.
> >
> > Regarding the open issues:
> >
> > - The savepoint fixes are not yet in. There is a PR for the savepoint
> > headers (#2194) and the savepoint disposal PR needs addressing of
> > comments.
> >
> > - The Cassandra connector fixes are either merged or very close to be
> > merged.
> >
> > - Metrics docs are not a blocker since the online docs are updated
> > after the release. Regarding the renaming, we should decide soon. Any
> > opinions here?
> >
> > - The YARN issues have been resolved.
> >
> > I will also make a pass over JIRA and our PRs to check whether we've
> > missed something.
> >
> > @Greg: You are right, the hash-based combine PR has been extensively
> > reviewed. Unfortunately, I'm not familiar with the code as I didn't
> > look into it and cannot tell whether we should merge it now. Is the
> > hash-based combine strategy activated by default or does the user has
> > to activate it manually? The later case would make me feel more
> > comfortable merging it for the release.
> >
> >
> > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mx...@apache.org>
> wrote:
> > > Yet another one for the release: FLINK-4144
> > > https://github.com/apache/flink/pull/2191
> > >
> > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mx...@apache.org>
> > wrote:
> > >> There is also FLINK-4141. We just found this during testing. PR is
> > >> waiting to be merged here: https://github.com/apache/flink/pull/2190
> > >>
> > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mx...@apache.org>
> > wrote:
> > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this release
> > >>> and actually a bit tricky to fix. I've addressed the other issues
and
> > >>> merged all pending pull requests. Good to go from my side.
> > >>>
> > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <
> chesnay@apache.org>
> > wrote:
> > >>>> There are also 2 fixes for Cassandra that  should be included:
> > >>>> https://github.com/apache/flink/pull/2167
> > >>>> https://github.com/apache/flink/pull/2183
> > >>>>
> > >>>> We should also include the documentation for the metrics stuff
> > (hopefully
> > >>>> merged today)
> > >>>> https://github.com/apache/flink/pull/2158
> > >>>>
> > >>>> In regards to metrics: To add a counter metric a user currently has
> > to call
> > >>>> "counter(...)" on
> > >>>> a MetricGroup. The point was raised in the documentation PR that we
> > may want
> > >>>> to give
> > >>>> them a more descriptive name like "addCounter(...)".
> > >>>>
> > >>>> I would be in favor of changing them but would like others to weigh
> > in on
> > >>>> this. IMO we
> > >>>> should nail this down before 1.1 .
> > >>>>
> > >>>> Regards,
> > >>>> Chesnay
> > >>>>
> > >>>>
> > >>>> On 30.06.2016 22:59, Greg Hogan wrote:
> > >>>>>
> > >>>>> It would be great if hash-based combine (FLINK-3477) could make it
> > in to
> > >>>>> be
> > >>>>> tested for this release. We've seen impressive improvements in
> > performance
> > >>>>> (though, admittedly, some sort-based enhancements are yet to be
> > worked
> > >>>>> on).
> > >>>>> This PR looks to be ripe.
> > >>>>>
> > >>>>> Also, as we tidy up a few things with Gelly and documentation,
what
> > is the
> > >>>>> schedule for a feature freeze and creating a 1.1 branch off
master?
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Greg
> > >>>>>
> > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <
> rmetzger@apache.org
> > >
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Sure Ufuk! Thanks a lot for taking care of the release
management.
> > >>>>>> I'll be on vacation in three weeks, for three weeks and I'm not
> > sure if
> > >>>>>> we
> > >>>>>> get the release done until then.
> > >>>>>>
> > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org>
> > wrote:
> > >>>>>>
> > >>>>>>> I would like to do it if that's OK with you Robert. I would
> follow
> > >>>>>>> your suggestion and wait a few days until the following
important
> > >>>>>>> fixes are in:
> > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
> > >>>>>>> https://github.com/apache/flink/pull/2083)
> > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
> > >>>>>>> - Table API time support (
> > https://github.com/apache/flink/pull/2150)
> > >>>>>>> - Kafka at-least-once Producer (
> > >>>>>>
> > >>>>>> https://github.com/apache/flink/pull/2108)
> > >>>>>>>
> > >>>>>>> - Cassandra connector fixes (
> > https://github.com/apache/flink/pull/2163)
> > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there
> > something
> > >>>>>>
> > >>>>>> else?)
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <
> > rmetzger@apache.org>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi,
> > >>>>>>>> it doesn't seem that there are volunteers for the RM, so I'll
> > probably
> > >>>>>>
> > >>>>>> do
> > >>>>>>>>
> > >>>>>>>> it.
> > >>>>>>>>
> > >>>>>>>> I try to do the first release candidate (mostly for testing)
> next
> > week
> > >>>>>>>
> > >>>>>>> (it
> > >>>>>>>>
> > >>>>>>>> depends on the JIRAs fixed by then)
> > >>>>>>>>
> > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
> > >>>>>>
> > >>>>>> henry.saputra@gmail.com
> > >>>>>>>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Thanks for the reply, @Max. I was not aware it was about
> dynamic
> > >>>>>>>
> > >>>>>>> scaling,
> > >>>>>>>>>
> > >>>>>>>>> which I think also asked for YARN support.
> > >>>>>>>>> I agree to list all related half merge JIRA for the
> > ResourceManager.
> > >>>>>>>>>
> > >>>>>>>>> Looking forward for the Apache Mesos integration design for
> sure
> > =)
> > >>>>>>>>>
> > >>>>>>>>> - Henry
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <
> > mxm@apache.org>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Robert, hi Henry,
> > >>>>>>>>>>
> > >>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features that
> > >>>>>>
> > >>>>>> justify
> > >>>>>>>>>>
> > >>>>>>>>>> a major release.
> > >>>>>>>>>>
> > >>>>>>>>>> @Henry We have plans to extend the ResourceManager to
interact
> > with
> > >>>>>>>>>> the Scheduler which will be a prerequisite for dynamic
> scaling.
> > I
> > >>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos
> > integration
> > >>>>>>>>>> won't require additional refactoring of the ResourceManager.
> > >>>>>>
> > >>>>>> Instead,
> > >>>>>>>>>>
> > >>>>>>>>>> we will create a new "Dispatcher" component that takes care
of
> > >>>>>>>>>> bootstrapping the initial node with the
> > JobManager/ResourceManager.
> > >>>>>>>>>>  From there on, everything will be handled by the Mesos
> > >>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC) who
> > came
> > >>>>>>
> > >>>>>> up
> > >>>>>>>>>>
> > >>>>>>>>>> with this design and he plans to publish it to the mailing
> list
> > >>>>>>
> > >>>>>> soon.
> > >>>>>>>>>>
> > >>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged" is
> > kind
> > >>>>>>
> > >>>>>> of
> > >>>>>>>>>>
> > >>>>>>>>>> hard to get for people who are not involved in the different
> > >>>>>>>>>> components.
> > >>>>>>>>>>
> > >>>>>>>>>> The Cassandra adapter seems like a pretty important thing to
> > have
> > >>>>>>
> > >>>>>> for
> > >>>>>>>>>>
> > >>>>>>>>>> the next release. In addition, I would like to merge
> FLINK-3667
> > and
> > >>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those
> are
> > a)
> > >>>>>>>>>> refactoring of the command-line and client classes b) adding
> > >>>>>>>>>> capability to resume cluster programmatically.
> > >>>>>>>>>>
> > >>>>>>>>>> Then we should also have a look at any other critical/major
> bugs
> > >>>>>>>
> > >>>>>>> listed
> > >>>>>>>>>
> > >>>>>>>>> in
> > >>>>>>>>>>
> > >>>>>>>>>> JIRA.
> > >>>>>>>>>>
> > >>>>>>>>>> Cheers,
> > >>>>>>>>>> Max
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> > >>>>>>>
> > >>>>>>> henry.saputra@gmail.com
> > >>>>>>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi Robert,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for staying the discussion.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Do you know if there any open tasks for the Resource Manager
> > left?
> > >>>>>>>>>>>
> > >>>>>>>>>>> That is probably needed for Mesos integration?
> > >>>>>>>>>>>
> > >>>>>>>>>>> - Henry
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> > >>>>>>>
> > >>>>>>> rmetzger@apache.org>
> > >>>>>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Flink 1.0.0 was released early March, so three months have
> > passed
> > >>>>>>>
> > >>>>>>> and
> > >>>>>>>>>
> > >>>>>>>>> I
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> think we should start discussing the scope of the next
major
> > >>>>>>>
> > >>>>>>> release
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> (1.1.0).
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>  From a high level point of view, we've added the following
> > new
> > >>>>>>>>>
> > >>>>>>>>> features:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> in master:
> > >>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
> > >>>>>>>>>>>> - The metrics system
> > >>>>>>>>>>>> - Kinesis Connector
> > >>>>>>>>>>>> - Persistent file sources for streaming
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Half merged:
> > >>>>>>>>>>>> - Resource manager refactoring
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Unmerged features:
> > >>>>>>>>>>>> - Cassandra connector
> > >>>>>>>>>>>> - Key groups ("rescaling from savepoints")
> > >>>>>>>>>>>> - Queryable state
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I'm pretty sure I forgot many other features / pull
> requests,
> > >>>>>>>
> > >>>>>>> please
> > >>>>>>>>>>
> > >>>>>>>>>> post
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki
> page
> > out
> > >>>>>>>
> > >>>>>>> of
> > >>>>>>>>>
> > >>>>>>>>> it.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Some immediate TODOs for us:
> > >>>>>>>>>>>> - Which of the unmerged features are we going to add to the
> > >>>>>>>
> > >>>>>>> release?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> - Which blockers do we need to address before releasing?
> > >>>>>>>>>>>> - Are there any volunteers for the release manager?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Regards,
> > >>>>>>>>>>>> Robert
> > >>>>>>>>>>>>
> > >>>>
> >
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Till Rohrmann <tr...@apache.org>]

I found another critical issue [1]. The murmur hash correction introduced
between Flink 1.0 and 1.1 breaks the backwards compatibility with respect
to savepoints. I think we have to fix this for the release.

@Ufuk, I'm not sure whether I find time this week to work on FLINK-4150. I
could make it a stretch goal for this sprint. Or we schedule it in next
week's sprint.

[1] https://issues.apache.org/jira/browse/FLINK-4154



On Tue, Jul 5, 2016 at 3:56 PM, Greg Hogan <co...@greghogan.com> wrote:

> Hi Ufuk,
>
> The old sort-based combine is still the default. The user calls
> .setCombineHint(CombineHint) to make a selection (I think this was
> originally overloaded on DataSet and it looks like the pr1517 documentation
> update does not reflect the new usage).
>
> I'd be glad to merge this in but I didn't want to preempt Gabor or Fabian.
>
> Greg
>
> On Mon, Jul 4, 2016 at 7:47 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
> > Thanks for the feedback. I would like to create a preview RC0 this
> > week like we did for the last releases, too. In past major releases,
> > we needed to create many release candidates, often for fixing just
> > some small issues. I would like to speed up the release process by
> > collecting as many issues as possible now with the RC0. Once these
> > issues are resolved, we can start voting with the RC1. This
> > essentially means that we have a feature freeze. I would create the
> > release-1.1 branch with RC1.
> >
> > Regarding the open issues:
> >
> > - The savepoint fixes are not yet in. There is a PR for the savepoint
> > headers (#2194) and the savepoint disposal PR needs addressing of
> > comments.
> >
> > - The Cassandra connector fixes are either merged or very close to be
> > merged.
> >
> > - Metrics docs are not a blocker since the online docs are updated
> > after the release. Regarding the renaming, we should decide soon. Any
> > opinions here?
> >
> > - The YARN issues have been resolved.
> >
> > I will also make a pass over JIRA and our PRs to check whether we've
> > missed something.
> >
> > @Greg: You are right, the hash-based combine PR has been extensively
> > reviewed. Unfortunately, I'm not familiar with the code as I didn't
> > look into it and cannot tell whether we should merge it now. Is the
> > hash-based combine strategy activated by default or does the user has
> > to activate it manually? The later case would make me feel more
> > comfortable merging it for the release.
> >
> >
> > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mx...@apache.org>
> wrote:
> > > Yet another one for the release: FLINK-4144
> > > https://github.com/apache/flink/pull/2191
> > >
> > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mx...@apache.org>
> > wrote:
> > >> There is also FLINK-4141. We just found this during testing. PR is
> > >> waiting to be merged here: https://github.com/apache/flink/pull/2190
> > >>
> > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mx...@apache.org>
> > wrote:
> > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this release
> > >>> and actually a bit tricky to fix. I've addressed the other issues and
> > >>> merged all pending pull requests. Good to go from my side.
> > >>>
> > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <
> chesnay@apache.org>
> > wrote:
> > >>>> There are also 2 fixes for Cassandra that  should be included:
> > >>>> https://github.com/apache/flink/pull/2167
> > >>>> https://github.com/apache/flink/pull/2183
> > >>>>
> > >>>> We should also include the documentation for the metrics stuff
> > (hopefully
> > >>>> merged today)
> > >>>> https://github.com/apache/flink/pull/2158
> > >>>>
> > >>>> In regards to metrics: To add a counter metric a user currently has
> > to call
> > >>>> "counter(...)" on
> > >>>> a MetricGroup. The point was raised in the documentation PR that we
> > may want
> > >>>> to give
> > >>>> them a more descriptive name like "addCounter(...)".
> > >>>>
> > >>>> I would be in favor of changing them but would like others to weigh
> > in on
> > >>>> this. IMO we
> > >>>> should nail this down before 1.1 .
> > >>>>
> > >>>> Regards,
> > >>>> Chesnay
> > >>>>
> > >>>>
> > >>>> On 30.06.2016 22:59, Greg Hogan wrote:
> > >>>>>
> > >>>>> It would be great if hash-based combine (FLINK-3477) could make it
> > in to
> > >>>>> be
> > >>>>> tested for this release. We've seen impressive improvements in
> > performance
> > >>>>> (though, admittedly, some sort-based enhancements are yet to be
> > worked
> > >>>>> on).
> > >>>>> This PR looks to be ripe.
> > >>>>>
> > >>>>> Also, as we tidy up a few things with Gelly and documentation, what
> > is the
> > >>>>> schedule for a feature freeze and creating a 1.1 branch off master?
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Greg
> > >>>>>
> > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <
> rmetzger@apache.org
> > >
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Sure Ufuk! Thanks a lot for taking care of the release management.
> > >>>>>> I'll be on vacation in three weeks, for three weeks and I'm not
> > sure if
> > >>>>>> we
> > >>>>>> get the release done until then.
> > >>>>>>
> > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org>
> > wrote:
> > >>>>>>
> > >>>>>>> I would like to do it if that's OK with you Robert. I would
> follow
> > >>>>>>> your suggestion and wait a few days until the following important
> > >>>>>>> fixes are in:
> > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
> > >>>>>>> https://github.com/apache/flink/pull/2083)
> > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
> > >>>>>>> - Table API time support (
> > https://github.com/apache/flink/pull/2150)
> > >>>>>>> - Kafka at-least-once Producer (
> > >>>>>>
> > >>>>>> https://github.com/apache/flink/pull/2108)
> > >>>>>>>
> > >>>>>>> - Cassandra connector fixes (
> > https://github.com/apache/flink/pull/2163)
> > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there
> > something
> > >>>>>>
> > >>>>>> else?)
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <
> > rmetzger@apache.org>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi,
> > >>>>>>>> it doesn't seem that there are volunteers for the RM, so I'll
> > probably
> > >>>>>>
> > >>>>>> do
> > >>>>>>>>
> > >>>>>>>> it.
> > >>>>>>>>
> > >>>>>>>> I try to do the first release candidate (mostly for testing)
> next
> > week
> > >>>>>>>
> > >>>>>>> (it
> > >>>>>>>>
> > >>>>>>>> depends on the JIRAs fixed by then)
> > >>>>>>>>
> > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
> > >>>>>>
> > >>>>>> henry.saputra@gmail.com
> > >>>>>>>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Thanks for the reply, @Max. I was not aware it was about
> dynamic
> > >>>>>>>
> > >>>>>>> scaling,
> > >>>>>>>>>
> > >>>>>>>>> which I think also asked for YARN support.
> > >>>>>>>>> I agree to list all related half merge JIRA for the
> > ResourceManager.
> > >>>>>>>>>
> > >>>>>>>>> Looking forward for the Apache Mesos integration design for
> sure
> > =)
> > >>>>>>>>>
> > >>>>>>>>> - Henry
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <
> > mxm@apache.org>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Robert, hi Henry,
> > >>>>>>>>>>
> > >>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features that
> > >>>>>>
> > >>>>>> justify
> > >>>>>>>>>>
> > >>>>>>>>>> a major release.
> > >>>>>>>>>>
> > >>>>>>>>>> @Henry We have plans to extend the ResourceManager to interact
> > with
> > >>>>>>>>>> the Scheduler which will be a prerequisite for dynamic
> scaling.
> > I
> > >>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos
> > integration
> > >>>>>>>>>> won't require additional refactoring of the ResourceManager.
> > >>>>>>
> > >>>>>> Instead,
> > >>>>>>>>>>
> > >>>>>>>>>> we will create a new "Dispatcher" component that takes care of
> > >>>>>>>>>> bootstrapping the initial node with the
> > JobManager/ResourceManager.
> > >>>>>>>>>>  From there on, everything will be handled by the Mesos
> > >>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC) who
> > came
> > >>>>>>
> > >>>>>> up
> > >>>>>>>>>>
> > >>>>>>>>>> with this design and he plans to publish it to the mailing
> list
> > >>>>>>
> > >>>>>> soon.
> > >>>>>>>>>>
> > >>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged" is
> > kind
> > >>>>>>
> > >>>>>> of
> > >>>>>>>>>>
> > >>>>>>>>>> hard to get for people who are not involved in the different
> > >>>>>>>>>> components.
> > >>>>>>>>>>
> > >>>>>>>>>> The Cassandra adapter seems like a pretty important thing to
> > have
> > >>>>>>
> > >>>>>> for
> > >>>>>>>>>>
> > >>>>>>>>>> the next release. In addition, I would like to merge
> FLINK-3667
> > and
> > >>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those
> are
> > a)
> > >>>>>>>>>> refactoring of the command-line and client classes b) adding
> > >>>>>>>>>> capability to resume cluster programmatically.
> > >>>>>>>>>>
> > >>>>>>>>>> Then we should also have a look at any other critical/major
> bugs
> > >>>>>>>
> > >>>>>>> listed
> > >>>>>>>>>
> > >>>>>>>>> in
> > >>>>>>>>>>
> > >>>>>>>>>> JIRA.
> > >>>>>>>>>>
> > >>>>>>>>>> Cheers,
> > >>>>>>>>>> Max
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> > >>>>>>>
> > >>>>>>> henry.saputra@gmail.com
> > >>>>>>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi Robert,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for staying the discussion.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Do you know if there any open tasks for the Resource Manager
> > left?
> > >>>>>>>>>>>
> > >>>>>>>>>>> That is probably needed for Mesos integration?
> > >>>>>>>>>>>
> > >>>>>>>>>>> - Henry
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> > >>>>>>>
> > >>>>>>> rmetzger@apache.org>
> > >>>>>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Flink 1.0.0 was released early March, so three months have
> > passed
> > >>>>>>>
> > >>>>>>> and
> > >>>>>>>>>
> > >>>>>>>>> I
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> think we should start discussing the scope of the next major
> > >>>>>>>
> > >>>>>>> release
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> (1.1.0).
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>  From a high level point of view, we've added the following
> > new
> > >>>>>>>>>
> > >>>>>>>>> features:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> in master:
> > >>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
> > >>>>>>>>>>>> - The metrics system
> > >>>>>>>>>>>> - Kinesis Connector
> > >>>>>>>>>>>> - Persistent file sources for streaming
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Half merged:
> > >>>>>>>>>>>> - Resource manager refactoring
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Unmerged features:
> > >>>>>>>>>>>> - Cassandra connector
> > >>>>>>>>>>>> - Key groups ("rescaling from savepoints")
> > >>>>>>>>>>>> - Queryable state
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I'm pretty sure I forgot many other features / pull
> requests,
> > >>>>>>>
> > >>>>>>> please
> > >>>>>>>>>>
> > >>>>>>>>>> post
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki
> page
> > out
> > >>>>>>>
> > >>>>>>> of
> > >>>>>>>>>
> > >>>>>>>>> it.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Some immediate TODOs for us:
> > >>>>>>>>>>>> - Which of the unmerged features are we going to add to the
> > >>>>>>>
> > >>>>>>> release?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> - Which blockers do we need to address before releasing?
> > >>>>>>>>>>>> - Are there any volunteers for the release manager?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Regards,
> > >>>>>>>>>>>> Robert
> > >>>>>>>>>>>>
> > >>>>
> >
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Greg Hogan <co...@greghogan.com>]

Hi Ufuk,

The old sort-based combine is still the default. The user calls
.setCombineHint(CombineHint) to make a selection (I think this was
originally overloaded on DataSet and it looks like the pr1517 documentation
update does not reflect the new usage).

I'd be glad to merge this in but I didn't want to preempt Gabor or Fabian.

Greg

On Mon, Jul 4, 2016 at 7:47 AM, Ufuk Celebi <uc...@apache.org> wrote:

> Thanks for the feedback. I would like to create a preview RC0 this
> week like we did for the last releases, too. In past major releases,
> we needed to create many release candidates, often for fixing just
> some small issues. I would like to speed up the release process by
> collecting as many issues as possible now with the RC0. Once these
> issues are resolved, we can start voting with the RC1. This
> essentially means that we have a feature freeze. I would create the
> release-1.1 branch with RC1.
>
> Regarding the open issues:
>
> - The savepoint fixes are not yet in. There is a PR for the savepoint
> headers (#2194) and the savepoint disposal PR needs addressing of
> comments.
>
> - The Cassandra connector fixes are either merged or very close to be
> merged.
>
> - Metrics docs are not a blocker since the online docs are updated
> after the release. Regarding the renaming, we should decide soon. Any
> opinions here?
>
> - The YARN issues have been resolved.
>
> I will also make a pass over JIRA and our PRs to check whether we've
> missed something.
>
> @Greg: You are right, the hash-based combine PR has been extensively
> reviewed. Unfortunately, I'm not familiar with the code as I didn't
> look into it and cannot tell whether we should merge it now. Is the
> hash-based combine strategy activated by default or does the user has
> to activate it manually? The later case would make me feel more
> comfortable merging it for the release.
>
>
> On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mx...@apache.org> wrote:
> > Yet another one for the release: FLINK-4144
> > https://github.com/apache/flink/pull/2191
> >
> > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mx...@apache.org>
> wrote:
> >> There is also FLINK-4141. We just found this during testing. PR is
> >> waiting to be merged here: https://github.com/apache/flink/pull/2190
> >>
> >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mx...@apache.org>
> wrote:
> >>> FLINK-3904 is not Yarn related. Not pressing to fix for this release
> >>> and actually a bit tricky to fix. I've addressed the other issues and
> >>> merged all pending pull requests. Good to go from my side.
> >>>
> >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <ch...@apache.org>
> wrote:
> >>>> There are also 2 fixes for Cassandra that  should be included:
> >>>> https://github.com/apache/flink/pull/2167
> >>>> https://github.com/apache/flink/pull/2183
> >>>>
> >>>> We should also include the documentation for the metrics stuff
> (hopefully
> >>>> merged today)
> >>>> https://github.com/apache/flink/pull/2158
> >>>>
> >>>> In regards to metrics: To add a counter metric a user currently has
> to call
> >>>> "counter(...)" on
> >>>> a MetricGroup. The point was raised in the documentation PR that we
> may want
> >>>> to give
> >>>> them a more descriptive name like "addCounter(...)".
> >>>>
> >>>> I would be in favor of changing them but would like others to weigh
> in on
> >>>> this. IMO we
> >>>> should nail this down before 1.1 .
> >>>>
> >>>> Regards,
> >>>> Chesnay
> >>>>
> >>>>
> >>>> On 30.06.2016 22:59, Greg Hogan wrote:
> >>>>>
> >>>>> It would be great if hash-based combine (FLINK-3477) could make it
> in to
> >>>>> be
> >>>>> tested for this release. We've seen impressive improvements in
> performance
> >>>>> (though, admittedly, some sort-based enhancements are yet to be
> worked
> >>>>> on).
> >>>>> This PR looks to be ripe.
> >>>>>
> >>>>> Also, as we tidy up a few things with Gelly and documentation, what
> is the
> >>>>> schedule for a feature freeze and creating a 1.1 branch off master?
> >>>>>
> >>>>> Thanks,
> >>>>> Greg
> >>>>>
> >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <rmetzger@apache.org
> >
> >>>>> wrote:
> >>>>>
> >>>>>> Sure Ufuk! Thanks a lot for taking care of the release management.
> >>>>>> I'll be on vacation in three weeks, for three weeks and I'm not
> sure if
> >>>>>> we
> >>>>>> get the release done until then.
> >>>>>>
> >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org>
> wrote:
> >>>>>>
> >>>>>>> I would like to do it if that's OK with you Robert. I would follow
> >>>>>>> your suggestion and wait a few days until the following important
> >>>>>>> fixes are in:
> >>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
> >>>>>>> https://github.com/apache/flink/pull/2083)
> >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
> >>>>>>> - Table API time support (
> https://github.com/apache/flink/pull/2150)
> >>>>>>> - Kafka at-least-once Producer (
> >>>>>>
> >>>>>> https://github.com/apache/flink/pull/2108)
> >>>>>>>
> >>>>>>> - Cassandra connector fixes (
> https://github.com/apache/flink/pull/2163)
> >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there
> something
> >>>>>>
> >>>>>> else?)
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <
> rmetzger@apache.org>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>> it doesn't seem that there are volunteers for the RM, so I'll
> probably
> >>>>>>
> >>>>>> do
> >>>>>>>>
> >>>>>>>> it.
> >>>>>>>>
> >>>>>>>> I try to do the first release candidate (mostly for testing) next
> week
> >>>>>>>
> >>>>>>> (it
> >>>>>>>>
> >>>>>>>> depends on the JIRAs fixed by then)
> >>>>>>>>
> >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
> >>>>>>
> >>>>>> henry.saputra@gmail.com
> >>>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Thanks for the reply, @Max. I was not aware it was about dynamic
> >>>>>>>
> >>>>>>> scaling,
> >>>>>>>>>
> >>>>>>>>> which I think also asked for YARN support.
> >>>>>>>>> I agree to list all related half merge JIRA for the
> ResourceManager.
> >>>>>>>>>
> >>>>>>>>> Looking forward for the Apache Mesos integration design for sure
> =)
> >>>>>>>>>
> >>>>>>>>> - Henry
> >>>>>>>>>
> >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <
> mxm@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Robert, hi Henry,
> >>>>>>>>>>
> >>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features that
> >>>>>>
> >>>>>> justify
> >>>>>>>>>>
> >>>>>>>>>> a major release.
> >>>>>>>>>>
> >>>>>>>>>> @Henry We have plans to extend the ResourceManager to interact
> with
> >>>>>>>>>> the Scheduler which will be a prerequisite for dynamic scaling.
> I
> >>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos
> integration
> >>>>>>>>>> won't require additional refactoring of the ResourceManager.
> >>>>>>
> >>>>>> Instead,
> >>>>>>>>>>
> >>>>>>>>>> we will create a new "Dispatcher" component that takes care of
> >>>>>>>>>> bootstrapping the initial node with the
> JobManager/ResourceManager.
> >>>>>>>>>>  From there on, everything will be handled by the Mesos
> >>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC) who
> came
> >>>>>>
> >>>>>> up
> >>>>>>>>>>
> >>>>>>>>>> with this design and he plans to publish it to the mailing list
> >>>>>>
> >>>>>> soon.
> >>>>>>>>>>
> >>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged" is
> kind
> >>>>>>
> >>>>>> of
> >>>>>>>>>>
> >>>>>>>>>> hard to get for people who are not involved in the different
> >>>>>>>>>> components.
> >>>>>>>>>>
> >>>>>>>>>> The Cassandra adapter seems like a pretty important thing to
> have
> >>>>>>
> >>>>>> for
> >>>>>>>>>>
> >>>>>>>>>> the next release. In addition, I would like to merge FLINK-3667
> and
> >>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those are
> a)
> >>>>>>>>>> refactoring of the command-line and client classes b) adding
> >>>>>>>>>> capability to resume cluster programmatically.
> >>>>>>>>>>
> >>>>>>>>>> Then we should also have a look at any other critical/major bugs
> >>>>>>>
> >>>>>>> listed
> >>>>>>>>>
> >>>>>>>>> in
> >>>>>>>>>>
> >>>>>>>>>> JIRA.
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> Max
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> >>>>>>>
> >>>>>>> henry.saputra@gmail.com
> >>>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Robert,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for staying the discussion.
> >>>>>>>>>>>
> >>>>>>>>>>> Do you know if there any open tasks for the Resource Manager
> left?
> >>>>>>>>>>>
> >>>>>>>>>>> That is probably needed for Mesos integration?
> >>>>>>>>>>>
> >>>>>>>>>>> - Henry
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> >>>>>>>
> >>>>>>> rmetzger@apache.org>
> >>>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Flink 1.0.0 was released early March, so three months have
> passed
> >>>>>>>
> >>>>>>> and
> >>>>>>>>>
> >>>>>>>>> I
> >>>>>>>>>>>>
> >>>>>>>>>>>> think we should start discussing the scope of the next major
> >>>>>>>
> >>>>>>> release
> >>>>>>>>>>>>
> >>>>>>>>>>>> (1.1.0).
> >>>>>>>>>>>>
> >>>>>>>>>>>>  From a high level point of view, we've added the following
> new
> >>>>>>>>>
> >>>>>>>>> features:
> >>>>>>>>>>>>
> >>>>>>>>>>>> in master:
> >>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
> >>>>>>>>>>>> - The metrics system
> >>>>>>>>>>>> - Kinesis Connector
> >>>>>>>>>>>> - Persistent file sources for streaming
> >>>>>>>>>>>>
> >>>>>>>>>>>> Half merged:
> >>>>>>>>>>>> - Resource manager refactoring
> >>>>>>>>>>>>
> >>>>>>>>>>>> Unmerged features:
> >>>>>>>>>>>> - Cassandra connector
> >>>>>>>>>>>> - Key groups ("rescaling from savepoints")
> >>>>>>>>>>>> - Queryable state
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm pretty sure I forgot many other features / pull requests,
> >>>>>>>
> >>>>>>> please
> >>>>>>>>>>
> >>>>>>>>>> post
> >>>>>>>>>>>>
> >>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki page
> out
> >>>>>>>
> >>>>>>> of
> >>>>>>>>>
> >>>>>>>>> it.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Some immediate TODOs for us:
> >>>>>>>>>>>> - Which of the unmerged features are we going to add to the
> >>>>>>>
> >>>>>>> release?
> >>>>>>>>>>>>
> >>>>>>>>>>>> - Which blockers do we need to address before releasing?
> >>>>>>>>>>>> - Are there any volunteers for the release manager?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Regards,
> >>>>>>>>>>>> Robert
> >>>>>>>>>>>>
> >>>>
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Ufuk Celebi <uc...@apache.org>]

Thanks for the feedback. I would like to create a preview RC0 this
week like we did for the last releases, too. In past major releases,
we needed to create many release candidates, often for fixing just
some small issues. I would like to speed up the release process by
collecting as many issues as possible now with the RC0. Once these
issues are resolved, we can start voting with the RC1. This
essentially means that we have a feature freeze. I would create the
release-1.1 branch with RC1.

Regarding the open issues:

- The savepoint fixes are not yet in. There is a PR for the savepoint
headers (#2194) and the savepoint disposal PR needs addressing of
comments.

- The Cassandra connector fixes are either merged or very close to be merged.

- Metrics docs are not a blocker since the online docs are updated
after the release. Regarding the renaming, we should decide soon. Any
opinions here?

- The YARN issues have been resolved.

I will also make a pass over JIRA and our PRs to check whether we've
missed something.

@Greg: You are right, the hash-based combine PR has been extensively
reviewed. Unfortunately, I'm not familiar with the code as I didn't
look into it and cannot tell whether we should merge it now. Is the
hash-based combine strategy activated by default or does the user has
to activate it manually? The later case would make me feel more
comfortable merging it for the release.


On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mx...@apache.org> wrote:
> Yet another one for the release: FLINK-4144
> https://github.com/apache/flink/pull/2191
>
> On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mx...@apache.org> wrote:
>> There is also FLINK-4141. We just found this during testing. PR is
>> waiting to be merged here: https://github.com/apache/flink/pull/2190
>>
>> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mx...@apache.org> wrote:
>>> FLINK-3904 is not Yarn related. Not pressing to fix for this release
>>> and actually a bit tricky to fix. I've addressed the other issues and
>>> merged all pending pull requests. Good to go from my side.
>>>
>>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <ch...@apache.org> wrote:
>>>> There are also 2 fixes for Cassandra that  should be included:
>>>> https://github.com/apache/flink/pull/2167
>>>> https://github.com/apache/flink/pull/2183
>>>>
>>>> We should also include the documentation for the metrics stuff (hopefully
>>>> merged today)
>>>> https://github.com/apache/flink/pull/2158
>>>>
>>>> In regards to metrics: To add a counter metric a user currently has to call
>>>> "counter(...)" on
>>>> a MetricGroup. The point was raised in the documentation PR that we may want
>>>> to give
>>>> them a more descriptive name like "addCounter(...)".
>>>>
>>>> I would be in favor of changing them but would like others to weigh in on
>>>> this. IMO we
>>>> should nail this down before 1.1 .
>>>>
>>>> Regards,
>>>> Chesnay
>>>>
>>>>
>>>> On 30.06.2016 22:59, Greg Hogan wrote:
>>>>>
>>>>> It would be great if hash-based combine (FLINK-3477) could make it in to
>>>>> be
>>>>> tested for this release. We've seen impressive improvements in performance
>>>>> (though, admittedly, some sort-based enhancements are yet to be worked
>>>>> on).
>>>>> This PR looks to be ripe.
>>>>>
>>>>> Also, as we tidy up a few things with Gelly and documentation, what is the
>>>>> schedule for a feature freeze and creating a 1.1 branch off master?
>>>>>
>>>>> Thanks,
>>>>> Greg
>>>>>
>>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <rm...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Sure Ufuk! Thanks a lot for taking care of the release management.
>>>>>> I'll be on vacation in three weeks, for three weeks and I'm not sure if
>>>>>> we
>>>>>> get the release done until then.
>>>>>>
>>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>>>>>
>>>>>>> I would like to do it if that's OK with you Robert. I would follow
>>>>>>> your suggestion and wait a few days until the following important
>>>>>>> fixes are in:
>>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
>>>>>>> https://github.com/apache/flink/pull/2083)
>>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
>>>>>>> - Table API time support (https://github.com/apache/flink/pull/2150)
>>>>>>> - Kafka at-least-once Producer (
>>>>>>
>>>>>> https://github.com/apache/flink/pull/2108)
>>>>>>>
>>>>>>> - Cassandra connector fixes (https://github.com/apache/flink/pull/2163)
>>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there something
>>>>>>
>>>>>> else?)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <rm...@apache.org>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>> it doesn't seem that there are volunteers for the RM, so I'll probably
>>>>>>
>>>>>> do
>>>>>>>>
>>>>>>>> it.
>>>>>>>>
>>>>>>>> I try to do the first release candidate (mostly for testing) next week
>>>>>>>
>>>>>>> (it
>>>>>>>>
>>>>>>>> depends on the JIRAs fixed by then)
>>>>>>>>
>>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
>>>>>>
>>>>>> henry.saputra@gmail.com
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for the reply, @Max. I was not aware it was about dynamic
>>>>>>>
>>>>>>> scaling,
>>>>>>>>>
>>>>>>>>> which I think also asked for YARN support.
>>>>>>>>> I agree to list all related half merge JIRA for the ResourceManager.
>>>>>>>>>
>>>>>>>>> Looking forward for the Apache Mesos integration design for sure =)
>>>>>>>>>
>>>>>>>>> - Henry
>>>>>>>>>
>>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <mx...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Robert, hi Henry,
>>>>>>>>>>
>>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features that
>>>>>>
>>>>>> justify
>>>>>>>>>>
>>>>>>>>>> a major release.
>>>>>>>>>>
>>>>>>>>>> @Henry We have plans to extend the ResourceManager to interact with
>>>>>>>>>> the Scheduler which will be a prerequisite for dynamic scaling. I
>>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos integration
>>>>>>>>>> won't require additional refactoring of the ResourceManager.
>>>>>>
>>>>>> Instead,
>>>>>>>>>>
>>>>>>>>>> we will create a new "Dispatcher" component that takes care of
>>>>>>>>>> bootstrapping the initial node with the JobManager/ResourceManager.
>>>>>>>>>>  From there on, everything will be handled by the Mesos
>>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC) who came
>>>>>>
>>>>>> up
>>>>>>>>>>
>>>>>>>>>> with this design and he plans to publish it to the mailing list
>>>>>>
>>>>>> soon.
>>>>>>>>>>
>>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged" is kind
>>>>>>
>>>>>> of
>>>>>>>>>>
>>>>>>>>>> hard to get for people who are not involved in the different
>>>>>>>>>> components.
>>>>>>>>>>
>>>>>>>>>> The Cassandra adapter seems like a pretty important thing to have
>>>>>>
>>>>>> for
>>>>>>>>>>
>>>>>>>>>> the next release. In addition, I would like to merge FLINK-3667 and
>>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those are a)
>>>>>>>>>> refactoring of the command-line and client classes b) adding
>>>>>>>>>> capability to resume cluster programmatically.
>>>>>>>>>>
>>>>>>>>>> Then we should also have a look at any other critical/major bugs
>>>>>>>
>>>>>>> listed
>>>>>>>>>
>>>>>>>>> in
>>>>>>>>>>
>>>>>>>>>> JIRA.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Max
>>>>>>>>>>
>>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
>>>>>>>
>>>>>>> henry.saputra@gmail.com
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Robert,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for staying the discussion.
>>>>>>>>>>>
>>>>>>>>>>> Do you know if there any open tasks for the Resource Manager left?
>>>>>>>>>>>
>>>>>>>>>>> That is probably needed for Mesos integration?
>>>>>>>>>>>
>>>>>>>>>>> - Henry
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
>>>>>>>
>>>>>>> rmetzger@apache.org>
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Flink 1.0.0 was released early March, so three months have passed
>>>>>>>
>>>>>>> and
>>>>>>>>>
>>>>>>>>> I
>>>>>>>>>>>>
>>>>>>>>>>>> think we should start discussing the scope of the next major
>>>>>>>
>>>>>>> release
>>>>>>>>>>>>
>>>>>>>>>>>> (1.1.0).
>>>>>>>>>>>>
>>>>>>>>>>>>  From a high level point of view, we've added the following new
>>>>>>>>>
>>>>>>>>> features:
>>>>>>>>>>>>
>>>>>>>>>>>> in master:
>>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
>>>>>>>>>>>> - The metrics system
>>>>>>>>>>>> - Kinesis Connector
>>>>>>>>>>>> - Persistent file sources for streaming
>>>>>>>>>>>>
>>>>>>>>>>>> Half merged:
>>>>>>>>>>>> - Resource manager refactoring
>>>>>>>>>>>>
>>>>>>>>>>>> Unmerged features:
>>>>>>>>>>>> - Cassandra connector
>>>>>>>>>>>> - Key groups ("rescaling from savepoints")
>>>>>>>>>>>> - Queryable state
>>>>>>>>>>>>
>>>>>>>>>>>> I'm pretty sure I forgot many other features / pull requests,
>>>>>>>
>>>>>>> please
>>>>>>>>>>
>>>>>>>>>> post
>>>>>>>>>>>>
>>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki page out
>>>>>>>
>>>>>>> of
>>>>>>>>>
>>>>>>>>> it.
>>>>>>>>>>>>
>>>>>>>>>>>> Some immediate TODOs for us:
>>>>>>>>>>>> - Which of the unmerged features are we going to add to the
>>>>>>>
>>>>>>> release?
>>>>>>>>>>>>
>>>>>>>>>>>> - Which blockers do we need to address before releasing?
>>>>>>>>>>>> - Are there any volunteers for the release manager?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Robert
>>>>>>>>>>>>
>>>>

Re: [DISCUSS] Releasing Flink 1.1.0

[Maximilian Michels <mx...@apache.org>]

Yet another one for the release: FLINK-4144
https://github.com/apache/flink/pull/2191

On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mx...@apache.org> wrote:
> There is also FLINK-4141. We just found this during testing. PR is
> waiting to be merged here: https://github.com/apache/flink/pull/2190
>
> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mx...@apache.org> wrote:
>> FLINK-3904 is not Yarn related. Not pressing to fix for this release
>> and actually a bit tricky to fix. I've addressed the other issues and
>> merged all pending pull requests. Good to go from my side.
>>
>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <ch...@apache.org> wrote:
>>> There are also 2 fixes for Cassandra that  should be included:
>>> https://github.com/apache/flink/pull/2167
>>> https://github.com/apache/flink/pull/2183
>>>
>>> We should also include the documentation for the metrics stuff (hopefully
>>> merged today)
>>> https://github.com/apache/flink/pull/2158
>>>
>>> In regards to metrics: To add a counter metric a user currently has to call
>>> "counter(...)" on
>>> a MetricGroup. The point was raised in the documentation PR that we may want
>>> to give
>>> them a more descriptive name like "addCounter(...)".
>>>
>>> I would be in favor of changing them but would like others to weigh in on
>>> this. IMO we
>>> should nail this down before 1.1 .
>>>
>>> Regards,
>>> Chesnay
>>>
>>>
>>> On 30.06.2016 22:59, Greg Hogan wrote:
>>>>
>>>> It would be great if hash-based combine (FLINK-3477) could make it in to
>>>> be
>>>> tested for this release. We've seen impressive improvements in performance
>>>> (though, admittedly, some sort-based enhancements are yet to be worked
>>>> on).
>>>> This PR looks to be ripe.
>>>>
>>>> Also, as we tidy up a few things with Gelly and documentation, what is the
>>>> schedule for a feature freeze and creating a 1.1 branch off master?
>>>>
>>>> Thanks,
>>>> Greg
>>>>
>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <rm...@apache.org>
>>>> wrote:
>>>>
>>>>> Sure Ufuk! Thanks a lot for taking care of the release management.
>>>>> I'll be on vacation in three weeks, for three weeks and I'm not sure if
>>>>> we
>>>>> get the release done until then.
>>>>>
>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>>>>
>>>>>> I would like to do it if that's OK with you Robert. I would follow
>>>>>> your suggestion and wait a few days until the following important
>>>>>> fixes are in:
>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
>>>>>> https://github.com/apache/flink/pull/2083)
>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
>>>>>> - Table API time support (https://github.com/apache/flink/pull/2150)
>>>>>> - Kafka at-least-once Producer (
>>>>>
>>>>> https://github.com/apache/flink/pull/2108)
>>>>>>
>>>>>> - Cassandra connector fixes (https://github.com/apache/flink/pull/2163)
>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there something
>>>>>
>>>>> else?)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <rm...@apache.org>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>> it doesn't seem that there are volunteers for the RM, so I'll probably
>>>>>
>>>>> do
>>>>>>>
>>>>>>> it.
>>>>>>>
>>>>>>> I try to do the first release candidate (mostly for testing) next week
>>>>>>
>>>>>> (it
>>>>>>>
>>>>>>> depends on the JIRAs fixed by then)
>>>>>>>
>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
>>>>>
>>>>> henry.saputra@gmail.com
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks for the reply, @Max. I was not aware it was about dynamic
>>>>>>
>>>>>> scaling,
>>>>>>>>
>>>>>>>> which I think also asked for YARN support.
>>>>>>>> I agree to list all related half merge JIRA for the ResourceManager.
>>>>>>>>
>>>>>>>> Looking forward for the Apache Mesos integration design for sure =)
>>>>>>>>
>>>>>>>> - Henry
>>>>>>>>
>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <mx...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Robert, hi Henry,
>>>>>>>>>
>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features that
>>>>>
>>>>> justify
>>>>>>>>>
>>>>>>>>> a major release.
>>>>>>>>>
>>>>>>>>> @Henry We have plans to extend the ResourceManager to interact with
>>>>>>>>> the Scheduler which will be a prerequisite for dynamic scaling. I
>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos integration
>>>>>>>>> won't require additional refactoring of the ResourceManager.
>>>>>
>>>>> Instead,
>>>>>>>>>
>>>>>>>>> we will create a new "Dispatcher" component that takes care of
>>>>>>>>> bootstrapping the initial node with the JobManager/ResourceManager.
>>>>>>>>>  From there on, everything will be handled by the Mesos
>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC) who came
>>>>>
>>>>> up
>>>>>>>>>
>>>>>>>>> with this design and he plans to publish it to the mailing list
>>>>>
>>>>> soon.
>>>>>>>>>
>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged" is kind
>>>>>
>>>>> of
>>>>>>>>>
>>>>>>>>> hard to get for people who are not involved in the different
>>>>>>>>> components.
>>>>>>>>>
>>>>>>>>> The Cassandra adapter seems like a pretty important thing to have
>>>>>
>>>>> for
>>>>>>>>>
>>>>>>>>> the next release. In addition, I would like to merge FLINK-3667 and
>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those are a)
>>>>>>>>> refactoring of the command-line and client classes b) adding
>>>>>>>>> capability to resume cluster programmatically.
>>>>>>>>>
>>>>>>>>> Then we should also have a look at any other critical/major bugs
>>>>>>
>>>>>> listed
>>>>>>>>
>>>>>>>> in
>>>>>>>>>
>>>>>>>>> JIRA.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Max
>>>>>>>>>
>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
>>>>>>
>>>>>> henry.saputra@gmail.com
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Robert,
>>>>>>>>>>
>>>>>>>>>> Thanks for staying the discussion.
>>>>>>>>>>
>>>>>>>>>> Do you know if there any open tasks for the Resource Manager left?
>>>>>>>>>>
>>>>>>>>>> That is probably needed for Mesos integration?
>>>>>>>>>>
>>>>>>>>>> - Henry
>>>>>>>>>>
>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
>>>>>>
>>>>>> rmetzger@apache.org>
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Flink 1.0.0 was released early March, so three months have passed
>>>>>>
>>>>>> and
>>>>>>>>
>>>>>>>> I
>>>>>>>>>>>
>>>>>>>>>>> think we should start discussing the scope of the next major
>>>>>>
>>>>>> release
>>>>>>>>>>>
>>>>>>>>>>> (1.1.0).
>>>>>>>>>>>
>>>>>>>>>>>  From a high level point of view, we've added the following new
>>>>>>>>
>>>>>>>> features:
>>>>>>>>>>>
>>>>>>>>>>> in master:
>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
>>>>>>>>>>> - The metrics system
>>>>>>>>>>> - Kinesis Connector
>>>>>>>>>>> - Persistent file sources for streaming
>>>>>>>>>>>
>>>>>>>>>>> Half merged:
>>>>>>>>>>> - Resource manager refactoring
>>>>>>>>>>>
>>>>>>>>>>> Unmerged features:
>>>>>>>>>>> - Cassandra connector
>>>>>>>>>>> - Key groups ("rescaling from savepoints")
>>>>>>>>>>> - Queryable state
>>>>>>>>>>>
>>>>>>>>>>> I'm pretty sure I forgot many other features / pull requests,
>>>>>>
>>>>>> please
>>>>>>>>>
>>>>>>>>> post
>>>>>>>>>>>
>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki page out
>>>>>>
>>>>>> of
>>>>>>>>
>>>>>>>> it.
>>>>>>>>>>>
>>>>>>>>>>> Some immediate TODOs for us:
>>>>>>>>>>> - Which of the unmerged features are we going to add to the
>>>>>>
>>>>>> release?
>>>>>>>>>>>
>>>>>>>>>>> - Which blockers do we need to address before releasing?
>>>>>>>>>>> - Are there any volunteers for the release manager?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Robert
>>>>>>>>>>>
>>>

Re: [DISCUSS] Releasing Flink 1.1.0

[Maximilian Michels <mx...@apache.org>]

There is also FLINK-4141. We just found this during testing. PR is
waiting to be merged here: https://github.com/apache/flink/pull/2190

On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mx...@apache.org> wrote:
> FLINK-3904 is not Yarn related. Not pressing to fix for this release
> and actually a bit tricky to fix. I've addressed the other issues and
> merged all pending pull requests. Good to go from my side.
>
> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <ch...@apache.org> wrote:
>> There are also 2 fixes for Cassandra that  should be included:
>> https://github.com/apache/flink/pull/2167
>> https://github.com/apache/flink/pull/2183
>>
>> We should also include the documentation for the metrics stuff (hopefully
>> merged today)
>> https://github.com/apache/flink/pull/2158
>>
>> In regards to metrics: To add a counter metric a user currently has to call
>> "counter(...)" on
>> a MetricGroup. The point was raised in the documentation PR that we may want
>> to give
>> them a more descriptive name like "addCounter(...)".
>>
>> I would be in favor of changing them but would like others to weigh in on
>> this. IMO we
>> should nail this down before 1.1 .
>>
>> Regards,
>> Chesnay
>>
>>
>> On 30.06.2016 22:59, Greg Hogan wrote:
>>>
>>> It would be great if hash-based combine (FLINK-3477) could make it in to
>>> be
>>> tested for this release. We've seen impressive improvements in performance
>>> (though, admittedly, some sort-based enhancements are yet to be worked
>>> on).
>>> This PR looks to be ripe.
>>>
>>> Also, as we tidy up a few things with Gelly and documentation, what is the
>>> schedule for a feature freeze and creating a 1.1 branch off master?
>>>
>>> Thanks,
>>> Greg
>>>
>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <rm...@apache.org>
>>> wrote:
>>>
>>>> Sure Ufuk! Thanks a lot for taking care of the release management.
>>>> I'll be on vacation in three weeks, for three weeks and I'm not sure if
>>>> we
>>>> get the release done until then.
>>>>
>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>>>
>>>>> I would like to do it if that's OK with you Robert. I would follow
>>>>> your suggestion and wait a few days until the following important
>>>>> fixes are in:
>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
>>>>> https://github.com/apache/flink/pull/2083)
>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
>>>>> - Table API time support (https://github.com/apache/flink/pull/2150)
>>>>> - Kafka at-least-once Producer (
>>>>
>>>> https://github.com/apache/flink/pull/2108)
>>>>>
>>>>> - Cassandra connector fixes (https://github.com/apache/flink/pull/2163)
>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there something
>>>>
>>>> else?)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <rm...@apache.org>
>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> it doesn't seem that there are volunteers for the RM, so I'll probably
>>>>
>>>> do
>>>>>>
>>>>>> it.
>>>>>>
>>>>>> I try to do the first release candidate (mostly for testing) next week
>>>>>
>>>>> (it
>>>>>>
>>>>>> depends on the JIRAs fixed by then)
>>>>>>
>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
>>>>
>>>> henry.saputra@gmail.com
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for the reply, @Max. I was not aware it was about dynamic
>>>>>
>>>>> scaling,
>>>>>>>
>>>>>>> which I think also asked for YARN support.
>>>>>>> I agree to list all related half merge JIRA for the ResourceManager.
>>>>>>>
>>>>>>> Looking forward for the Apache Mesos integration design for sure =)
>>>>>>>
>>>>>>> - Henry
>>>>>>>
>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <mx...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Robert, hi Henry,
>>>>>>>>
>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features that
>>>>
>>>> justify
>>>>>>>>
>>>>>>>> a major release.
>>>>>>>>
>>>>>>>> @Henry We have plans to extend the ResourceManager to interact with
>>>>>>>> the Scheduler which will be a prerequisite for dynamic scaling. I
>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos integration
>>>>>>>> won't require additional refactoring of the ResourceManager.
>>>>
>>>> Instead,
>>>>>>>>
>>>>>>>> we will create a new "Dispatcher" component that takes care of
>>>>>>>> bootstrapping the initial node with the JobManager/ResourceManager.
>>>>>>>>  From there on, everything will be handled by the Mesos
>>>>>>>> ResourceManager. I recently discussed this with Eron (CC) who came
>>>>
>>>> up
>>>>>>>>
>>>>>>>> with this design and he plans to publish it to the mailing list
>>>>
>>>> soon.
>>>>>>>>
>>>>>>>> How about listing relevant JIRA issues here? "Half Merged" is kind
>>>>
>>>> of
>>>>>>>>
>>>>>>>> hard to get for people who are not involved in the different
>>>>>>>> components.
>>>>>>>>
>>>>>>>> The Cassandra adapter seems like a pretty important thing to have
>>>>
>>>> for
>>>>>>>>
>>>>>>>> the next release. In addition, I would like to merge FLINK-3667 and
>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those are a)
>>>>>>>> refactoring of the command-line and client classes b) adding
>>>>>>>> capability to resume cluster programmatically.
>>>>>>>>
>>>>>>>> Then we should also have a look at any other critical/major bugs
>>>>>
>>>>> listed
>>>>>>>
>>>>>>> in
>>>>>>>>
>>>>>>>> JIRA.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Max
>>>>>>>>
>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
>>>>>
>>>>> henry.saputra@gmail.com
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Robert,
>>>>>>>>>
>>>>>>>>> Thanks for staying the discussion.
>>>>>>>>>
>>>>>>>>> Do you know if there any open tasks for the Resource Manager left?
>>>>>>>>>
>>>>>>>>> That is probably needed for Mesos integration?
>>>>>>>>>
>>>>>>>>> - Henry
>>>>>>>>>
>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
>>>>>
>>>>> rmetzger@apache.org>
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Flink 1.0.0 was released early March, so three months have passed
>>>>>
>>>>> and
>>>>>>>
>>>>>>> I
>>>>>>>>>>
>>>>>>>>>> think we should start discussing the scope of the next major
>>>>>
>>>>> release
>>>>>>>>>>
>>>>>>>>>> (1.1.0).
>>>>>>>>>>
>>>>>>>>>>  From a high level point of view, we've added the following new
>>>>>>>
>>>>>>> features:
>>>>>>>>>>
>>>>>>>>>> in master:
>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
>>>>>>>>>> - The metrics system
>>>>>>>>>> - Kinesis Connector
>>>>>>>>>> - Persistent file sources for streaming
>>>>>>>>>>
>>>>>>>>>> Half merged:
>>>>>>>>>> - Resource manager refactoring
>>>>>>>>>>
>>>>>>>>>> Unmerged features:
>>>>>>>>>> - Cassandra connector
>>>>>>>>>> - Key groups ("rescaling from savepoints")
>>>>>>>>>> - Queryable state
>>>>>>>>>>
>>>>>>>>>> I'm pretty sure I forgot many other features / pull requests,
>>>>>
>>>>> please
>>>>>>>>
>>>>>>>> post
>>>>>>>>>>
>>>>>>>>>> them to this thread. I'll collect them and create a Wiki page out
>>>>>
>>>>> of
>>>>>>>
>>>>>>> it.
>>>>>>>>>>
>>>>>>>>>> Some immediate TODOs for us:
>>>>>>>>>> - Which of the unmerged features are we going to add to the
>>>>>
>>>>> release?
>>>>>>>>>>
>>>>>>>>>> - Which blockers do we need to address before releasing?
>>>>>>>>>> - Are there any volunteers for the release manager?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Robert
>>>>>>>>>>
>>

Re: [DISCUSS] Releasing Flink 1.1.0

[Maximilian Michels <mx...@apache.org>]

FLINK-3904 is not Yarn related. Not pressing to fix for this release
and actually a bit tricky to fix. I've addressed the other issues and
merged all pending pull requests. Good to go from my side.

On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <ch...@apache.org> wrote:
> There are also 2 fixes for Cassandra that  should be included:
> https://github.com/apache/flink/pull/2167
> https://github.com/apache/flink/pull/2183
>
> We should also include the documentation for the metrics stuff (hopefully
> merged today)
> https://github.com/apache/flink/pull/2158
>
> In regards to metrics: To add a counter metric a user currently has to call
> "counter(...)" on
> a MetricGroup. The point was raised in the documentation PR that we may want
> to give
> them a more descriptive name like "addCounter(...)".
>
> I would be in favor of changing them but would like others to weigh in on
> this. IMO we
> should nail this down before 1.1 .
>
> Regards,
> Chesnay
>
>
> On 30.06.2016 22:59, Greg Hogan wrote:
>>
>> It would be great if hash-based combine (FLINK-3477) could make it in to
>> be
>> tested for this release. We've seen impressive improvements in performance
>> (though, admittedly, some sort-based enhancements are yet to be worked
>> on).
>> This PR looks to be ripe.
>>
>> Also, as we tidy up a few things with Gelly and documentation, what is the
>> schedule for a feature freeze and creating a 1.1 branch off master?
>>
>> Thanks,
>> Greg
>>
>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <rm...@apache.org>
>> wrote:
>>
>>> Sure Ufuk! Thanks a lot for taking care of the release management.
>>> I'll be on vacation in three weeks, for three weeks and I'm not sure if
>>> we
>>> get the release done until then.
>>>
>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>>
>>>> I would like to do it if that's OK with you Robert. I would follow
>>>> your suggestion and wait a few days until the following important
>>>> fixes are in:
>>>> - Savepoint headers and proper disposal (FLINK-4067 and
>>>> https://github.com/apache/flink/pull/2083)
>>>> - Metrics (https://github.com/apache/flink/pull/2146)
>>>> - Table API time support (https://github.com/apache/flink/pull/2150)
>>>> - Kafka at-least-once Producer (
>>>
>>> https://github.com/apache/flink/pull/2108)
>>>>
>>>> - Cassandra connector fixes (https://github.com/apache/flink/pull/2163)
>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there something
>>>
>>> else?)
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <rm...@apache.org>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>> it doesn't seem that there are volunteers for the RM, so I'll probably
>>>
>>> do
>>>>>
>>>>> it.
>>>>>
>>>>> I try to do the first release candidate (mostly for testing) next week
>>>>
>>>> (it
>>>>>
>>>>> depends on the JIRAs fixed by then)
>>>>>
>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
>>>
>>> henry.saputra@gmail.com
>>>>>
>>>>> wrote:
>>>>>
>>>>>> Thanks for the reply, @Max. I was not aware it was about dynamic
>>>>
>>>> scaling,
>>>>>>
>>>>>> which I think also asked for YARN support.
>>>>>> I agree to list all related half merge JIRA for the ResourceManager.
>>>>>>
>>>>>> Looking forward for the Apache Mesos integration design for sure =)
>>>>>>
>>>>>> - Henry
>>>>>>
>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <mx...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Robert, hi Henry,
>>>>>>>
>>>>>>> +1 for a 1.1.0 release soon! We have enough new features that
>>>
>>> justify
>>>>>>>
>>>>>>> a major release.
>>>>>>>
>>>>>>> @Henry We have plans to extend the ResourceManager to interact with
>>>>>>> the Scheduler which will be a prerequisite for dynamic scaling. I
>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos integration
>>>>>>> won't require additional refactoring of the ResourceManager.
>>>
>>> Instead,
>>>>>>>
>>>>>>> we will create a new "Dispatcher" component that takes care of
>>>>>>> bootstrapping the initial node with the JobManager/ResourceManager.
>>>>>>>  From there on, everything will be handled by the Mesos
>>>>>>> ResourceManager. I recently discussed this with Eron (CC) who came
>>>
>>> up
>>>>>>>
>>>>>>> with this design and he plans to publish it to the mailing list
>>>
>>> soon.
>>>>>>>
>>>>>>> How about listing relevant JIRA issues here? "Half Merged" is kind
>>>
>>> of
>>>>>>>
>>>>>>> hard to get for people who are not involved in the different
>>>>>>> components.
>>>>>>>
>>>>>>> The Cassandra adapter seems like a pretty important thing to have
>>>
>>> for
>>>>>>>
>>>>>>> the next release. In addition, I would like to merge FLINK-3667 and
>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those are a)
>>>>>>> refactoring of the command-line and client classes b) adding
>>>>>>> capability to resume cluster programmatically.
>>>>>>>
>>>>>>> Then we should also have a look at any other critical/major bugs
>>>>
>>>> listed
>>>>>>
>>>>>> in
>>>>>>>
>>>>>>> JIRA.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Max
>>>>>>>
>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
>>>>
>>>> henry.saputra@gmail.com
>>>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Robert,
>>>>>>>>
>>>>>>>> Thanks for staying the discussion.
>>>>>>>>
>>>>>>>> Do you know if there any open tasks for the Resource Manager left?
>>>>>>>>
>>>>>>>> That is probably needed for Mesos integration?
>>>>>>>>
>>>>>>>> - Henry
>>>>>>>>
>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
>>>>
>>>> rmetzger@apache.org>
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Flink 1.0.0 was released early March, so three months have passed
>>>>
>>>> and
>>>>>>
>>>>>> I
>>>>>>>>>
>>>>>>>>> think we should start discussing the scope of the next major
>>>>
>>>> release
>>>>>>>>>
>>>>>>>>> (1.1.0).
>>>>>>>>>
>>>>>>>>>  From a high level point of view, we've added the following new
>>>>>>
>>>>>> features:
>>>>>>>>>
>>>>>>>>> in master:
>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
>>>>>>>>> - The metrics system
>>>>>>>>> - Kinesis Connector
>>>>>>>>> - Persistent file sources for streaming
>>>>>>>>>
>>>>>>>>> Half merged:
>>>>>>>>> - Resource manager refactoring
>>>>>>>>>
>>>>>>>>> Unmerged features:
>>>>>>>>> - Cassandra connector
>>>>>>>>> - Key groups ("rescaling from savepoints")
>>>>>>>>> - Queryable state
>>>>>>>>>
>>>>>>>>> I'm pretty sure I forgot many other features / pull requests,
>>>>
>>>> please
>>>>>>>
>>>>>>> post
>>>>>>>>>
>>>>>>>>> them to this thread. I'll collect them and create a Wiki page out
>>>>
>>>> of
>>>>>>
>>>>>> it.
>>>>>>>>>
>>>>>>>>> Some immediate TODOs for us:
>>>>>>>>> - Which of the unmerged features are we going to add to the
>>>>
>>>> release?
>>>>>>>>>
>>>>>>>>> - Which blockers do we need to address before releasing?
>>>>>>>>> - Are there any volunteers for the release manager?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Robert
>>>>>>>>>
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Chesnay Schepler <ch...@apache.org>]

There are also 2 fixes for Cassandra that  should be included:
https://github.com/apache/flink/pull/2167
https://github.com/apache/flink/pull/2183

We should also include the documentation for the metrics stuff 
(hopefully merged today)
https://github.com/apache/flink/pull/2158

In regards to metrics: To add a counter metric a user currently has to 
call "counter(...)" on
a MetricGroup. The point was raised in the documentation PR that we may 
want to give
them a more descriptive name like "addCounter(...)".

I would be in favor of changing them but would like others to weigh in 
on this. IMO we
should nail this down before 1.1 .

Regards,
Chesnay

On 30.06.2016 22:59, Greg Hogan wrote:
> It would be great if hash-based combine (FLINK-3477) could make it in to be
> tested for this release. We've seen impressive improvements in performance
> (though, admittedly, some sort-based enhancements are yet to be worked on).
> This PR looks to be ripe.
>
> Also, as we tidy up a few things with Gelly and documentation, what is the
> schedule for a feature freeze and creating a 1.1 branch off master?
>
> Thanks,
> Greg
>
> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <rm...@apache.org> wrote:
>
>> Sure Ufuk! Thanks a lot for taking care of the release management.
>> I'll be on vacation in three weeks, for three weeks and I'm not sure if we
>> get the release done until then.
>>
>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>
>>> I would like to do it if that's OK with you Robert. I would follow
>>> your suggestion and wait a few days until the following important
>>> fixes are in:
>>> - Savepoint headers and proper disposal (FLINK-4067 and
>>> https://github.com/apache/flink/pull/2083)
>>> - Metrics (https://github.com/apache/flink/pull/2146)
>>> - Table API time support (https://github.com/apache/flink/pull/2150)
>>> - Kafka at-least-once Producer (
>> https://github.com/apache/flink/pull/2108)
>>> - Cassandra connector fixes (https://github.com/apache/flink/pull/2163)
>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there something
>> else?)
>>>
>>>
>>>
>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <rm...@apache.org>
>>> wrote:
>>>> Hi,
>>>> it doesn't seem that there are volunteers for the RM, so I'll probably
>> do
>>>> it.
>>>>
>>>> I try to do the first release candidate (mostly for testing) next week
>>> (it
>>>> depends on the JIRAs fixed by then)
>>>>
>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
>> henry.saputra@gmail.com
>>>> wrote:
>>>>
>>>>> Thanks for the reply, @Max. I was not aware it was about dynamic
>>> scaling,
>>>>> which I think also asked for YARN support.
>>>>> I agree to list all related half merge JIRA for the ResourceManager.
>>>>>
>>>>> Looking forward for the Apache Mesos integration design for sure =)
>>>>>
>>>>> - Henry
>>>>>
>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <mx...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Robert, hi Henry,
>>>>>>
>>>>>> +1 for a 1.1.0 release soon! We have enough new features that
>> justify
>>>>>> a major release.
>>>>>>
>>>>>> @Henry We have plans to extend the ResourceManager to interact with
>>>>>> the Scheduler which will be a prerequisite for dynamic scaling. I
>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos integration
>>>>>> won't require additional refactoring of the ResourceManager.
>> Instead,
>>>>>> we will create a new "Dispatcher" component that takes care of
>>>>>> bootstrapping the initial node with the JobManager/ResourceManager.
>>>>>>  From there on, everything will be handled by the Mesos
>>>>>> ResourceManager. I recently discussed this with Eron (CC) who came
>> up
>>>>>> with this design and he plans to publish it to the mailing list
>> soon.
>>>>>> How about listing relevant JIRA issues here? "Half Merged" is kind
>> of
>>>>>> hard to get for people who are not involved in the different
>>>>>> components.
>>>>>>
>>>>>> The Cassandra adapter seems like a pretty important thing to have
>> for
>>>>>> the next release. In addition, I would like to merge FLINK-3667 and
>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those are a)
>>>>>> refactoring of the command-line and client classes b) adding
>>>>>> capability to resume cluster programmatically.
>>>>>>
>>>>>> Then we should also have a look at any other critical/major bugs
>>> listed
>>>>> in
>>>>>> JIRA.
>>>>>>
>>>>>> Cheers,
>>>>>> Max
>>>>>>
>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
>>> henry.saputra@gmail.com
>>>>>> wrote:
>>>>>>> Hi Robert,
>>>>>>>
>>>>>>> Thanks for staying the discussion.
>>>>>>>
>>>>>>> Do you know if there any open tasks for the Resource Manager left?
>>>>>>>
>>>>>>> That is probably needed for Mesos integration?
>>>>>>>
>>>>>>> - Henry
>>>>>>>
>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
>>> rmetzger@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Flink 1.0.0 was released early March, so three months have passed
>>> and
>>>>> I
>>>>>>>> think we should start discussing the scope of the next major
>>> release
>>>>>>>> (1.1.0).
>>>>>>>>
>>>>>>>>  From a high level point of view, we've added the following new
>>>>> features:
>>>>>>>> in master:
>>>>>>>> - Table API Refactoring, SQL, StreamSQL
>>>>>>>> - The metrics system
>>>>>>>> - Kinesis Connector
>>>>>>>> - Persistent file sources for streaming
>>>>>>>>
>>>>>>>> Half merged:
>>>>>>>> - Resource manager refactoring
>>>>>>>>
>>>>>>>> Unmerged features:
>>>>>>>> - Cassandra connector
>>>>>>>> - Key groups ("rescaling from savepoints")
>>>>>>>> - Queryable state
>>>>>>>>
>>>>>>>> I'm pretty sure I forgot many other features / pull requests,
>>> please
>>>>>> post
>>>>>>>> them to this thread. I'll collect them and create a Wiki page out
>>> of
>>>>> it.
>>>>>>>> Some immediate TODOs for us:
>>>>>>>> - Which of the unmerged features are we going to add to the
>>> release?
>>>>>>>> - Which blockers do we need to address before releasing?
>>>>>>>> - Are there any volunteers for the release manager?
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Robert
>>>>>>>>

Re: [DISCUSS] Releasing Flink 1.1.0

[Greg Hogan <co...@greghogan.com>]

It would be great if hash-based combine (FLINK-3477) could make it in to be
tested for this release. We've seen impressive improvements in performance
(though, admittedly, some sort-based enhancements are yet to be worked on).
This PR looks to be ripe.

Also, as we tidy up a few things with Gelly and documentation, what is the
schedule for a feature freeze and creating a 1.1 branch off master?

Thanks,
Greg

On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <rm...@apache.org> wrote:

> Sure Ufuk! Thanks a lot for taking care of the release management.
> I'll be on vacation in three weeks, for three weeks and I'm not sure if we
> get the release done until then.
>
> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
> > I would like to do it if that's OK with you Robert. I would follow
> > your suggestion and wait a few days until the following important
> > fixes are in:
> > - Savepoint headers and proper disposal (FLINK-4067 and
> > https://github.com/apache/flink/pull/2083)
> > - Metrics (https://github.com/apache/flink/pull/2146)
> > - Table API time support (https://github.com/apache/flink/pull/2150)
> > - Kafka at-least-once Producer (
> https://github.com/apache/flink/pull/2108)
> > - Cassandra connector fixes (https://github.com/apache/flink/pull/2163)
> > - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there something
> else?)
> >
> >
> >
> >
> > On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <rm...@apache.org>
> > wrote:
> > > Hi,
> > > it doesn't seem that there are volunteers for the RM, so I'll probably
> do
> > > it.
> > >
> > > I try to do the first release candidate (mostly for testing) next week
> > (it
> > > depends on the JIRAs fixed by then)
> > >
> > > On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
> henry.saputra@gmail.com
> > >
> > > wrote:
> > >
> > >> Thanks for the reply, @Max. I was not aware it was about dynamic
> > scaling,
> > >> which I think also asked for YARN support.
> > >> I agree to list all related half merge JIRA for the ResourceManager.
> > >>
> > >> Looking forward for the Apache Mesos integration design for sure =)
> > >>
> > >> - Henry
> > >>
> > >> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <mx...@apache.org>
> > >> wrote:
> > >>
> > >> > Hi Robert, hi Henry,
> > >> >
> > >> > +1 for a 1.1.0 release soon! We have enough new features that
> justify
> > >> > a major release.
> > >> >
> > >> > @Henry We have plans to extend the ResourceManager to interact with
> > >> > the Scheduler which will be a prerequisite for dynamic scaling. I
> > >> > think this is out of scope for 1.1.0. The upcoming Mesos integration
> > >> > won't require additional refactoring of the ResourceManager.
> Instead,
> > >> > we will create a new "Dispatcher" component that takes care of
> > >> > bootstrapping the initial node with the JobManager/ResourceManager.
> > >> > From there on, everything will be handled by the Mesos
> > >> > ResourceManager. I recently discussed this with Eron (CC) who came
> up
> > >> > with this design and he plans to publish it to the mailing list
> soon.
> > >> >
> > >> > How about listing relevant JIRA issues here? "Half Merged" is kind
> of
> > >> > hard to get for people who are not involved in the different
> > >> > components.
> > >> >
> > >> > The Cassandra adapter seems like a pretty important thing to have
> for
> > >> > the next release. In addition, I would like to merge FLINK-3667 and
> > >> > FLINK-3937. Robert is doing a review at the moment :) Those are a)
> > >> > refactoring of the command-line and client classes b) adding
> > >> > capability to resume cluster programmatically.
> > >> >
> > >> > Then we should also have a look at any other critical/major bugs
> > listed
> > >> in
> > >> > JIRA.
> > >> >
> > >> > Cheers,
> > >> > Max
> > >> >
> > >> > On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> > henry.saputra@gmail.com
> > >> >
> > >> > wrote:
> > >> > > Hi Robert,
> > >> > >
> > >> > > Thanks for staying the discussion.
> > >> > >
> > >> > > Do you know if there any open tasks for the Resource Manager left?
> > >> > >
> > >> > > That is probably needed for Mesos integration?
> > >> > >
> > >> > > - Henry
> > >> > >
> > >> > > On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> > rmetzger@apache.org>
> > >> > > wrote:
> > >> > >
> > >> > >> Hi,
> > >> > >>
> > >> > >> Flink 1.0.0 was released early March, so three months have passed
> > and
> > >> I
> > >> > >> think we should start discussing the scope of the next major
> > release
> > >> > >> (1.1.0).
> > >> > >>
> > >> > >> From a high level point of view, we've added the following new
> > >> features:
> > >> > >> in master:
> > >> > >> - Table API Refactoring, SQL, StreamSQL
> > >> > >> - The metrics system
> > >> > >> - Kinesis Connector
> > >> > >> - Persistent file sources for streaming
> > >> > >>
> > >> > >> Half merged:
> > >> > >> - Resource manager refactoring
> > >> > >>
> > >> > >> Unmerged features:
> > >> > >> - Cassandra connector
> > >> > >> - Key groups ("rescaling from savepoints")
> > >> > >> - Queryable state
> > >> > >>
> > >> > >> I'm pretty sure I forgot many other features / pull requests,
> > please
> > >> > post
> > >> > >> them to this thread. I'll collect them and create a Wiki page out
> > of
> > >> it.
> > >> > >>
> > >> > >> Some immediate TODOs for us:
> > >> > >> - Which of the unmerged features are we going to add to the
> > release?
> > >> > >> - Which blockers do we need to address before releasing?
> > >> > >> - Are there any volunteers for the release manager?
> > >> > >>
> > >> > >>
> > >> > >> Regards,
> > >> > >> Robert
> > >> > >>
> > >> >
> > >>
> >
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Robert Metzger <rm...@apache.org>]

Sure Ufuk! Thanks a lot for taking care of the release management.
I'll be on vacation in three weeks, for three weeks and I'm not sure if we
get the release done until then.

On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uc...@apache.org> wrote:

> I would like to do it if that's OK with you Robert. I would follow
> your suggestion and wait a few days until the following important
> fixes are in:
> - Savepoint headers and proper disposal (FLINK-4067 and
> https://github.com/apache/flink/pull/2083)
> - Metrics (https://github.com/apache/flink/pull/2146)
> - Table API time support (https://github.com/apache/flink/pull/2150)
> - Kafka at-least-once Producer (https://github.com/apache/flink/pull/2108)
> - Cassandra connector fixes (https://github.com/apache/flink/pull/2163)
> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there something else?)
>
>
>
>
> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <rm...@apache.org>
> wrote:
> > Hi,
> > it doesn't seem that there are volunteers for the RM, so I'll probably do
> > it.
> >
> > I try to do the first release candidate (mostly for testing) next week
> (it
> > depends on the JIRAs fixed by then)
> >
> > On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <henry.saputra@gmail.com
> >
> > wrote:
> >
> >> Thanks for the reply, @Max. I was not aware it was about dynamic
> scaling,
> >> which I think also asked for YARN support.
> >> I agree to list all related half merge JIRA for the ResourceManager.
> >>
> >> Looking forward for the Apache Mesos integration design for sure =)
> >>
> >> - Henry
> >>
> >> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <mx...@apache.org>
> >> wrote:
> >>
> >> > Hi Robert, hi Henry,
> >> >
> >> > +1 for a 1.1.0 release soon! We have enough new features that justify
> >> > a major release.
> >> >
> >> > @Henry We have plans to extend the ResourceManager to interact with
> >> > the Scheduler which will be a prerequisite for dynamic scaling. I
> >> > think this is out of scope for 1.1.0. The upcoming Mesos integration
> >> > won't require additional refactoring of the ResourceManager. Instead,
> >> > we will create a new "Dispatcher" component that takes care of
> >> > bootstrapping the initial node with the JobManager/ResourceManager.
> >> > From there on, everything will be handled by the Mesos
> >> > ResourceManager. I recently discussed this with Eron (CC) who came up
> >> > with this design and he plans to publish it to the mailing list soon.
> >> >
> >> > How about listing relevant JIRA issues here? "Half Merged" is kind of
> >> > hard to get for people who are not involved in the different
> >> > components.
> >> >
> >> > The Cassandra adapter seems like a pretty important thing to have for
> >> > the next release. In addition, I would like to merge FLINK-3667 and
> >> > FLINK-3937. Robert is doing a review at the moment :) Those are a)
> >> > refactoring of the command-line and client classes b) adding
> >> > capability to resume cluster programmatically.
> >> >
> >> > Then we should also have a look at any other critical/major bugs
> listed
> >> in
> >> > JIRA.
> >> >
> >> > Cheers,
> >> > Max
> >> >
> >> > On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> henry.saputra@gmail.com
> >> >
> >> > wrote:
> >> > > Hi Robert,
> >> > >
> >> > > Thanks for staying the discussion.
> >> > >
> >> > > Do you know if there any open tasks for the Resource Manager left?
> >> > >
> >> > > That is probably needed for Mesos integration?
> >> > >
> >> > > - Henry
> >> > >
> >> > > On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> rmetzger@apache.org>
> >> > > wrote:
> >> > >
> >> > >> Hi,
> >> > >>
> >> > >> Flink 1.0.0 was released early March, so three months have passed
> and
> >> I
> >> > >> think we should start discussing the scope of the next major
> release
> >> > >> (1.1.0).
> >> > >>
> >> > >> From a high level point of view, we've added the following new
> >> features:
> >> > >> in master:
> >> > >> - Table API Refactoring, SQL, StreamSQL
> >> > >> - The metrics system
> >> > >> - Kinesis Connector
> >> > >> - Persistent file sources for streaming
> >> > >>
> >> > >> Half merged:
> >> > >> - Resource manager refactoring
> >> > >>
> >> > >> Unmerged features:
> >> > >> - Cassandra connector
> >> > >> - Key groups ("rescaling from savepoints")
> >> > >> - Queryable state
> >> > >>
> >> > >> I'm pretty sure I forgot many other features / pull requests,
> please
> >> > post
> >> > >> them to this thread. I'll collect them and create a Wiki page out
> of
> >> it.
> >> > >>
> >> > >> Some immediate TODOs for us:
> >> > >> - Which of the unmerged features are we going to add to the
> release?
> >> > >> - Which blockers do we need to address before releasing?
> >> > >> - Are there any volunteers for the release manager?
> >> > >>
> >> > >>
> >> > >> Regards,
> >> > >> Robert
> >> > >>
> >> >
> >>
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Ufuk Celebi <uc...@apache.org>]

I would like to do it if that's OK with you Robert. I would follow
your suggestion and wait a few days until the following important
fixes are in:
- Savepoint headers and proper disposal (FLINK-4067 and
https://github.com/apache/flink/pull/2083)
- Metrics (https://github.com/apache/flink/pull/2146)
- Table API time support (https://github.com/apache/flink/pull/2150)
- Kafka at-least-once Producer (https://github.com/apache/flink/pull/2108)
- Cassandra connector fixes (https://github.com/apache/flink/pull/2163)
- YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there something else?)




On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <rm...@apache.org> wrote:
> Hi,
> it doesn't seem that there are volunteers for the RM, so I'll probably do
> it.
>
> I try to do the first release candidate (mostly for testing) next week (it
> depends on the JIRAs fixed by then)
>
> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <he...@gmail.com>
> wrote:
>
>> Thanks for the reply, @Max. I was not aware it was about dynamic scaling,
>> which I think also asked for YARN support.
>> I agree to list all related half merge JIRA for the ResourceManager.
>>
>> Looking forward for the Apache Mesos integration design for sure =)
>>
>> - Henry
>>
>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <mx...@apache.org>
>> wrote:
>>
>> > Hi Robert, hi Henry,
>> >
>> > +1 for a 1.1.0 release soon! We have enough new features that justify
>> > a major release.
>> >
>> > @Henry We have plans to extend the ResourceManager to interact with
>> > the Scheduler which will be a prerequisite for dynamic scaling. I
>> > think this is out of scope for 1.1.0. The upcoming Mesos integration
>> > won't require additional refactoring of the ResourceManager. Instead,
>> > we will create a new "Dispatcher" component that takes care of
>> > bootstrapping the initial node with the JobManager/ResourceManager.
>> > From there on, everything will be handled by the Mesos
>> > ResourceManager. I recently discussed this with Eron (CC) who came up
>> > with this design and he plans to publish it to the mailing list soon.
>> >
>> > How about listing relevant JIRA issues here? "Half Merged" is kind of
>> > hard to get for people who are not involved in the different
>> > components.
>> >
>> > The Cassandra adapter seems like a pretty important thing to have for
>> > the next release. In addition, I would like to merge FLINK-3667 and
>> > FLINK-3937. Robert is doing a review at the moment :) Those are a)
>> > refactoring of the command-line and client classes b) adding
>> > capability to resume cluster programmatically.
>> >
>> > Then we should also have a look at any other critical/major bugs listed
>> in
>> > JIRA.
>> >
>> > Cheers,
>> > Max
>> >
>> > On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <henry.saputra@gmail.com
>> >
>> > wrote:
>> > > Hi Robert,
>> > >
>> > > Thanks for staying the discussion.
>> > >
>> > > Do you know if there any open tasks for the Resource Manager left?
>> > >
>> > > That is probably needed for Mesos integration?
>> > >
>> > > - Henry
>> > >
>> > > On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <rm...@apache.org>
>> > > wrote:
>> > >
>> > >> Hi,
>> > >>
>> > >> Flink 1.0.0 was released early March, so three months have passed and
>> I
>> > >> think we should start discussing the scope of the next major release
>> > >> (1.1.0).
>> > >>
>> > >> From a high level point of view, we've added the following new
>> features:
>> > >> in master:
>> > >> - Table API Refactoring, SQL, StreamSQL
>> > >> - The metrics system
>> > >> - Kinesis Connector
>> > >> - Persistent file sources for streaming
>> > >>
>> > >> Half merged:
>> > >> - Resource manager refactoring
>> > >>
>> > >> Unmerged features:
>> > >> - Cassandra connector
>> > >> - Key groups ("rescaling from savepoints")
>> > >> - Queryable state
>> > >>
>> > >> I'm pretty sure I forgot many other features / pull requests, please
>> > post
>> > >> them to this thread. I'll collect them and create a Wiki page out of
>> it.
>> > >>
>> > >> Some immediate TODOs for us:
>> > >> - Which of the unmerged features are we going to add to the release?
>> > >> - Which blockers do we need to address before releasing?
>> > >> - Are there any volunteers for the release manager?
>> > >>
>> > >>
>> > >> Regards,
>> > >> Robert
>> > >>
>> >
>>

Re: [DISCUSS] Releasing Flink 1.1.0

[Robert Metzger <rm...@apache.org>]

Hi,
it doesn't seem that there are volunteers for the RM, so I'll probably do
it.

I try to do the first release candidate (mostly for testing) next week (it
depends on the JIRAs fixed by then)

On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <he...@gmail.com>
wrote:

> Thanks for the reply, @Max. I was not aware it was about dynamic scaling,
> which I think also asked for YARN support.
> I agree to list all related half merge JIRA for the ResourceManager.
>
> Looking forward for the Apache Mesos integration design for sure =)
>
> - Henry
>
> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <mx...@apache.org>
> wrote:
>
> > Hi Robert, hi Henry,
> >
> > +1 for a 1.1.0 release soon! We have enough new features that justify
> > a major release.
> >
> > @Henry We have plans to extend the ResourceManager to interact with
> > the Scheduler which will be a prerequisite for dynamic scaling. I
> > think this is out of scope for 1.1.0. The upcoming Mesos integration
> > won't require additional refactoring of the ResourceManager. Instead,
> > we will create a new "Dispatcher" component that takes care of
> > bootstrapping the initial node with the JobManager/ResourceManager.
> > From there on, everything will be handled by the Mesos
> > ResourceManager. I recently discussed this with Eron (CC) who came up
> > with this design and he plans to publish it to the mailing list soon.
> >
> > How about listing relevant JIRA issues here? "Half Merged" is kind of
> > hard to get for people who are not involved in the different
> > components.
> >
> > The Cassandra adapter seems like a pretty important thing to have for
> > the next release. In addition, I would like to merge FLINK-3667 and
> > FLINK-3937. Robert is doing a review at the moment :) Those are a)
> > refactoring of the command-line and client classes b) adding
> > capability to resume cluster programmatically.
> >
> > Then we should also have a look at any other critical/major bugs listed
> in
> > JIRA.
> >
> > Cheers,
> > Max
> >
> > On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <henry.saputra@gmail.com
> >
> > wrote:
> > > Hi Robert,
> > >
> > > Thanks for staying the discussion.
> > >
> > > Do you know if there any open tasks for the Resource Manager left?
> > >
> > > That is probably needed for Mesos integration?
> > >
> > > - Henry
> > >
> > > On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <rm...@apache.org>
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> Flink 1.0.0 was released early March, so three months have passed and
> I
> > >> think we should start discussing the scope of the next major release
> > >> (1.1.0).
> > >>
> > >> From a high level point of view, we've added the following new
> features:
> > >> in master:
> > >> - Table API Refactoring, SQL, StreamSQL
> > >> - The metrics system
> > >> - Kinesis Connector
> > >> - Persistent file sources for streaming
> > >>
> > >> Half merged:
> > >> - Resource manager refactoring
> > >>
> > >> Unmerged features:
> > >> - Cassandra connector
> > >> - Key groups ("rescaling from savepoints")
> > >> - Queryable state
> > >>
> > >> I'm pretty sure I forgot many other features / pull requests, please
> > post
> > >> them to this thread. I'll collect them and create a Wiki page out of
> it.
> > >>
> > >> Some immediate TODOs for us:
> > >> - Which of the unmerged features are we going to add to the release?
> > >> - Which blockers do we need to address before releasing?
> > >> - Are there any volunteers for the release manager?
> > >>
> > >>
> > >> Regards,
> > >> Robert
> > >>
> >
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Henry Saputra <he...@gmail.com>]

Thanks for the reply, @Max. I was not aware it was about dynamic scaling,
which I think also asked for YARN support.
I agree to list all related half merge JIRA for the ResourceManager.

Looking forward for the Apache Mesos integration design for sure =)

- Henry

On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <mx...@apache.org> wrote:

> Hi Robert, hi Henry,
>
> +1 for a 1.1.0 release soon! We have enough new features that justify
> a major release.
>
> @Henry We have plans to extend the ResourceManager to interact with
> the Scheduler which will be a prerequisite for dynamic scaling. I
> think this is out of scope for 1.1.0. The upcoming Mesos integration
> won't require additional refactoring of the ResourceManager. Instead,
> we will create a new "Dispatcher" component that takes care of
> bootstrapping the initial node with the JobManager/ResourceManager.
> From there on, everything will be handled by the Mesos
> ResourceManager. I recently discussed this with Eron (CC) who came up
> with this design and he plans to publish it to the mailing list soon.
>
> How about listing relevant JIRA issues here? "Half Merged" is kind of
> hard to get for people who are not involved in the different
> components.
>
> The Cassandra adapter seems like a pretty important thing to have for
> the next release. In addition, I would like to merge FLINK-3667 and
> FLINK-3937. Robert is doing a review at the moment :) Those are a)
> refactoring of the command-line and client classes b) adding
> capability to resume cluster programmatically.
>
> Then we should also have a look at any other critical/major bugs listed in
> JIRA.
>
> Cheers,
> Max
>
> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <he...@gmail.com>
> wrote:
> > Hi Robert,
> >
> > Thanks for staying the discussion.
> >
> > Do you know if there any open tasks for the Resource Manager left?
> >
> > That is probably needed for Mesos integration?
> >
> > - Henry
> >
> > On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <rm...@apache.org>
> > wrote:
> >
> >> Hi,
> >>
> >> Flink 1.0.0 was released early March, so three months have passed and I
> >> think we should start discussing the scope of the next major release
> >> (1.1.0).
> >>
> >> From a high level point of view, we've added the following new features:
> >> in master:
> >> - Table API Refactoring, SQL, StreamSQL
> >> - The metrics system
> >> - Kinesis Connector
> >> - Persistent file sources for streaming
> >>
> >> Half merged:
> >> - Resource manager refactoring
> >>
> >> Unmerged features:
> >> - Cassandra connector
> >> - Key groups ("rescaling from savepoints")
> >> - Queryable state
> >>
> >> I'm pretty sure I forgot many other features / pull requests, please
> post
> >> them to this thread. I'll collect them and create a Wiki page out of it.
> >>
> >> Some immediate TODOs for us:
> >> - Which of the unmerged features are we going to add to the release?
> >> - Which blockers do we need to address before releasing?
> >> - Are there any volunteers for the release manager?
> >>
> >>
> >> Regards,
> >> Robert
> >>
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Andrea Sella <an...@radicalbit.io>]

Hi Robert,

Yes, it works in the current master. Thank you.

Cheers,
Andrea

2016-06-16 14:03 GMT+02:00 Robert Metzger <rm...@apache.org>:

> @Andrea: Do I understand correctly that the scala-shell works with HA in
> the current master? If so, it will be included automatically into the
> release.
>
> I created a Wiki page for the 1.1 release:
> https://cwiki.apache.org/confluence/display/FLINK/1.1+Release
> I don't think it makes sense to copy a long list of JIRAs into the Wiki. I
> would rather prefer to properly maintain our JIRA instance.
>
> There are currently 56 unresolved JIRAs with "fixVersion" set to 1.1.
>
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.1.0%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
>
> I will go through the list of issues and try to unassign some of them from
> the release.
> Component shepherds should keep an eye on their component's issues and set
> the fixVersion accordingly.
>
> Regards,
> Robert
>
> On Thu, Jun 16, 2016 at 2:01 PM, Vishnu Viswanath <
> vishnu.viswanath25@gmail.com> wrote:
>
> > Ok, Makes sense. Thanks Fabian!
> >
> > On Thu, Jun 16, 2016 at 7:27 AM, Fabian Hueske <fh...@gmail.com>
> wrote:
> >
> > > Hi Vishnu,
> > >
> > > no, queryable state would not be part of the 1.1 release according to
> > > Robert's proposal.
> > >
> > > IMO, adding a major feature like queryable state shortly before a
> release
> > > is not a good idea and likely to delay the release or cause problems
> > later.
> > > Major features need a bit of exposure in the master branch before
> > including
> > > them in a release.
> > >
> > > The open PR for queryable state (#2051) should be merged after the
> > release
> > > branch was forked off.
> > >
> > > Best, Fabian
> > >
> > > 2016-06-16 13:18 GMT+02:00 Vishnu Viswanath <
> > vishnu.viswanath25@gmail.com
> > > >:
> > >
> > > > Hi,
> > > >
> > > > Will queryable state available in 1.1.0?
> > > >
> > > > Thanks,
> > > > Vishnu
> > > >
> > > > On Thursday, 16 June 2016, Aljoscha Krettek <al...@apache.org>
> > wrote:
> > > >
> > > > > Hi,
> > > > > +1 for releasing what we have now and making it rock solid.
> > > > >
> > > > > The support for merging windows (session windows) is also new, btw.
> > :-)
> > > > >
> > > > > Cheers,
> > > > > Aljoscha
> > > > >
> > > > > On Thu, 16 Jun 2016 at 11:12 Maximilian Michels <mxm@apache.org
> > > > > <javascript:;>> wrote:
> > > > >
> > > > > > Hi Robert, hi Henry,
> > > > > >
> > > > > > +1 for a 1.1.0 release soon! We have enough new features that
> > justify
> > > > > > a major release.
> > > > > >
> > > > > > @Henry We have plans to extend the ResourceManager to interact
> with
> > > > > > the Scheduler which will be a prerequisite for dynamic scaling. I
> > > > > > think this is out of scope for 1.1.0. The upcoming Mesos
> > integration
> > > > > > won't require additional refactoring of the ResourceManager.
> > Instead,
> > > > > > we will create a new "Dispatcher" component that takes care of
> > > > > > bootstrapping the initial node with the
> JobManager/ResourceManager.
> > > > > > From there on, everything will be handled by the Mesos
> > > > > > ResourceManager. I recently discussed this with Eron (CC) who
> came
> > up
> > > > > > with this design and he plans to publish it to the mailing list
> > soon.
> > > > > >
> > > > > > How about listing relevant JIRA issues here? "Half Merged" is
> kind
> > of
> > > > > > hard to get for people who are not involved in the different
> > > > > > components.
> > > > > >
> > > > > > The Cassandra adapter seems like a pretty important thing to have
> > for
> > > > > > the next release. In addition, I would like to merge FLINK-3667
> and
> > > > > > FLINK-3937. Robert is doing a review at the moment :) Those are
> a)
> > > > > > refactoring of the command-line and client classes b) adding
> > > > > > capability to resume cluster programmatically.
> > > > > >
> > > > > > Then we should also have a look at any other critical/major bugs
> > > listed
> > > > > in
> > > > > > JIRA.
> > > > > >
> > > > > > Cheers,
> > > > > > Max
> > > > > >
> > > > > > On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> > > > henry.saputra@gmail.com
> > > > > <javascript:;>>
> > > > > > wrote:
> > > > > > > Hi Robert,
> > > > > > >
> > > > > > > Thanks for staying the discussion.
> > > > > > >
> > > > > > > Do you know if there any open tasks for the Resource Manager
> > left?
> > > > > > >
> > > > > > > That is probably needed for Mesos integration?
> > > > > > >
> > > > > > > - Henry
> > > > > > >
> > > > > > > On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> > > > rmetzger@apache.org
> > > > > <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi,
> > > > > > >>
> > > > > > >> Flink 1.0.0 was released early March, so three months have
> > passed
> > > > and
> > > > > I
> > > > > > >> think we should start discussing the scope of the next major
> > > release
> > > > > > >> (1.1.0).
> > > > > > >>
> > > > > > >> From a high level point of view, we've added the following new
> > > > > features:
> > > > > > >> in master:
> > > > > > >> - Table API Refactoring, SQL, StreamSQL
> > > > > > >> - The metrics system
> > > > > > >> - Kinesis Connector
> > > > > > >> - Persistent file sources for streaming
> > > > > > >>
> > > > > > >> Half merged:
> > > > > > >> - Resource manager refactoring
> > > > > > >>
> > > > > > >> Unmerged features:
> > > > > > >> - Cassandra connector
> > > > > > >> - Key groups ("rescaling from savepoints")
> > > > > > >> - Queryable state
> > > > > > >>
> > > > > > >> I'm pretty sure I forgot many other features / pull requests,
> > > please
> > > > > > post
> > > > > > >> them to this thread. I'll collect them and create a Wiki page
> > out
> > > of
> > > > > it.
> > > > > > >>
> > > > > > >> Some immediate TODOs for us:
> > > > > > >> - Which of the unmerged features are we going to add to the
> > > release?
> > > > > > >> - Which blockers do we need to address before releasing?
> > > > > > >> - Are there any volunteers for the release manager?
> > > > > > >>
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> Robert
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Thanks and Regards,
> > > > Vishnu Viswanath,
> > > > *www.vishnuviswanath.com <http://www.vishnuviswanath.com>*
> > > >
> > >
> >
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Robert Metzger <rm...@apache.org>]

@Andrea: Do I understand correctly that the scala-shell works with HA in
the current master? If so, it will be included automatically into the
release.

I created a Wiki page for the 1.1 release:
https://cwiki.apache.org/confluence/display/FLINK/1.1+Release
I don't think it makes sense to copy a long list of JIRAs into the Wiki. I
would rather prefer to properly maintain our JIRA instance.

There are currently 56 unresolved JIRAs with "fixVersion" set to 1.1.

https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.1.0%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC

I will go through the list of issues and try to unassign some of them from
the release.
Component shepherds should keep an eye on their component's issues and set
the fixVersion accordingly.

Regards,
Robert

On Thu, Jun 16, 2016 at 2:01 PM, Vishnu Viswanath <
vishnu.viswanath25@gmail.com> wrote:

> Ok, Makes sense. Thanks Fabian!
>
> On Thu, Jun 16, 2016 at 7:27 AM, Fabian Hueske <fh...@gmail.com> wrote:
>
> > Hi Vishnu,
> >
> > no, queryable state would not be part of the 1.1 release according to
> > Robert's proposal.
> >
> > IMO, adding a major feature like queryable state shortly before a release
> > is not a good idea and likely to delay the release or cause problems
> later.
> > Major features need a bit of exposure in the master branch before
> including
> > them in a release.
> >
> > The open PR for queryable state (#2051) should be merged after the
> release
> > branch was forked off.
> >
> > Best, Fabian
> >
> > 2016-06-16 13:18 GMT+02:00 Vishnu Viswanath <
> vishnu.viswanath25@gmail.com
> > >:
> >
> > > Hi,
> > >
> > > Will queryable state available in 1.1.0?
> > >
> > > Thanks,
> > > Vishnu
> > >
> > > On Thursday, 16 June 2016, Aljoscha Krettek <al...@apache.org>
> wrote:
> > >
> > > > Hi,
> > > > +1 for releasing what we have now and making it rock solid.
> > > >
> > > > The support for merging windows (session windows) is also new, btw.
> :-)
> > > >
> > > > Cheers,
> > > > Aljoscha
> > > >
> > > > On Thu, 16 Jun 2016 at 11:12 Maximilian Michels <mxm@apache.org
> > > > <javascript:;>> wrote:
> > > >
> > > > > Hi Robert, hi Henry,
> > > > >
> > > > > +1 for a 1.1.0 release soon! We have enough new features that
> justify
> > > > > a major release.
> > > > >
> > > > > @Henry We have plans to extend the ResourceManager to interact with
> > > > > the Scheduler which will be a prerequisite for dynamic scaling. I
> > > > > think this is out of scope for 1.1.0. The upcoming Mesos
> integration
> > > > > won't require additional refactoring of the ResourceManager.
> Instead,
> > > > > we will create a new "Dispatcher" component that takes care of
> > > > > bootstrapping the initial node with the JobManager/ResourceManager.
> > > > > From there on, everything will be handled by the Mesos
> > > > > ResourceManager. I recently discussed this with Eron (CC) who came
> up
> > > > > with this design and he plans to publish it to the mailing list
> soon.
> > > > >
> > > > > How about listing relevant JIRA issues here? "Half Merged" is kind
> of
> > > > > hard to get for people who are not involved in the different
> > > > > components.
> > > > >
> > > > > The Cassandra adapter seems like a pretty important thing to have
> for
> > > > > the next release. In addition, I would like to merge FLINK-3667 and
> > > > > FLINK-3937. Robert is doing a review at the moment :) Those are a)
> > > > > refactoring of the command-line and client classes b) adding
> > > > > capability to resume cluster programmatically.
> > > > >
> > > > > Then we should also have a look at any other critical/major bugs
> > listed
> > > > in
> > > > > JIRA.
> > > > >
> > > > > Cheers,
> > > > > Max
> > > > >
> > > > > On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> > > henry.saputra@gmail.com
> > > > <javascript:;>>
> > > > > wrote:
> > > > > > Hi Robert,
> > > > > >
> > > > > > Thanks for staying the discussion.
> > > > > >
> > > > > > Do you know if there any open tasks for the Resource Manager
> left?
> > > > > >
> > > > > > That is probably needed for Mesos integration?
> > > > > >
> > > > > > - Henry
> > > > > >
> > > > > > On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> > > rmetzger@apache.org
> > > > <javascript:;>>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> Flink 1.0.0 was released early March, so three months have
> passed
> > > and
> > > > I
> > > > > >> think we should start discussing the scope of the next major
> > release
> > > > > >> (1.1.0).
> > > > > >>
> > > > > >> From a high level point of view, we've added the following new
> > > > features:
> > > > > >> in master:
> > > > > >> - Table API Refactoring, SQL, StreamSQL
> > > > > >> - The metrics system
> > > > > >> - Kinesis Connector
> > > > > >> - Persistent file sources for streaming
> > > > > >>
> > > > > >> Half merged:
> > > > > >> - Resource manager refactoring
> > > > > >>
> > > > > >> Unmerged features:
> > > > > >> - Cassandra connector
> > > > > >> - Key groups ("rescaling from savepoints")
> > > > > >> - Queryable state
> > > > > >>
> > > > > >> I'm pretty sure I forgot many other features / pull requests,
> > please
> > > > > post
> > > > > >> them to this thread. I'll collect them and create a Wiki page
> out
> > of
> > > > it.
> > > > > >>
> > > > > >> Some immediate TODOs for us:
> > > > > >> - Which of the unmerged features are we going to add to the
> > release?
> > > > > >> - Which blockers do we need to address before releasing?
> > > > > >> - Are there any volunteers for the release manager?
> > > > > >>
> > > > > >>
> > > > > >> Regards,
> > > > > >> Robert
> > > > > >>
> > > > >
> > > >
> > >
> > >
> > > --
> > > Thanks and Regards,
> > > Vishnu Viswanath,
> > > *www.vishnuviswanath.com <http://www.vishnuviswanath.com>*
> > >
> >
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Vishnu Viswanath <vi...@gmail.com>]

Ok, Makes sense. Thanks Fabian!

On Thu, Jun 16, 2016 at 7:27 AM, Fabian Hueske <fh...@gmail.com> wrote:

> Hi Vishnu,
>
> no, queryable state would not be part of the 1.1 release according to
> Robert's proposal.
>
> IMO, adding a major feature like queryable state shortly before a release
> is not a good idea and likely to delay the release or cause problems later.
> Major features need a bit of exposure in the master branch before including
> them in a release.
>
> The open PR for queryable state (#2051) should be merged after the release
> branch was forked off.
>
> Best, Fabian
>
> 2016-06-16 13:18 GMT+02:00 Vishnu Viswanath <vishnu.viswanath25@gmail.com
> >:
>
> > Hi,
> >
> > Will queryable state available in 1.1.0?
> >
> > Thanks,
> > Vishnu
> >
> > On Thursday, 16 June 2016, Aljoscha Krettek <al...@apache.org> wrote:
> >
> > > Hi,
> > > +1 for releasing what we have now and making it rock solid.
> > >
> > > The support for merging windows (session windows) is also new, btw. :-)
> > >
> > > Cheers,
> > > Aljoscha
> > >
> > > On Thu, 16 Jun 2016 at 11:12 Maximilian Michels <mxm@apache.org
> > > <javascript:;>> wrote:
> > >
> > > > Hi Robert, hi Henry,
> > > >
> > > > +1 for a 1.1.0 release soon! We have enough new features that justify
> > > > a major release.
> > > >
> > > > @Henry We have plans to extend the ResourceManager to interact with
> > > > the Scheduler which will be a prerequisite for dynamic scaling. I
> > > > think this is out of scope for 1.1.0. The upcoming Mesos integration
> > > > won't require additional refactoring of the ResourceManager. Instead,
> > > > we will create a new "Dispatcher" component that takes care of
> > > > bootstrapping the initial node with the JobManager/ResourceManager.
> > > > From there on, everything will be handled by the Mesos
> > > > ResourceManager. I recently discussed this with Eron (CC) who came up
> > > > with this design and he plans to publish it to the mailing list soon.
> > > >
> > > > How about listing relevant JIRA issues here? "Half Merged" is kind of
> > > > hard to get for people who are not involved in the different
> > > > components.
> > > >
> > > > The Cassandra adapter seems like a pretty important thing to have for
> > > > the next release. In addition, I would like to merge FLINK-3667 and
> > > > FLINK-3937. Robert is doing a review at the moment :) Those are a)
> > > > refactoring of the command-line and client classes b) adding
> > > > capability to resume cluster programmatically.
> > > >
> > > > Then we should also have a look at any other critical/major bugs
> listed
> > > in
> > > > JIRA.
> > > >
> > > > Cheers,
> > > > Max
> > > >
> > > > On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> > henry.saputra@gmail.com
> > > <javascript:;>>
> > > > wrote:
> > > > > Hi Robert,
> > > > >
> > > > > Thanks for staying the discussion.
> > > > >
> > > > > Do you know if there any open tasks for the Resource Manager left?
> > > > >
> > > > > That is probably needed for Mesos integration?
> > > > >
> > > > > - Henry
> > > > >
> > > > > On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> > rmetzger@apache.org
> > > <javascript:;>>
> > > > > wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> Flink 1.0.0 was released early March, so three months have passed
> > and
> > > I
> > > > >> think we should start discussing the scope of the next major
> release
> > > > >> (1.1.0).
> > > > >>
> > > > >> From a high level point of view, we've added the following new
> > > features:
> > > > >> in master:
> > > > >> - Table API Refactoring, SQL, StreamSQL
> > > > >> - The metrics system
> > > > >> - Kinesis Connector
> > > > >> - Persistent file sources for streaming
> > > > >>
> > > > >> Half merged:
> > > > >> - Resource manager refactoring
> > > > >>
> > > > >> Unmerged features:
> > > > >> - Cassandra connector
> > > > >> - Key groups ("rescaling from savepoints")
> > > > >> - Queryable state
> > > > >>
> > > > >> I'm pretty sure I forgot many other features / pull requests,
> please
> > > > post
> > > > >> them to this thread. I'll collect them and create a Wiki page out
> of
> > > it.
> > > > >>
> > > > >> Some immediate TODOs for us:
> > > > >> - Which of the unmerged features are we going to add to the
> release?
> > > > >> - Which blockers do we need to address before releasing?
> > > > >> - Are there any volunteers for the release manager?
> > > > >>
> > > > >>
> > > > >> Regards,
> > > > >> Robert
> > > > >>
> > > >
> > >
> >
> >
> > --
> > Thanks and Regards,
> > Vishnu Viswanath,
> > *www.vishnuviswanath.com <http://www.vishnuviswanath.com>*
> >
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Fabian Hueske <fh...@gmail.com>]

Hi Vishnu,

no, queryable state would not be part of the 1.1 release according to
Robert's proposal.

IMO, adding a major feature like queryable state shortly before a release
is not a good idea and likely to delay the release or cause problems later.
Major features need a bit of exposure in the master branch before including
them in a release.

The open PR for queryable state (#2051) should be merged after the release
branch was forked off.

Best, Fabian

2016-06-16 13:18 GMT+02:00 Vishnu Viswanath <vi...@gmail.com>:

> Hi,
>
> Will queryable state available in 1.1.0?
>
> Thanks,
> Vishnu
>
> On Thursday, 16 June 2016, Aljoscha Krettek <al...@apache.org> wrote:
>
> > Hi,
> > +1 for releasing what we have now and making it rock solid.
> >
> > The support for merging windows (session windows) is also new, btw. :-)
> >
> > Cheers,
> > Aljoscha
> >
> > On Thu, 16 Jun 2016 at 11:12 Maximilian Michels <mxm@apache.org
> > <javascript:;>> wrote:
> >
> > > Hi Robert, hi Henry,
> > >
> > > +1 for a 1.1.0 release soon! We have enough new features that justify
> > > a major release.
> > >
> > > @Henry We have plans to extend the ResourceManager to interact with
> > > the Scheduler which will be a prerequisite for dynamic scaling. I
> > > think this is out of scope for 1.1.0. The upcoming Mesos integration
> > > won't require additional refactoring of the ResourceManager. Instead,
> > > we will create a new "Dispatcher" component that takes care of
> > > bootstrapping the initial node with the JobManager/ResourceManager.
> > > From there on, everything will be handled by the Mesos
> > > ResourceManager. I recently discussed this with Eron (CC) who came up
> > > with this design and he plans to publish it to the mailing list soon.
> > >
> > > How about listing relevant JIRA issues here? "Half Merged" is kind of
> > > hard to get for people who are not involved in the different
> > > components.
> > >
> > > The Cassandra adapter seems like a pretty important thing to have for
> > > the next release. In addition, I would like to merge FLINK-3667 and
> > > FLINK-3937. Robert is doing a review at the moment :) Those are a)
> > > refactoring of the command-line and client classes b) adding
> > > capability to resume cluster programmatically.
> > >
> > > Then we should also have a look at any other critical/major bugs listed
> > in
> > > JIRA.
> > >
> > > Cheers,
> > > Max
> > >
> > > On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
> henry.saputra@gmail.com
> > <javascript:;>>
> > > wrote:
> > > > Hi Robert,
> > > >
> > > > Thanks for staying the discussion.
> > > >
> > > > Do you know if there any open tasks for the Resource Manager left?
> > > >
> > > > That is probably needed for Mesos integration?
> > > >
> > > > - Henry
> > > >
> > > > On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
> rmetzger@apache.org
> > <javascript:;>>
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> Flink 1.0.0 was released early March, so three months have passed
> and
> > I
> > > >> think we should start discussing the scope of the next major release
> > > >> (1.1.0).
> > > >>
> > > >> From a high level point of view, we've added the following new
> > features:
> > > >> in master:
> > > >> - Table API Refactoring, SQL, StreamSQL
> > > >> - The metrics system
> > > >> - Kinesis Connector
> > > >> - Persistent file sources for streaming
> > > >>
> > > >> Half merged:
> > > >> - Resource manager refactoring
> > > >>
> > > >> Unmerged features:
> > > >> - Cassandra connector
> > > >> - Key groups ("rescaling from savepoints")
> > > >> - Queryable state
> > > >>
> > > >> I'm pretty sure I forgot many other features / pull requests, please
> > > post
> > > >> them to this thread. I'll collect them and create a Wiki page out of
> > it.
> > > >>
> > > >> Some immediate TODOs for us:
> > > >> - Which of the unmerged features are we going to add to the release?
> > > >> - Which blockers do we need to address before releasing?
> > > >> - Are there any volunteers for the release manager?
> > > >>
> > > >>
> > > >> Regards,
> > > >> Robert
> > > >>
> > >
> >
>
>
> --
> Thanks and Regards,
> Vishnu Viswanath,
> *www.vishnuviswanath.com <http://www.vishnuviswanath.com>*
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Vishnu Viswanath <vi...@gmail.com>]

Hi,

Will queryable state available in 1.1.0?

Thanks,
Vishnu

On Thursday, 16 June 2016, Aljoscha Krettek <al...@apache.org> wrote:

> Hi,
> +1 for releasing what we have now and making it rock solid.
>
> The support for merging windows (session windows) is also new, btw. :-)
>
> Cheers,
> Aljoscha
>
> On Thu, 16 Jun 2016 at 11:12 Maximilian Michels <mxm@apache.org
> <javascript:;>> wrote:
>
> > Hi Robert, hi Henry,
> >
> > +1 for a 1.1.0 release soon! We have enough new features that justify
> > a major release.
> >
> > @Henry We have plans to extend the ResourceManager to interact with
> > the Scheduler which will be a prerequisite for dynamic scaling. I
> > think this is out of scope for 1.1.0. The upcoming Mesos integration
> > won't require additional refactoring of the ResourceManager. Instead,
> > we will create a new "Dispatcher" component that takes care of
> > bootstrapping the initial node with the JobManager/ResourceManager.
> > From there on, everything will be handled by the Mesos
> > ResourceManager. I recently discussed this with Eron (CC) who came up
> > with this design and he plans to publish it to the mailing list soon.
> >
> > How about listing relevant JIRA issues here? "Half Merged" is kind of
> > hard to get for people who are not involved in the different
> > components.
> >
> > The Cassandra adapter seems like a pretty important thing to have for
> > the next release. In addition, I would like to merge FLINK-3667 and
> > FLINK-3937. Robert is doing a review at the moment :) Those are a)
> > refactoring of the command-line and client classes b) adding
> > capability to resume cluster programmatically.
> >
> > Then we should also have a look at any other critical/major bugs listed
> in
> > JIRA.
> >
> > Cheers,
> > Max
> >
> > On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <henry.saputra@gmail.com
> <javascript:;>>
> > wrote:
> > > Hi Robert,
> > >
> > > Thanks for staying the discussion.
> > >
> > > Do you know if there any open tasks for the Resource Manager left?
> > >
> > > That is probably needed for Mesos integration?
> > >
> > > - Henry
> > >
> > > On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <rmetzger@apache.org
> <javascript:;>>
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> Flink 1.0.0 was released early March, so three months have passed and
> I
> > >> think we should start discussing the scope of the next major release
> > >> (1.1.0).
> > >>
> > >> From a high level point of view, we've added the following new
> features:
> > >> in master:
> > >> - Table API Refactoring, SQL, StreamSQL
> > >> - The metrics system
> > >> - Kinesis Connector
> > >> - Persistent file sources for streaming
> > >>
> > >> Half merged:
> > >> - Resource manager refactoring
> > >>
> > >> Unmerged features:
> > >> - Cassandra connector
> > >> - Key groups ("rescaling from savepoints")
> > >> - Queryable state
> > >>
> > >> I'm pretty sure I forgot many other features / pull requests, please
> > post
> > >> them to this thread. I'll collect them and create a Wiki page out of
> it.
> > >>
> > >> Some immediate TODOs for us:
> > >> - Which of the unmerged features are we going to add to the release?
> > >> - Which blockers do we need to address before releasing?
> > >> - Are there any volunteers for the release manager?
> > >>
> > >>
> > >> Regards,
> > >> Robert
> > >>
> >
>


-- 
Thanks and Regards,
Vishnu Viswanath,
*www.vishnuviswanath.com <http://www.vishnuviswanath.com>*

Re: [DISCUSS] Releasing Flink 1.1.0

[Aljoscha Krettek <al...@apache.org>]

Hi,
+1 for releasing what we have now and making it rock solid.

The support for merging windows (session windows) is also new, btw. :-)

Cheers,
Aljoscha

On Thu, 16 Jun 2016 at 11:12 Maximilian Michels <mx...@apache.org> wrote:

> Hi Robert, hi Henry,
>
> +1 for a 1.1.0 release soon! We have enough new features that justify
> a major release.
>
> @Henry We have plans to extend the ResourceManager to interact with
> the Scheduler which will be a prerequisite for dynamic scaling. I
> think this is out of scope for 1.1.0. The upcoming Mesos integration
> won't require additional refactoring of the ResourceManager. Instead,
> we will create a new "Dispatcher" component that takes care of
> bootstrapping the initial node with the JobManager/ResourceManager.
> From there on, everything will be handled by the Mesos
> ResourceManager. I recently discussed this with Eron (CC) who came up
> with this design and he plans to publish it to the mailing list soon.
>
> How about listing relevant JIRA issues here? "Half Merged" is kind of
> hard to get for people who are not involved in the different
> components.
>
> The Cassandra adapter seems like a pretty important thing to have for
> the next release. In addition, I would like to merge FLINK-3667 and
> FLINK-3937. Robert is doing a review at the moment :) Those are a)
> refactoring of the command-line and client classes b) adding
> capability to resume cluster programmatically.
>
> Then we should also have a look at any other critical/major bugs listed in
> JIRA.
>
> Cheers,
> Max
>
> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <he...@gmail.com>
> wrote:
> > Hi Robert,
> >
> > Thanks for staying the discussion.
> >
> > Do you know if there any open tasks for the Resource Manager left?
> >
> > That is probably needed for Mesos integration?
> >
> > - Henry
> >
> > On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <rm...@apache.org>
> > wrote:
> >
> >> Hi,
> >>
> >> Flink 1.0.0 was released early March, so three months have passed and I
> >> think we should start discussing the scope of the next major release
> >> (1.1.0).
> >>
> >> From a high level point of view, we've added the following new features:
> >> in master:
> >> - Table API Refactoring, SQL, StreamSQL
> >> - The metrics system
> >> - Kinesis Connector
> >> - Persistent file sources for streaming
> >>
> >> Half merged:
> >> - Resource manager refactoring
> >>
> >> Unmerged features:
> >> - Cassandra connector
> >> - Key groups ("rescaling from savepoints")
> >> - Queryable state
> >>
> >> I'm pretty sure I forgot many other features / pull requests, please
> post
> >> them to this thread. I'll collect them and create a Wiki page out of it.
> >>
> >> Some immediate TODOs for us:
> >> - Which of the unmerged features are we going to add to the release?
> >> - Which blockers do we need to address before releasing?
> >> - Are there any volunteers for the release manager?
> >>
> >>
> >> Regards,
> >> Robert
> >>
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Maximilian Michels <mx...@apache.org>]

Hi Robert, hi Henry,

+1 for a 1.1.0 release soon! We have enough new features that justify
a major release.

@Henry We have plans to extend the ResourceManager to interact with
the Scheduler which will be a prerequisite for dynamic scaling. I
think this is out of scope for 1.1.0. The upcoming Mesos integration
won't require additional refactoring of the ResourceManager. Instead,
we will create a new "Dispatcher" component that takes care of
bootstrapping the initial node with the JobManager/ResourceManager.
From there on, everything will be handled by the Mesos
ResourceManager. I recently discussed this with Eron (CC) who came up
with this design and he plans to publish it to the mailing list soon.

How about listing relevant JIRA issues here? "Half Merged" is kind of
hard to get for people who are not involved in the different
components.

The Cassandra adapter seems like a pretty important thing to have for
the next release. In addition, I would like to merge FLINK-3667 and
FLINK-3937. Robert is doing a review at the moment :) Those are a)
refactoring of the command-line and client classes b) adding
capability to resume cluster programmatically.

Then we should also have a look at any other critical/major bugs listed in JIRA.

Cheers,
Max

On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <he...@gmail.com> wrote:
> Hi Robert,
>
> Thanks for staying the discussion.
>
> Do you know if there any open tasks for the Resource Manager left?
>
> That is probably needed for Mesos integration?
>
> - Henry
>
> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
>> Hi,
>>
>> Flink 1.0.0 was released early March, so three months have passed and I
>> think we should start discussing the scope of the next major release
>> (1.1.0).
>>
>> From a high level point of view, we've added the following new features:
>> in master:
>> - Table API Refactoring, SQL, StreamSQL
>> - The metrics system
>> - Kinesis Connector
>> - Persistent file sources for streaming
>>
>> Half merged:
>> - Resource manager refactoring
>>
>> Unmerged features:
>> - Cassandra connector
>> - Key groups ("rescaling from savepoints")
>> - Queryable state
>>
>> I'm pretty sure I forgot many other features / pull requests, please post
>> them to this thread. I'll collect them and create a Wiki page out of it.
>>
>> Some immediate TODOs for us:
>> - Which of the unmerged features are we going to add to the release?
>> - Which blockers do we need to address before releasing?
>> - Are there any volunteers for the release manager?
>>
>>
>> Regards,
>> Robert
>>

Re: [DISCUSS] Releasing Flink 1.1.0

[Andrea Sella <an...@radicalbit.io>]

Hi Robert,

Another interesting features to add in 1.1.0 should be the scala-shell
integration with YARN in HA[1].

Thank you,
Andrea

[1]
http://mail-archives.apache.org/mod_mbox/flink-dev/201604.mbox/%3CCABsXpV1Qu7v8cwxx-DkQ4JRNBvGNh-EF2MxdjzyYjsWUpQn01g%40mail.gmail.com%3E

2016-06-15 22:50 GMT+02:00 Henry Saputra <he...@gmail.com>:

> Hi Robert,
>
> Thanks for staying the discussion.
>
> Do you know if there any open tasks for the Resource Manager left?
>
> That is probably needed for Mesos integration?
>
> - Henry
>
> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
> > Hi,
> >
> > Flink 1.0.0 was released early March, so three months have passed and I
> > think we should start discussing the scope of the next major release
> > (1.1.0).
> >
> > From a high level point of view, we've added the following new features:
> > in master:
> > - Table API Refactoring, SQL, StreamSQL
> > - The metrics system
> > - Kinesis Connector
> > - Persistent file sources for streaming
> >
> > Half merged:
> > - Resource manager refactoring
> >
> > Unmerged features:
> > - Cassandra connector
> > - Key groups ("rescaling from savepoints")
> > - Queryable state
> >
> > I'm pretty sure I forgot many other features / pull requests, please post
> > them to this thread. I'll collect them and create a Wiki page out of it.
> >
> > Some immediate TODOs for us:
> > - Which of the unmerged features are we going to add to the release?
> > - Which blockers do we need to address before releasing?
> > - Are there any volunteers for the release manager?
> >
> >
> > Regards,
> > Robert
> >
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Henry Saputra <he...@gmail.com>]

Hi Robert,

Thanks for staying the discussion.

Do you know if there any open tasks for the Resource Manager left?

That is probably needed for Mesos integration?

- Henry

On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <rm...@apache.org>
wrote:

> Hi,
>
> Flink 1.0.0 was released early March, so three months have passed and I
> think we should start discussing the scope of the next major release
> (1.1.0).
>
> From a high level point of view, we've added the following new features:
> in master:
> - Table API Refactoring, SQL, StreamSQL
> - The metrics system
> - Kinesis Connector
> - Persistent file sources for streaming
>
> Half merged:
> - Resource manager refactoring
>
> Unmerged features:
> - Cassandra connector
> - Key groups ("rescaling from savepoints")
> - Queryable state
>
> I'm pretty sure I forgot many other features / pull requests, please post
> them to this thread. I'll collect them and create a Wiki page out of it.
>
> Some immediate TODOs for us:
> - Which of the unmerged features are we going to add to the release?
> - Which blockers do we need to address before releasing?
> - Are there any volunteers for the release manager?
>
>
> Regards,
> Robert
>