You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Flavio Pompermaier <po...@okkam.it> on 2016/11/14 13:47:37 UTC
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Hi to all,
for the 1.2 release do you think it will be possible to include also a PR
to implement FLINK-4491 <https://issues.apache.org/jira/browse/FLINK-4491>?
The PR is waiting for a review at https://github.com/apache/flink/pull/2790.
Best,
Flavio
On Fri, Oct 28, 2016 at 5:59 PM, Fabian Hueske <fh...@gmail.com> wrote:
> Hi Jark,
>
> FLINK-4469: I agree, UDTFs would be a great feature. I don't think we
> should make it a blocker but I will help to get it the master before the
> feature freeze.
> FLINK-4704: Regarding the renaming of the namespace, we have two positive
> votes in JIRA, but we should also start a discussion on the dev and user
> list to collect some more input. The actual change should be simple.
>
> @Everybody: I created a wiki page for the 1.2 release [1] and added all
> features discussed in this thread so far.
> Please have a look at the page and update it if you know the JIRAs and/or
> state of the features.
>
> Cheers, Fabian
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/1.2+Release
>
> 2016-10-28 15:28 GMT+02:00 Kostas Kloudas <k....@data-artisans.com>:
>
> > Hello,
> >
> > As Aljoshca said, the TriggerDSL depends on another PR, before it can be
> > merged.
> > In addition, this will change also the whole structure of how we test
> > window/trigger
> > related functionality, so some refactoring will be required.
> >
> > Kostas
> >
> > > On Oct 28, 2016, at 3:06 PM, Aljoscha Krettek <al...@apache.org>
> > wrote:
> > >
> > > @Till: For the TriggerDSL we have to merge my open PR [1] for
> refactoring
> > > the WindowOperator/Trigger tests and then refactor the tests of the
> > > TriggerDSL to use the new test facilities introduced there. Plus then
> > maybe
> > > some fixups.
> > >
> > > On Fri, 28 Oct 2016 at 11:17 Jark Wu <wu...@alibaba-inc.com>
> wrote:
> > >
> > >> +1 to the schedule
> > >>
> > >> I hope to add
> > >>
> > >> 1) Table API: User defined table function (FLINK-4469 <
> > >> https://issues.apache.org/jira/browse/FLINK-4469>)
> > >> 2) Table API: Move Table API to org.apache.flink.table (FLINK-4704 <
> > >> https://issues.apache.org/jira/browse/FLINK-4704>)
> > >>
> > >> The 1) has been discussed for a long time and a pull request has been
> on
> > >> github. The 2) would be a large change, but I hope it could be
> included
> > in
> > >> 1.2 release, otherwise it will be more difficult to fix it.
> > >>
> > >> - Jark Wu
> > >>
> > >>> 在 2016年10月26日，下午8:09，Till Rohrmann <tr...@apache.org> 写道：
> > >>>
> > >>> +1 for the features to include.
> > >>>
> > >>> What is the state of the Trigger DSL? How much is left to be done
> > before
> > >>> merging?
> > >>>
> > >>> Cheers,
> > >>> Till
> > >>>
> > >>> On Tue, Oct 25, 2016 at 4:32 PM, Aljoscha Krettek <
> aljoscha@apache.org
> > >
> > >>> wrote:
> > >>>
> > >>>> +1 the schedule proposed so far.
> > >>>>
> > >>>> Do we also want to get in the "Trigger DSL" that we've had brewing
> > for a
> > >>>> while now?
> > >>>>
> > >>>> On Mon, 17 Oct 2016 at 16:17 Stephan Ewen <se...@apache.org> wrote:
> > >>>>
> > >>>>> I think this sounds very reasonable, +1 to the schedule.
> > >>>>>
> > >>>>> I would definitely add
> > >>>>> - FLIP-10 (unify checkpoints and savepoints)
> > >>>>> - FLIP-7 (metrics in web UI)
> > >>>>> - FLIP-12 (async request operators)
> > >>>>>
> > >>>>> Those should be all safe bets, as they are basically done.
> > >>>>>
> > >>>>> Let's see what else is in shape until the feature freeze day.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Mon, Oct 17, 2016 at 2:18 PM, Fabian Hueske <fh...@gmail.com>
> > >>>> wrote:
> > >>>>>
> > >>>>>> Hi everybody,
> > >>>>>>
> > >>>>>> Flink 1.1.0 was released in August and I think it is time to kick
> > off
> > >> a
> > >>>>>> discussion about the schedule and scope of Flink 1.2.0.
> > >>>>>>
> > >>>>>> == Scope
> > >>>>>>
> > >>>>>> We started to collect features for Flink 1.2.0 in the Flink
> Release
> > >>>> wiki
> > >>>>>> page [1].
> > >>>>>> I copy the feature list for convenience:
> > >>>>>>
> > >>>>>> - Dynamic Scaling / Key Groups (FLINK-3755
> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3755>) (Design Doc
> > >>>>>> <https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-
> > >>>>>> LuBCyPUWyFd9l3T9WyssQ63w/edit#heading=h.2suhu1fjxmp4>
> > >>>>>> )
> > >>>>>> - Rescalable Non-Partitioned State (FLIP-8
> > >>>>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-
> > >>>>>> 8%3A+Rescalable+Non-Partitioned+State>
> > >>>>>> )
> > >>>>>> - Backwards compatible savepoints (FLINK-4797
> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-4797>)
> > >>>>>> -
> > >>>>>>
> > >>>>>> Integrate Flink with Apache Mesos (FLINK-1984
> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-1984>) (Design Doc
> > >>>>>> <https://docs.google.com/document/d/
> 1WItafBmGbjlaBbP8Of5PAFOH9GUJQ
> > >>>>>> xf5S4hjEuPchuU/edit#heading=h.33uxp8abvuf0>
> > >>>>>> )
> > >>>>>> - Secure Data Access (FLINK-3930
> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3930>) (Design Doc
> > >>>>>> <https://docs.google.com/document/d/1-
> > >>>> GQB6uVOyoaXGwtqwqLV8BHDxWiMO2W
> > >>>>>> nVzBoJ8oPaAs/edit#heading=h.dr22iualfm0h>
> > >>>>>> )
> > >>>>>> - Queryable State (FLINK-3779
> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3779>)
> > >>>>>> - Metrics in Webinterface (FLIP-7
> > >>>>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-
> > >>>>>> 7%3A+Expose+metrics+to+WebInterface>
> > >>>>>> )
> > >>>>>> - Kafka 0.10 support (FLINK-4035
> > >>>>>> <http://issues.apache.org/jira/browse/FLINK-4035>)
> > >>>>>> - Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
> > >>>>>> - Table API: Scalar Functions (FLINK-3097)
> > >>>>>>
> > >>>>>>
> > >>>>>> Is a feature missing or something too ambitious?
> > >>>>>>
> > >>>>>> == Schedule
> > >>>>>>
> > >>>>>> In my opinion we should try to release 1.2.0 before Christmas.
> That
> > >>>>> means,
> > >>>>>> we should have a RC1 by end November / early December and a
> feature
> > >>>>> freeze
> > >>>>>> two weeks earlier in mid of November. Does that sound appropriate
> or
> > >>>>> should
> > >>>>>> we aim for an earlier or later release?
> > >>>>>>
> > >>>>>> What do others think about the scope and schedule?
> > >>>>>> Please discuss.
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Fabian
> > >>>>>>
> > >>>>>> [1]
> > >>>>>> https://cwiki.apache.org/confluence/display/FLINK/
> > >>>>>> Flink+Release+and+Feature+Plan
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> > >>
> >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Till Rohrmann <tr...@apache.org>.
We might also think about addressing:
Relocate Flink's Hadoop dependency and its transitive dependencies
(FLINK-5297),
because a user reported that they cannot use the system due to a dependency
issue.
Cheers,
Till
On Thu, Dec 8, 2016 at 10:17 AM, Robert Metzger <rm...@apache.org> wrote:
> Thank you for your responses Max and Vijay.
> So I understand that Mesos is basically ready for the 1.2 release.
>
> Regarding the security changes: Having Hadoop, Kafka and Zookeeper
> integration is a big improvement and a much requested feature. I'm super
> excited to have that in :)
> Are all the other security changes useless without authorization, or could
> we consider releasing 1.2 without it? (Another way to think about it: How
> close is the PR to being merged. If its just a final review & we are done,
> I would actually try to get it in. But if there's a lot of uncertainty, I
> would prefer to move it to the next release)
>
> I agree regarding FLINK-2821, that's important for many deployments.
>
>
> The updated list:
> - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
> - RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
> - RESOLVED [Split for 1.3] Add Flink 1.1 savepoint backwards compatability
> (FLINK-4797)
> - RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos (FLINK-1984)
> - UNRESOLVED Secure Data Access (FLINK-3930)
> - RESOLVED Queryable State (FLINK-3779)
> - RESOLVED Metrics in Webinterface (FLINK-4389)
> - RESOLVED Kafka 0.10 support (FLINK-4035)
> - RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
> - RESOLVED Table API: Scalar Functions (FLINK-3097)
> Added by Stephan:
> - NON-BLOCKING [Pending PR] Provide support for asynchronous operations
> over streams (FLINK-4391)
> - ONGOING [beginning of next week] Unify Savepoints and Checkpoints
> (FLINK-4484)
> Added by Fabian:
> - ONGOING [Pending PR] Clean up the packages of the Table API (FLINK-4704)
> - UNRESOLVED Move Row to flink-core (FLINK-5186)
> Added by Max:
> - ONGOING [Pending PR] Change Akka configuration to allow accessing actors
> from different URLs (FLINK-2821)
>
>
> On Wed, Dec 7, 2016 at 12:40 PM, Maximilian Michels <mx...@apache.org>
> wrote:
>
> > > - UNRESOLVED Integrate Flink with Apache Mesos (FLINK-1984)
> >
> > The initial integration is already completed with the last issues
> > being resolved in the Mesos component:
> > https://issues.apache.org/jira/browse/FLINK/component/12331068/ The
> > implementation will be further refined after the next release and with
> > the merge of FLIP-6. We're missing documentation on how to deploy a
> > Flink Mesos cluster.
> >
> > > - UNRESOLVED Secure Data Access (FLINK-3930)
> >
> > We have support for Kerberos authentication with Haddop, Kafka,
> > Zookeper, and all services supporting JAAS. Additionally, we
> > implemented SSL encryption for all communications paths, i.e. web
> > interface, Akka, Netty, BlobServer. We still lack support for
> > authorization: Vijay's PR is blocked because we haven't found time to
> > properly review the sensitive network changes.
> >
> > I'd like to add the Akka changes for containered environments which
> > should be ready by the end of the week:
> > https://issues.apache.org/jira/browse/FLINK-2821
> >
> > -Max
> >
> > On Tue, Dec 6, 2016 at 8:57 PM, Vijay <vi...@yahoo.com.invalid>
> > wrote:
> > >>>Secure Data Access (FLINK-3930)
> > >
> > > The PR for the work is still under review and I hope this could be
> > included in the release.
> > >
> > > Regards,
> > > Vijay
> > >
> > > Sent from my iPhone
> > >
> > >> On Dec 6, 2016, at 11:51 AM, Robert Metzger <rm...@apache.org>
> > wrote:
> > >>
> > >> UNRESOLVED Secure Data Access (FLINK-3930)
> > >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Vijay <vi...@yahoo.com.INVALID>.
Hi Stephan,
The new handler changes for Netty layer is straightforward and it is not really an workaround. Moreover if we think there is a better way to handle it in future we can easily unwind it as it is just a pluggable handler. Pushing these changes before FLIP-6 merge certainly avoids lot of conflicts and also we already spent cycles in reviewing and fixing rest of the authorization changes for other layers.
Regards,
Vijay
Sent from my iPhone
> On Dec 12, 2016, at 8:43 AM, Stephan Ewen <se...@apache.org> wrote:
>
> Hi Vijay!
>
> The workaround you suggest may be doable, but I am wondering how much that
> helps, because the authorization feature would be incomplete like that and
> thus of limited use.
>
> I would also assume that merging it properly and in full use after the 1.2
> release would be a bit better - in general, we have often avoided last
> minute additions of sensitive and complex features.
>
> Do you think it is more urgent to have this in Flink?
>
> Best,
> Stephan
>
>
>> On Mon, Dec 12, 2016 at 2:49 PM, Vijay <vi...@yahoo.com.invalid> wrote:
>>
>> Max and Ufuk, I respect your concerns and fully understand the importance
>> of the network layer stack in Flink code base. Will you be comfortable to
>> merge the code if I remove the Netty layer changes and leave the rest of
>> the code. We can address the Netty code changes post 1.2 release?
>>
>> Regards,
>> Vijay
>>
>> Sent from my iPhone
>>
>>> On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <uc...@apache.org> wrote:
>>>
>>> On 12 December 2016 at 12:30:31, Maximilian Michels (mxm@apache.org)
>> wrote:
>>>>> It seems like we lack the resources for now to properly to take
>>>> care
>>>> of your pull request before the release. Unless someone from
>>>> the
>>>> community is really eager to help out here, I would be in favor
>>>> of
>>>> merging the pull request to the master after the release branch
>>>> has
>>>> been forked off. We should make sure it gets the attention it deserves
>>>> then.
>>>
>>> Thanks Max! I fully agree with your reasoning. +1 to not include this in
>> 1.2 now, but look at it afterwards. I hope that OK with you Vijay.
>>>
>>> – Ufuk
>>>
>>>
>>
>>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Stephan Ewen <se...@apache.org>.
Very nice, thanks Robert!
On Tue, Dec 20, 2016 at 6:02 PM, Robert Metzger <rm...@apache.org> wrote:
> Okay, I forked off a release-1.2 branch and updated the version in master
> to 1.3-SNAPSHOT.
> I'll create the first RC (non voting) in the next few days.
>
> On Tue, Dec 20, 2016 at 4:39 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
> > Hi Aljoscha,
> > thanks a lot for resolving this last big release blocker!
> >
> > I'll definitively fork off in the release branch in the next few hours.
> > Stephan asked me to wait for a few more minutes for his maven changes.
> >
> > The JIRAs you've mentioned seem to be pretty isolated, so its fine to
> > merge them afterwards to both branches.
> >
> > Regards,
> > Robert
> >
> >
> > On Tue, Dec 20, 2016 at 4:17 PM, Aljoscha Krettek <al...@apache.org>
> > wrote:
> >
> >> I just merged the most important backwards compatibility changes, with
> >> tests.
> >>
> >> I think this one is still a blocker:
> >> https://issues.apache.org/jira/browse/FLINK-5320 (WindowedStream.fold()
> >> cannot be used). And this one is a potential blocker for some users:
> >> https://issues.apache.org/jira/browse/FLINK-5363.
> >>
> >> IMHO, we can cut the branch today and I'll get them in on master and
> both
> >> the 1.2 branch. What do you think?
> >>
> >> On Tue, 20 Dec 2016 at 15:24 Robert Metzger <rm...@apache.org>
> wrote:
> >>
> >> > Quick update here: I talked to Aljoscha offline, and the backwards
> >> > compatibility is still being tested (there were some bugs identified
> >> while
> >> > writing the tests).
> >> >
> >> > Also, Stephan made some fixes to the build infrastructure (
> >> > https://github.com/apache/flink/pull/3029) that would be good to be
> >> > included into the release branch.
> >> > To finally get the FLIP-6 branch merged to master, I'm considering
> >> > branching off the 1.2 release later today. It will be a little bit
> more
> >> > overhead for Stephan and Aljoscha, but it will unblock all features
> >> waiting
> >> > for a Flink 1.3 master.
> >> >
> >> >
> >> >
> >> >
> >> > On Fri, Dec 16, 2016 at 6:05 PM, Robert Metzger <rm...@apache.org>
> >> > wrote:
> >> >
> >> > > Thank you Aljoscha and Fabian for the updates.
> >> > > I propose *Monday evening (6pm Berlin, 9am US west coast) for
> feature
> >> > > freezing Flink 1.2 *then. This means that I'll create a release-1.2
> >> fork
> >> > > and create a 1.2 RC0 (non-voting) release candidate for testing.
> >> > >
> >> > > I don't think that I'll create the first (voting) RC until January
> >> > because
> >> > > of christmas and new years activities. Most of the committers I know
> >> are
> >> > > out of office during these 1,5 weeks.
> >> > >
> >> > >
> >> > >
> >> > > On Fri, Dec 16, 2016 at 5:55 PM, Fabian Hueske <fh...@gmail.com>
> >> > wrote:
> >> > >
> >> > >> Hi,
> >> > >>
> >> > >> I merged the Table API refactoring changes:
> >> > >>
> >> > >> - RESOLVED Clean up the packages of the Table API (FLINK-4704)
> >> > >> - RESOLVED Move Row to flink-core (FLINK-5186)
> >> > >>
> >> > >> No blockers left from my side.
> >> > >>
> >> > >> Cheers, Fabian
> >> > >>
> >> > >> 2016-12-16 17:47 GMT+01:00 Aljoscha Krettek <al...@apache.org>:
> >> > >>
> >> > >> > Yes, I'm confident that we can finish the tests until then and
> >> merge
> >> > the
> >> > >> > code.
> >> > >> >
> >> > >> > On Fri, Dec 16, 2016, 17:41 Robert Metzger <rm...@apache.org>
> >> > wrote:
> >> > >> >
> >> > >> > > Thank you for the update. Do you think you get it done until
> >> Monday
> >> > >> > > evening?
> >> > >> > >
> >> > >> > > On Fri, Dec 16, 2016 at 5:23 PM, Aljoscha Krettek <
> >> > >> aljoscha@apache.org>
> >> > >> > > wrote:
> >> > >> > >
> >> > >> > > > Hi,
> >> > >> > > > we're still working on making the backwards compatibility
> from
> >> 1.1
> >> > >> > > > savepoints a reality. We have most of the code and some tests
> >> now
> >> > >> but
> >> > >> > it
> >> > >> > > > still needs some work. This is the issue that tracks the
> >> progress
> >> > on
> >> > >> > the
> >> > >> > > > operators that we would like to make backwards compatible:
> >> > >> > > > https://issues.apache.org/jira/browse/FLINK-5292
> >> > >> > > >
> >> > >> > > > Cheers,
> >> > >> > > > Aljoscha
> >> > >> > > >
> >> > >> > > > On Tue, 13 Dec 2016 at 11:22 Feng Wang <
> feng.wang@outlook.com>
> >> > >> wrote:
> >> > >> > > >
> >> > >> > > > > It will be pretty good if 1.2 branch could be forked off
> >> within
> >> > >> this
> >> > >> > > > week,
> >> > >> > > > > and our guys working on FLIP-6 hope FLIP-6 branch could be
> >> > merged
> >> > >> > into
> >> > >> > > > > master as soon as possible.
> >> > >> > > > >
> >> > >> > > > > Best Regards,
> >> > >> > > > >
> >> > >> > > > > Feng Wang
> >> > >> > > > >
> >> > >> > > > > Alibaba
> >> > >> > > > >
> >> > >> > > > > ________________________________
> >> > >> > > > > From: Robert Metzger <rm...@apache.org>
> >> > >> > > > > Sent: Tuesday, December 13, 2016 4:58 AM
> >> > >> > > > > To: dev@flink.apache.org
> >> > >> > > > > Subject: Re: [DISCUSS] Schedule and Scope for Flink 1.2
> >> > >> > > > >
> >> > >> > > > > Thank you all for figuring out a solution for the security
> >> pull
> >> > >> > > request.
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > Lets try to get 1.2 feature freezed as fast as possible so
> >> that
> >> > we
> >> > >> > can
> >> > >> > > > > "unblock" waiting features like FLIP-6 and the remaining
> >> > security
> >> > >> > > > changes.
> >> > >> > > > >
> >> > >> > > > > *What do you think about Friday evening (6pm Berlin, 9am US
> >> west
> >> > >> > coast)
> >> > >> > > > for
> >> > >> > > > > feature freezing Flink 1.2?* (only bugfixes are allowed in
> >> > >> > afterwards)
> >> > >> > > > > I'll then fork-off a "release-1.2" branch and update the
> >> version
> >> > >> in
> >> > >> > > > > "master" to 1.3-SNAPSHOT.
> >> > >> > > > > Please object if you have a bigger change or any other
> >> > >> reservations
> >> > >> > > > > regarding the feature freeze date!
> >> > >> > > > >
> >> > >> > > > > This is my current view of things on the release:
> >> > >> > > > >
> >> > >> > > > > - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
> >> > >> > > > > - RESOLVED Add Rescalable Non-Partitioned State
> (FLINK-4379)
> >> > >> > > > > - UNRESOLVED Add Flink 1.1 savepoint backwards
> compatability
> >> > >> > > (FLINK-4797)
> >> > >> > > > > - RESOLVED [Split for 1.3] Integrate Flink with Apache
> Mesos
> >> > >> > > (FLINK-1984)
> >> > >> > > > > - UNDER DISCUSSION Secure Data Access (FLINK-3930)
> >> > >> > > > > - RESOLVED Queryable State (FLINK-3779)
> >> > >> > > > > - RESOLVED Metrics in Webinterface (FLINK-4389)
> >> > >> > > > > - RESOLVED Kafka 0.10 support (FLINK-4035)
> >> > >> > > > > - RESOLVED Table API: Group Window Aggregates (FLINK-4691,
> >> > >> FLIP-11)
> >> > >> > > > > - RESOLVED Table API: Scalar Functions (FLINK-3097)
> >> > >> > > > > Added by Stephan:
> >> > >> > > > > - NON-BLOCKING [Pending PR] Provide support for
> asynchronous
> >> > >> > operations
> >> > >> > > > > over streams (FLINK-4391)
> >> > >> > > > > - NON-BLOCKING [beginning of next week] Unify Savepoints
> and
> >> > >> > > Checkpoints
> >> > >> > > > > (FLINK-4484)
> >> > >> > > > > Added by Fabian:
> >> > >> > > > > - ONGOING [Pending PR] Clean up the packages of the Table
> API
> >> > >> > > > (FLINK-4704)
> >> > >> > > > > Move Row to flink-core (
> >> > >> > > > > Added by Max:
> >> > >> > > > > - ONGOING [Pending PR] Change Akka configuration to allow
> >> > >> accessing
> >> > >> > > > actors
> >> > >> > > > > from different URLs (FLINK-2821)
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > On Mon, Dec 12, 2016 at 5:43 PM, Stephan Ewen <
> >> sewen@apache.org
> >> > >
> >> > >> > > wrote:
> >> > >> > > > >
> >> > >> > > > > > Hi Vijay!
> >> > >> > > > > >
> >> > >> > > > > > The workaround you suggest may be doable, but I am
> >> wondering
> >> > how
> >> > >> > much
> >> > >> > > > > that
> >> > >> > > > > > helps, because the authorization feature would be
> >> incomplete
> >> > >> like
> >> > >> > > that
> >> > >> > > > > and
> >> > >> > > > > > thus of limited use.
> >> > >> > > > > >
> >> > >> > > > > > I would also assume that merging it properly and in full
> >> use
> >> > >> after
> >> > >> > > the
> >> > >> > > > > 1.2
> >> > >> > > > > > release would be a bit better - in general, we have often
> >> > >> avoided
> >> > >> > > last
> >> > >> > > > > > minute additions of sensitive and complex features.
> >> > >> > > > > >
> >> > >> > > > > > Do you think it is more urgent to have this in Flink?
> >> > >> > > > > >
> >> > >> > > > > > Best,
> >> > >> > > > > > Stephan
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > > On Mon, Dec 12, 2016 at 2:49 PM, Vijay
> >> > >> > <vijikarthi@yahoo.com.invalid
> >> > >> > > >
> >> > >> > > > > > wrote:
> >> > >> > > > > >
> >> > >> > > > > > > Max and Ufuk, I respect your concerns and fully
> >> understand
> >> > the
> >> > >> > > > > importance
> >> > >> > > > > > > of the network layer stack in Flink code base. Will you
> >> be
> >> > >> > > > comfortable
> >> > >> > > > > to
> >> > >> > > > > > > merge the code if I remove the Netty layer changes and
> >> leave
> >> > >> the
> >> > >> > > rest
> >> > >> > > > > of
> >> > >> > > > > > > the code. We can address the Netty code changes post
> 1.2
> >> > >> release?
> >> > >> > > > > > >
> >> > >> > > > > > > Regards,
> >> > >> > > > > > > Vijay
> >> > >> > > > > > >
> >> > >> > > > > > > Sent from my iPhone
> >> > >> > > > > > >
> >> > >> > > > > > > > On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <
> >> uce@apache.org>
> >> > >> > wrote:
> >> > >> > > > > > > >
> >> > >> > > > > > > > On 12 December 2016 at 12:30:31, Maximilian Michels (
> >> > >> > > > mxm@apache.org)
> >> > >> > > > > > > wrote:
> >> > >> > > > > > > >>> It seems like we lack the resources for now to
> >> properly
> >> > to
> >> > >> > take
> >> > >> > > > > > > >> care
> >> > >> > > > > > > >> of your pull request before the release. Unless
> >> someone
> >> > >> from
> >> > >> > > > > > > >> the
> >> > >> > > > > > > >> community is really eager to help out here, I would
> >> be in
> >> > >> > favor
> >> > >> > > > > > > >> of
> >> > >> > > > > > > >> merging the pull request to the master after the
> >> release
> >> > >> > branch
> >> > >> > > > > > > >> has
> >> > >> > > > > > > >> been forked off. We should make sure it gets the
> >> > attention
> >> > >> it
> >> > >> > > > > deserves
> >> > >> > > > > > > >> then.
> >> > >> > > > > > > >
> >> > >> > > > > > > > Thanks Max! I fully agree with your reasoning. +1 to
> >> not
> >> > >> > include
> >> > >> > > > this
> >> > >> > > > > > in
> >> > >> > > > > > > 1.2 now, but look at it afterwards. I hope that OK with
> >> you
> >> > >> > Vijay.
> >> > >> > > > > > > >
> >> > >> > > > > > > > - Ufuk
> >> > >> > > > > > > >
> >> > >> > > > > > > >
> >> > >> > > > > > >
> >> > >> > > > > > >
> >> > >> > > > > >
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Robert Metzger <rm...@apache.org>.
Okay, I forked off a release-1.2 branch and updated the version in master
to 1.3-SNAPSHOT.
I'll create the first RC (non voting) in the next few days.
On Tue, Dec 20, 2016 at 4:39 PM, Robert Metzger <rm...@apache.org> wrote:
> Hi Aljoscha,
> thanks a lot for resolving this last big release blocker!
>
> I'll definitively fork off in the release branch in the next few hours.
> Stephan asked me to wait for a few more minutes for his maven changes.
>
> The JIRAs you've mentioned seem to be pretty isolated, so its fine to
> merge them afterwards to both branches.
>
> Regards,
> Robert
>
>
> On Tue, Dec 20, 2016 at 4:17 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> I just merged the most important backwards compatibility changes, with
>> tests.
>>
>> I think this one is still a blocker:
>> https://issues.apache.org/jira/browse/FLINK-5320 (WindowedStream.fold()
>> cannot be used). And this one is a potential blocker for some users:
>> https://issues.apache.org/jira/browse/FLINK-5363.
>>
>> IMHO, we can cut the branch today and I'll get them in on master and both
>> the 1.2 branch. What do you think?
>>
>> On Tue, 20 Dec 2016 at 15:24 Robert Metzger <rm...@apache.org> wrote:
>>
>> > Quick update here: I talked to Aljoscha offline, and the backwards
>> > compatibility is still being tested (there were some bugs identified
>> while
>> > writing the tests).
>> >
>> > Also, Stephan made some fixes to the build infrastructure (
>> > https://github.com/apache/flink/pull/3029) that would be good to be
>> > included into the release branch.
>> > To finally get the FLIP-6 branch merged to master, I'm considering
>> > branching off the 1.2 release later today. It will be a little bit more
>> > overhead for Stephan and Aljoscha, but it will unblock all features
>> waiting
>> > for a Flink 1.3 master.
>> >
>> >
>> >
>> >
>> > On Fri, Dec 16, 2016 at 6:05 PM, Robert Metzger <rm...@apache.org>
>> > wrote:
>> >
>> > > Thank you Aljoscha and Fabian for the updates.
>> > > I propose *Monday evening (6pm Berlin, 9am US west coast) for feature
>> > > freezing Flink 1.2 *then. This means that I'll create a release-1.2
>> fork
>> > > and create a 1.2 RC0 (non-voting) release candidate for testing.
>> > >
>> > > I don't think that I'll create the first (voting) RC until January
>> > because
>> > > of christmas and new years activities. Most of the committers I know
>> are
>> > > out of office during these 1,5 weeks.
>> > >
>> > >
>> > >
>> > > On Fri, Dec 16, 2016 at 5:55 PM, Fabian Hueske <fh...@gmail.com>
>> > wrote:
>> > >
>> > >> Hi,
>> > >>
>> > >> I merged the Table API refactoring changes:
>> > >>
>> > >> - RESOLVED Clean up the packages of the Table API (FLINK-4704)
>> > >> - RESOLVED Move Row to flink-core (FLINK-5186)
>> > >>
>> > >> No blockers left from my side.
>> > >>
>> > >> Cheers, Fabian
>> > >>
>> > >> 2016-12-16 17:47 GMT+01:00 Aljoscha Krettek <al...@apache.org>:
>> > >>
>> > >> > Yes, I'm confident that we can finish the tests until then and
>> merge
>> > the
>> > >> > code.
>> > >> >
>> > >> > On Fri, Dec 16, 2016, 17:41 Robert Metzger <rm...@apache.org>
>> > wrote:
>> > >> >
>> > >> > > Thank you for the update. Do you think you get it done until
>> Monday
>> > >> > > evening?
>> > >> > >
>> > >> > > On Fri, Dec 16, 2016 at 5:23 PM, Aljoscha Krettek <
>> > >> aljoscha@apache.org>
>> > >> > > wrote:
>> > >> > >
>> > >> > > > Hi,
>> > >> > > > we're still working on making the backwards compatibility from
>> 1.1
>> > >> > > > savepoints a reality. We have most of the code and some tests
>> now
>> > >> but
>> > >> > it
>> > >> > > > still needs some work. This is the issue that tracks the
>> progress
>> > on
>> > >> > the
>> > >> > > > operators that we would like to make backwards compatible:
>> > >> > > > https://issues.apache.org/jira/browse/FLINK-5292
>> > >> > > >
>> > >> > > > Cheers,
>> > >> > > > Aljoscha
>> > >> > > >
>> > >> > > > On Tue, 13 Dec 2016 at 11:22 Feng Wang <fe...@outlook.com>
>> > >> wrote:
>> > >> > > >
>> > >> > > > > It will be pretty good if 1.2 branch could be forked off
>> within
>> > >> this
>> > >> > > > week,
>> > >> > > > > and our guys working on FLIP-6 hope FLIP-6 branch could be
>> > merged
>> > >> > into
>> > >> > > > > master as soon as possible.
>> > >> > > > >
>> > >> > > > > Best Regards,
>> > >> > > > >
>> > >> > > > > Feng Wang
>> > >> > > > >
>> > >> > > > > Alibaba
>> > >> > > > >
>> > >> > > > > ________________________________
>> > >> > > > > From: Robert Metzger <rm...@apache.org>
>> > >> > > > > Sent: Tuesday, December 13, 2016 4:58 AM
>> > >> > > > > To: dev@flink.apache.org
>> > >> > > > > Subject: Re: [DISCUSS] Schedule and Scope for Flink 1.2
>> > >> > > > >
>> > >> > > > > Thank you all for figuring out a solution for the security
>> pull
>> > >> > > request.
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > Lets try to get 1.2 feature freezed as fast as possible so
>> that
>> > we
>> > >> > can
>> > >> > > > > "unblock" waiting features like FLIP-6 and the remaining
>> > security
>> > >> > > > changes.
>> > >> > > > >
>> > >> > > > > *What do you think about Friday evening (6pm Berlin, 9am US
>> west
>> > >> > coast)
>> > >> > > > for
>> > >> > > > > feature freezing Flink 1.2?* (only bugfixes are allowed in
>> > >> > afterwards)
>> > >> > > > > I'll then fork-off a "release-1.2" branch and update the
>> version
>> > >> in
>> > >> > > > > "master" to 1.3-SNAPSHOT.
>> > >> > > > > Please object if you have a bigger change or any other
>> > >> reservations
>> > >> > > > > regarding the feature freeze date!
>> > >> > > > >
>> > >> > > > > This is my current view of things on the release:
>> > >> > > > >
>> > >> > > > > - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
>> > >> > > > > - RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
>> > >> > > > > - UNRESOLVED Add Flink 1.1 savepoint backwards compatability
>> > >> > > (FLINK-4797)
>> > >> > > > > - RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos
>> > >> > > (FLINK-1984)
>> > >> > > > > - UNDER DISCUSSION Secure Data Access (FLINK-3930)
>> > >> > > > > - RESOLVED Queryable State (FLINK-3779)
>> > >> > > > > - RESOLVED Metrics in Webinterface (FLINK-4389)
>> > >> > > > > - RESOLVED Kafka 0.10 support (FLINK-4035)
>> > >> > > > > - RESOLVED Table API: Group Window Aggregates (FLINK-4691,
>> > >> FLIP-11)
>> > >> > > > > - RESOLVED Table API: Scalar Functions (FLINK-3097)
>> > >> > > > > Added by Stephan:
>> > >> > > > > - NON-BLOCKING [Pending PR] Provide support for asynchronous
>> > >> > operations
>> > >> > > > > over streams (FLINK-4391)
>> > >> > > > > - NON-BLOCKING [beginning of next week] Unify Savepoints and
>> > >> > > Checkpoints
>> > >> > > > > (FLINK-4484)
>> > >> > > > > Added by Fabian:
>> > >> > > > > - ONGOING [Pending PR] Clean up the packages of the Table API
>> > >> > > > (FLINK-4704)
>> > >> > > > > Move Row to flink-core (
>> > >> > > > > Added by Max:
>> > >> > > > > - ONGOING [Pending PR] Change Akka configuration to allow
>> > >> accessing
>> > >> > > > actors
>> > >> > > > > from different URLs (FLINK-2821)
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > On Mon, Dec 12, 2016 at 5:43 PM, Stephan Ewen <
>> sewen@apache.org
>> > >
>> > >> > > wrote:
>> > >> > > > >
>> > >> > > > > > Hi Vijay!
>> > >> > > > > >
>> > >> > > > > > The workaround you suggest may be doable, but I am
>> wondering
>> > how
>> > >> > much
>> > >> > > > > that
>> > >> > > > > > helps, because the authorization feature would be
>> incomplete
>> > >> like
>> > >> > > that
>> > >> > > > > and
>> > >> > > > > > thus of limited use.
>> > >> > > > > >
>> > >> > > > > > I would also assume that merging it properly and in full
>> use
>> > >> after
>> > >> > > the
>> > >> > > > > 1.2
>> > >> > > > > > release would be a bit better - in general, we have often
>> > >> avoided
>> > >> > > last
>> > >> > > > > > minute additions of sensitive and complex features.
>> > >> > > > > >
>> > >> > > > > > Do you think it is more urgent to have this in Flink?
>> > >> > > > > >
>> > >> > > > > > Best,
>> > >> > > > > > Stephan
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > > On Mon, Dec 12, 2016 at 2:49 PM, Vijay
>> > >> > <vijikarthi@yahoo.com.invalid
>> > >> > > >
>> > >> > > > > > wrote:
>> > >> > > > > >
>> > >> > > > > > > Max and Ufuk, I respect your concerns and fully
>> understand
>> > the
>> > >> > > > > importance
>> > >> > > > > > > of the network layer stack in Flink code base. Will you
>> be
>> > >> > > > comfortable
>> > >> > > > > to
>> > >> > > > > > > merge the code if I remove the Netty layer changes and
>> leave
>> > >> the
>> > >> > > rest
>> > >> > > > > of
>> > >> > > > > > > the code. We can address the Netty code changes post 1.2
>> > >> release?
>> > >> > > > > > >
>> > >> > > > > > > Regards,
>> > >> > > > > > > Vijay
>> > >> > > > > > >
>> > >> > > > > > > Sent from my iPhone
>> > >> > > > > > >
>> > >> > > > > > > > On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <
>> uce@apache.org>
>> > >> > wrote:
>> > >> > > > > > > >
>> > >> > > > > > > > On 12 December 2016 at 12:30:31, Maximilian Michels (
>> > >> > > > mxm@apache.org)
>> > >> > > > > > > wrote:
>> > >> > > > > > > >>> It seems like we lack the resources for now to
>> properly
>> > to
>> > >> > take
>> > >> > > > > > > >> care
>> > >> > > > > > > >> of your pull request before the release. Unless
>> someone
>> > >> from
>> > >> > > > > > > >> the
>> > >> > > > > > > >> community is really eager to help out here, I would
>> be in
>> > >> > favor
>> > >> > > > > > > >> of
>> > >> > > > > > > >> merging the pull request to the master after the
>> release
>> > >> > branch
>> > >> > > > > > > >> has
>> > >> > > > > > > >> been forked off. We should make sure it gets the
>> > attention
>> > >> it
>> > >> > > > > deserves
>> > >> > > > > > > >> then.
>> > >> > > > > > > >
>> > >> > > > > > > > Thanks Max! I fully agree with your reasoning. +1 to
>> not
>> > >> > include
>> > >> > > > this
>> > >> > > > > > in
>> > >> > > > > > > 1.2 now, but look at it afterwards. I hope that OK with
>> you
>> > >> > Vijay.
>> > >> > > > > > > >
>> > >> > > > > > > > - Ufuk
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Robert Metzger <rm...@apache.org>.
Hi Aljoscha,
thanks a lot for resolving this last big release blocker!
I'll definitively fork off in the release branch in the next few hours.
Stephan asked me to wait for a few more minutes for his maven changes.
The JIRAs you've mentioned seem to be pretty isolated, so its fine to merge
them afterwards to both branches.
Regards,
Robert
On Tue, Dec 20, 2016 at 4:17 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> I just merged the most important backwards compatibility changes, with
> tests.
>
> I think this one is still a blocker:
> https://issues.apache.org/jira/browse/FLINK-5320 (WindowedStream.fold()
> cannot be used). And this one is a potential blocker for some users:
> https://issues.apache.org/jira/browse/FLINK-5363.
>
> IMHO, we can cut the branch today and I'll get them in on master and both
> the 1.2 branch. What do you think?
>
> On Tue, 20 Dec 2016 at 15:24 Robert Metzger <rm...@apache.org> wrote:
>
> > Quick update here: I talked to Aljoscha offline, and the backwards
> > compatibility is still being tested (there were some bugs identified
> while
> > writing the tests).
> >
> > Also, Stephan made some fixes to the build infrastructure (
> > https://github.com/apache/flink/pull/3029) that would be good to be
> > included into the release branch.
> > To finally get the FLIP-6 branch merged to master, I'm considering
> > branching off the 1.2 release later today. It will be a little bit more
> > overhead for Stephan and Aljoscha, but it will unblock all features
> waiting
> > for a Flink 1.3 master.
> >
> >
> >
> >
> > On Fri, Dec 16, 2016 at 6:05 PM, Robert Metzger <rm...@apache.org>
> > wrote:
> >
> > > Thank you Aljoscha and Fabian for the updates.
> > > I propose *Monday evening (6pm Berlin, 9am US west coast) for feature
> > > freezing Flink 1.2 *then. This means that I'll create a release-1.2
> fork
> > > and create a 1.2 RC0 (non-voting) release candidate for testing.
> > >
> > > I don't think that I'll create the first (voting) RC until January
> > because
> > > of christmas and new years activities. Most of the committers I know
> are
> > > out of office during these 1,5 weeks.
> > >
> > >
> > >
> > > On Fri, Dec 16, 2016 at 5:55 PM, Fabian Hueske <fh...@gmail.com>
> > wrote:
> > >
> > >> Hi,
> > >>
> > >> I merged the Table API refactoring changes:
> > >>
> > >> - RESOLVED Clean up the packages of the Table API (FLINK-4704)
> > >> - RESOLVED Move Row to flink-core (FLINK-5186)
> > >>
> > >> No blockers left from my side.
> > >>
> > >> Cheers, Fabian
> > >>
> > >> 2016-12-16 17:47 GMT+01:00 Aljoscha Krettek <al...@apache.org>:
> > >>
> > >> > Yes, I'm confident that we can finish the tests until then and merge
> > the
> > >> > code.
> > >> >
> > >> > On Fri, Dec 16, 2016, 17:41 Robert Metzger <rm...@apache.org>
> > wrote:
> > >> >
> > >> > > Thank you for the update. Do you think you get it done until
> Monday
> > >> > > evening?
> > >> > >
> > >> > > On Fri, Dec 16, 2016 at 5:23 PM, Aljoscha Krettek <
> > >> aljoscha@apache.org>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi,
> > >> > > > we're still working on making the backwards compatibility from
> 1.1
> > >> > > > savepoints a reality. We have most of the code and some tests
> now
> > >> but
> > >> > it
> > >> > > > still needs some work. This is the issue that tracks the
> progress
> > on
> > >> > the
> > >> > > > operators that we would like to make backwards compatible:
> > >> > > > https://issues.apache.org/jira/browse/FLINK-5292
> > >> > > >
> > >> > > > Cheers,
> > >> > > > Aljoscha
> > >> > > >
> > >> > > > On Tue, 13 Dec 2016 at 11:22 Feng Wang <fe...@outlook.com>
> > >> wrote:
> > >> > > >
> > >> > > > > It will be pretty good if 1.2 branch could be forked off
> within
> > >> this
> > >> > > > week,
> > >> > > > > and our guys working on FLIP-6 hope FLIP-6 branch could be
> > merged
> > >> > into
> > >> > > > > master as soon as possible.
> > >> > > > >
> > >> > > > > Best Regards,
> > >> > > > >
> > >> > > > > Feng Wang
> > >> > > > >
> > >> > > > > Alibaba
> > >> > > > >
> > >> > > > > ________________________________
> > >> > > > > From: Robert Metzger <rm...@apache.org>
> > >> > > > > Sent: Tuesday, December 13, 2016 4:58 AM
> > >> > > > > To: dev@flink.apache.org
> > >> > > > > Subject: Re: [DISCUSS] Schedule and Scope for Flink 1.2
> > >> > > > >
> > >> > > > > Thank you all for figuring out a solution for the security
> pull
> > >> > > request.
> > >> > > > >
> > >> > > > >
> > >> > > > > Lets try to get 1.2 feature freezed as fast as possible so
> that
> > we
> > >> > can
> > >> > > > > "unblock" waiting features like FLIP-6 and the remaining
> > security
> > >> > > > changes.
> > >> > > > >
> > >> > > > > *What do you think about Friday evening (6pm Berlin, 9am US
> west
> > >> > coast)
> > >> > > > for
> > >> > > > > feature freezing Flink 1.2?* (only bugfixes are allowed in
> > >> > afterwards)
> > >> > > > > I'll then fork-off a "release-1.2" branch and update the
> version
> > >> in
> > >> > > > > "master" to 1.3-SNAPSHOT.
> > >> > > > > Please object if you have a bigger change or any other
> > >> reservations
> > >> > > > > regarding the feature freeze date!
> > >> > > > >
> > >> > > > > This is my current view of things on the release:
> > >> > > > >
> > >> > > > > - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
> > >> > > > > - RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
> > >> > > > > - UNRESOLVED Add Flink 1.1 savepoint backwards compatability
> > >> > > (FLINK-4797)
> > >> > > > > - RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos
> > >> > > (FLINK-1984)
> > >> > > > > - UNDER DISCUSSION Secure Data Access (FLINK-3930)
> > >> > > > > - RESOLVED Queryable State (FLINK-3779)
> > >> > > > > - RESOLVED Metrics in Webinterface (FLINK-4389)
> > >> > > > > - RESOLVED Kafka 0.10 support (FLINK-4035)
> > >> > > > > - RESOLVED Table API: Group Window Aggregates (FLINK-4691,
> > >> FLIP-11)
> > >> > > > > - RESOLVED Table API: Scalar Functions (FLINK-3097)
> > >> > > > > Added by Stephan:
> > >> > > > > - NON-BLOCKING [Pending PR] Provide support for asynchronous
> > >> > operations
> > >> > > > > over streams (FLINK-4391)
> > >> > > > > - NON-BLOCKING [beginning of next week] Unify Savepoints and
> > >> > > Checkpoints
> > >> > > > > (FLINK-4484)
> > >> > > > > Added by Fabian:
> > >> > > > > - ONGOING [Pending PR] Clean up the packages of the Table API
> > >> > > > (FLINK-4704)
> > >> > > > > Move Row to flink-core (
> > >> > > > > Added by Max:
> > >> > > > > - ONGOING [Pending PR] Change Akka configuration to allow
> > >> accessing
> > >> > > > actors
> > >> > > > > from different URLs (FLINK-2821)
> > >> > > > >
> > >> > > > >
> > >> > > > > On Mon, Dec 12, 2016 at 5:43 PM, Stephan Ewen <
> sewen@apache.org
> > >
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > Hi Vijay!
> > >> > > > > >
> > >> > > > > > The workaround you suggest may be doable, but I am wondering
> > how
> > >> > much
> > >> > > > > that
> > >> > > > > > helps, because the authorization feature would be incomplete
> > >> like
> > >> > > that
> > >> > > > > and
> > >> > > > > > thus of limited use.
> > >> > > > > >
> > >> > > > > > I would also assume that merging it properly and in full use
> > >> after
> > >> > > the
> > >> > > > > 1.2
> > >> > > > > > release would be a bit better - in general, we have often
> > >> avoided
> > >> > > last
> > >> > > > > > minute additions of sensitive and complex features.
> > >> > > > > >
> > >> > > > > > Do you think it is more urgent to have this in Flink?
> > >> > > > > >
> > >> > > > > > Best,
> > >> > > > > > Stephan
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Mon, Dec 12, 2016 at 2:49 PM, Vijay
> > >> > <vijikarthi@yahoo.com.invalid
> > >> > > >
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Max and Ufuk, I respect your concerns and fully understand
> > the
> > >> > > > > importance
> > >> > > > > > > of the network layer stack in Flink code base. Will you be
> > >> > > > comfortable
> > >> > > > > to
> > >> > > > > > > merge the code if I remove the Netty layer changes and
> leave
> > >> the
> > >> > > rest
> > >> > > > > of
> > >> > > > > > > the code. We can address the Netty code changes post 1.2
> > >> release?
> > >> > > > > > >
> > >> > > > > > > Regards,
> > >> > > > > > > Vijay
> > >> > > > > > >
> > >> > > > > > > Sent from my iPhone
> > >> > > > > > >
> > >> > > > > > > > On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <
> uce@apache.org>
> > >> > wrote:
> > >> > > > > > > >
> > >> > > > > > > > On 12 December 2016 at 12:30:31, Maximilian Michels (
> > >> > > > mxm@apache.org)
> > >> > > > > > > wrote:
> > >> > > > > > > >>> It seems like we lack the resources for now to
> properly
> > to
> > >> > take
> > >> > > > > > > >> care
> > >> > > > > > > >> of your pull request before the release. Unless someone
> > >> from
> > >> > > > > > > >> the
> > >> > > > > > > >> community is really eager to help out here, I would be
> in
> > >> > favor
> > >> > > > > > > >> of
> > >> > > > > > > >> merging the pull request to the master after the
> release
> > >> > branch
> > >> > > > > > > >> has
> > >> > > > > > > >> been forked off. We should make sure it gets the
> > attention
> > >> it
> > >> > > > > deserves
> > >> > > > > > > >> then.
> > >> > > > > > > >
> > >> > > > > > > > Thanks Max! I fully agree with your reasoning. +1 to not
> > >> > include
> > >> > > > this
> > >> > > > > > in
> > >> > > > > > > 1.2 now, but look at it afterwards. I hope that OK with
> you
> > >> > Vijay.
> > >> > > > > > > >
> > >> > > > > > > > - Ufuk
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Aljoscha Krettek <al...@apache.org>.
I just merged the most important backwards compatibility changes, with
tests.
I think this one is still a blocker:
https://issues.apache.org/jira/browse/FLINK-5320 (WindowedStream.fold()
cannot be used). And this one is a potential blocker for some users:
https://issues.apache.org/jira/browse/FLINK-5363.
IMHO, we can cut the branch today and I'll get them in on master and both
the 1.2 branch. What do you think?
On Tue, 20 Dec 2016 at 15:24 Robert Metzger <rm...@apache.org> wrote:
> Quick update here: I talked to Aljoscha offline, and the backwards
> compatibility is still being tested (there were some bugs identified while
> writing the tests).
>
> Also, Stephan made some fixes to the build infrastructure (
> https://github.com/apache/flink/pull/3029) that would be good to be
> included into the release branch.
> To finally get the FLIP-6 branch merged to master, I'm considering
> branching off the 1.2 release later today. It will be a little bit more
> overhead for Stephan and Aljoscha, but it will unblock all features waiting
> for a Flink 1.3 master.
>
>
>
>
> On Fri, Dec 16, 2016 at 6:05 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
> > Thank you Aljoscha and Fabian for the updates.
> > I propose *Monday evening (6pm Berlin, 9am US west coast) for feature
> > freezing Flink 1.2 *then. This means that I'll create a release-1.2 fork
> > and create a 1.2 RC0 (non-voting) release candidate for testing.
> >
> > I don't think that I'll create the first (voting) RC until January
> because
> > of christmas and new years activities. Most of the committers I know are
> > out of office during these 1,5 weeks.
> >
> >
> >
> > On Fri, Dec 16, 2016 at 5:55 PM, Fabian Hueske <fh...@gmail.com>
> wrote:
> >
> >> Hi,
> >>
> >> I merged the Table API refactoring changes:
> >>
> >> - RESOLVED Clean up the packages of the Table API (FLINK-4704)
> >> - RESOLVED Move Row to flink-core (FLINK-5186)
> >>
> >> No blockers left from my side.
> >>
> >> Cheers, Fabian
> >>
> >> 2016-12-16 17:47 GMT+01:00 Aljoscha Krettek <al...@apache.org>:
> >>
> >> > Yes, I'm confident that we can finish the tests until then and merge
> the
> >> > code.
> >> >
> >> > On Fri, Dec 16, 2016, 17:41 Robert Metzger <rm...@apache.org>
> wrote:
> >> >
> >> > > Thank you for the update. Do you think you get it done until Monday
> >> > > evening?
> >> > >
> >> > > On Fri, Dec 16, 2016 at 5:23 PM, Aljoscha Krettek <
> >> aljoscha@apache.org>
> >> > > wrote:
> >> > >
> >> > > > Hi,
> >> > > > we're still working on making the backwards compatibility from 1.1
> >> > > > savepoints a reality. We have most of the code and some tests now
> >> but
> >> > it
> >> > > > still needs some work. This is the issue that tracks the progress
> on
> >> > the
> >> > > > operators that we would like to make backwards compatible:
> >> > > > https://issues.apache.org/jira/browse/FLINK-5292
> >> > > >
> >> > > > Cheers,
> >> > > > Aljoscha
> >> > > >
> >> > > > On Tue, 13 Dec 2016 at 11:22 Feng Wang <fe...@outlook.com>
> >> wrote:
> >> > > >
> >> > > > > It will be pretty good if 1.2 branch could be forked off within
> >> this
> >> > > > week,
> >> > > > > and our guys working on FLIP-6 hope FLIP-6 branch could be
> merged
> >> > into
> >> > > > > master as soon as possible.
> >> > > > >
> >> > > > > Best Regards,
> >> > > > >
> >> > > > > Feng Wang
> >> > > > >
> >> > > > > Alibaba
> >> > > > >
> >> > > > > ________________________________
> >> > > > > From: Robert Metzger <rm...@apache.org>
> >> > > > > Sent: Tuesday, December 13, 2016 4:58 AM
> >> > > > > To: dev@flink.apache.org
> >> > > > > Subject: Re: [DISCUSS] Schedule and Scope for Flink 1.2
> >> > > > >
> >> > > > > Thank you all for figuring out a solution for the security pull
> >> > > request.
> >> > > > >
> >> > > > >
> >> > > > > Lets try to get 1.2 feature freezed as fast as possible so that
> we
> >> > can
> >> > > > > "unblock" waiting features like FLIP-6 and the remaining
> security
> >> > > > changes.
> >> > > > >
> >> > > > > *What do you think about Friday evening (6pm Berlin, 9am US west
> >> > coast)
> >> > > > for
> >> > > > > feature freezing Flink 1.2?* (only bugfixes are allowed in
> >> > afterwards)
> >> > > > > I'll then fork-off a "release-1.2" branch and update the version
> >> in
> >> > > > > "master" to 1.3-SNAPSHOT.
> >> > > > > Please object if you have a bigger change or any other
> >> reservations
> >> > > > > regarding the feature freeze date!
> >> > > > >
> >> > > > > This is my current view of things on the release:
> >> > > > >
> >> > > > > - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
> >> > > > > - RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
> >> > > > > - UNRESOLVED Add Flink 1.1 savepoint backwards compatability
> >> > > (FLINK-4797)
> >> > > > > - RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos
> >> > > (FLINK-1984)
> >> > > > > - UNDER DISCUSSION Secure Data Access (FLINK-3930)
> >> > > > > - RESOLVED Queryable State (FLINK-3779)
> >> > > > > - RESOLVED Metrics in Webinterface (FLINK-4389)
> >> > > > > - RESOLVED Kafka 0.10 support (FLINK-4035)
> >> > > > > - RESOLVED Table API: Group Window Aggregates (FLINK-4691,
> >> FLIP-11)
> >> > > > > - RESOLVED Table API: Scalar Functions (FLINK-3097)
> >> > > > > Added by Stephan:
> >> > > > > - NON-BLOCKING [Pending PR] Provide support for asynchronous
> >> > operations
> >> > > > > over streams (FLINK-4391)
> >> > > > > - NON-BLOCKING [beginning of next week] Unify Savepoints and
> >> > > Checkpoints
> >> > > > > (FLINK-4484)
> >> > > > > Added by Fabian:
> >> > > > > - ONGOING [Pending PR] Clean up the packages of the Table API
> >> > > > (FLINK-4704)
> >> > > > > Move Row to flink-core (
> >> > > > > Added by Max:
> >> > > > > - ONGOING [Pending PR] Change Akka configuration to allow
> >> accessing
> >> > > > actors
> >> > > > > from different URLs (FLINK-2821)
> >> > > > >
> >> > > > >
> >> > > > > On Mon, Dec 12, 2016 at 5:43 PM, Stephan Ewen <sewen@apache.org
> >
> >> > > wrote:
> >> > > > >
> >> > > > > > Hi Vijay!
> >> > > > > >
> >> > > > > > The workaround you suggest may be doable, but I am wondering
> how
> >> > much
> >> > > > > that
> >> > > > > > helps, because the authorization feature would be incomplete
> >> like
> >> > > that
> >> > > > > and
> >> > > > > > thus of limited use.
> >> > > > > >
> >> > > > > > I would also assume that merging it properly and in full use
> >> after
> >> > > the
> >> > > > > 1.2
> >> > > > > > release would be a bit better - in general, we have often
> >> avoided
> >> > > last
> >> > > > > > minute additions of sensitive and complex features.
> >> > > > > >
> >> > > > > > Do you think it is more urgent to have this in Flink?
> >> > > > > >
> >> > > > > > Best,
> >> > > > > > Stephan
> >> > > > > >
> >> > > > > >
> >> > > > > > On Mon, Dec 12, 2016 at 2:49 PM, Vijay
> >> > <vijikarthi@yahoo.com.invalid
> >> > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Max and Ufuk, I respect your concerns and fully understand
> the
> >> > > > > importance
> >> > > > > > > of the network layer stack in Flink code base. Will you be
> >> > > > comfortable
> >> > > > > to
> >> > > > > > > merge the code if I remove the Netty layer changes and leave
> >> the
> >> > > rest
> >> > > > > of
> >> > > > > > > the code. We can address the Netty code changes post 1.2
> >> release?
> >> > > > > > >
> >> > > > > > > Regards,
> >> > > > > > > Vijay
> >> > > > > > >
> >> > > > > > > Sent from my iPhone
> >> > > > > > >
> >> > > > > > > > On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <uc...@apache.org>
> >> > wrote:
> >> > > > > > > >
> >> > > > > > > > On 12 December 2016 at 12:30:31, Maximilian Michels (
> >> > > > mxm@apache.org)
> >> > > > > > > wrote:
> >> > > > > > > >>> It seems like we lack the resources for now to properly
> to
> >> > take
> >> > > > > > > >> care
> >> > > > > > > >> of your pull request before the release. Unless someone
> >> from
> >> > > > > > > >> the
> >> > > > > > > >> community is really eager to help out here, I would be in
> >> > favor
> >> > > > > > > >> of
> >> > > > > > > >> merging the pull request to the master after the release
> >> > branch
> >> > > > > > > >> has
> >> > > > > > > >> been forked off. We should make sure it gets the
> attention
> >> it
> >> > > > > deserves
> >> > > > > > > >> then.
> >> > > > > > > >
> >> > > > > > > > Thanks Max! I fully agree with your reasoning. +1 to not
> >> > include
> >> > > > this
> >> > > > > > in
> >> > > > > > > 1.2 now, but look at it afterwards. I hope that OK with you
> >> > Vijay.
> >> > > > > > > >
> >> > > > > > > > - Ufuk
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Robert Metzger <rm...@apache.org>.
Quick update here: I talked to Aljoscha offline, and the backwards
compatibility is still being tested (there were some bugs identified while
writing the tests).
Also, Stephan made some fixes to the build infrastructure (
https://github.com/apache/flink/pull/3029) that would be good to be
included into the release branch.
To finally get the FLIP-6 branch merged to master, I'm considering
branching off the 1.2 release later today. It will be a little bit more
overhead for Stephan and Aljoscha, but it will unblock all features waiting
for a Flink 1.3 master.
On Fri, Dec 16, 2016 at 6:05 PM, Robert Metzger <rm...@apache.org> wrote:
> Thank you Aljoscha and Fabian for the updates.
> I propose *Monday evening (6pm Berlin, 9am US west coast) for feature
> freezing Flink 1.2 *then. This means that I'll create a release-1.2 fork
> and create a 1.2 RC0 (non-voting) release candidate for testing.
>
> I don't think that I'll create the first (voting) RC until January because
> of christmas and new years activities. Most of the committers I know are
> out of office during these 1,5 weeks.
>
>
>
> On Fri, Dec 16, 2016 at 5:55 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
>> Hi,
>>
>> I merged the Table API refactoring changes:
>>
>> - RESOLVED Clean up the packages of the Table API (FLINK-4704)
>> - RESOLVED Move Row to flink-core (FLINK-5186)
>>
>> No blockers left from my side.
>>
>> Cheers, Fabian
>>
>> 2016-12-16 17:47 GMT+01:00 Aljoscha Krettek <al...@apache.org>:
>>
>> > Yes, I'm confident that we can finish the tests until then and merge the
>> > code.
>> >
>> > On Fri, Dec 16, 2016, 17:41 Robert Metzger <rm...@apache.org> wrote:
>> >
>> > > Thank you for the update. Do you think you get it done until Monday
>> > > evening?
>> > >
>> > > On Fri, Dec 16, 2016 at 5:23 PM, Aljoscha Krettek <
>> aljoscha@apache.org>
>> > > wrote:
>> > >
>> > > > Hi,
>> > > > we're still working on making the backwards compatibility from 1.1
>> > > > savepoints a reality. We have most of the code and some tests now
>> but
>> > it
>> > > > still needs some work. This is the issue that tracks the progress on
>> > the
>> > > > operators that we would like to make backwards compatible:
>> > > > https://issues.apache.org/jira/browse/FLINK-5292
>> > > >
>> > > > Cheers,
>> > > > Aljoscha
>> > > >
>> > > > On Tue, 13 Dec 2016 at 11:22 Feng Wang <fe...@outlook.com>
>> wrote:
>> > > >
>> > > > > It will be pretty good if 1.2 branch could be forked off within
>> this
>> > > > week,
>> > > > > and our guys working on FLIP-6 hope FLIP-6 branch could be merged
>> > into
>> > > > > master as soon as possible.
>> > > > >
>> > > > > Best Regards,
>> > > > >
>> > > > > Feng Wang
>> > > > >
>> > > > > Alibaba
>> > > > >
>> > > > > ________________________________
>> > > > > From: Robert Metzger <rm...@apache.org>
>> > > > > Sent: Tuesday, December 13, 2016 4:58 AM
>> > > > > To: dev@flink.apache.org
>> > > > > Subject: Re: [DISCUSS] Schedule and Scope for Flink 1.2
>> > > > >
>> > > > > Thank you all for figuring out a solution for the security pull
>> > > request.
>> > > > >
>> > > > >
>> > > > > Lets try to get 1.2 feature freezed as fast as possible so that we
>> > can
>> > > > > "unblock" waiting features like FLIP-6 and the remaining security
>> > > > changes.
>> > > > >
>> > > > > *What do you think about Friday evening (6pm Berlin, 9am US west
>> > coast)
>> > > > for
>> > > > > feature freezing Flink 1.2?* (only bugfixes are allowed in
>> > afterwards)
>> > > > > I'll then fork-off a "release-1.2" branch and update the version
>> in
>> > > > > "master" to 1.3-SNAPSHOT.
>> > > > > Please object if you have a bigger change or any other
>> reservations
>> > > > > regarding the feature freeze date!
>> > > > >
>> > > > > This is my current view of things on the release:
>> > > > >
>> > > > > - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
>> > > > > - RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
>> > > > > - UNRESOLVED Add Flink 1.1 savepoint backwards compatability
>> > > (FLINK-4797)
>> > > > > - RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos
>> > > (FLINK-1984)
>> > > > > - UNDER DISCUSSION Secure Data Access (FLINK-3930)
>> > > > > - RESOLVED Queryable State (FLINK-3779)
>> > > > > - RESOLVED Metrics in Webinterface (FLINK-4389)
>> > > > > - RESOLVED Kafka 0.10 support (FLINK-4035)
>> > > > > - RESOLVED Table API: Group Window Aggregates (FLINK-4691,
>> FLIP-11)
>> > > > > - RESOLVED Table API: Scalar Functions (FLINK-3097)
>> > > > > Added by Stephan:
>> > > > > - NON-BLOCKING [Pending PR] Provide support for asynchronous
>> > operations
>> > > > > over streams (FLINK-4391)
>> > > > > - NON-BLOCKING [beginning of next week] Unify Savepoints and
>> > > Checkpoints
>> > > > > (FLINK-4484)
>> > > > > Added by Fabian:
>> > > > > - ONGOING [Pending PR] Clean up the packages of the Table API
>> > > > (FLINK-4704)
>> > > > > Move Row to flink-core (
>> > > > > Added by Max:
>> > > > > - ONGOING [Pending PR] Change Akka configuration to allow
>> accessing
>> > > > actors
>> > > > > from different URLs (FLINK-2821)
>> > > > >
>> > > > >
>> > > > > On Mon, Dec 12, 2016 at 5:43 PM, Stephan Ewen <se...@apache.org>
>> > > wrote:
>> > > > >
>> > > > > > Hi Vijay!
>> > > > > >
>> > > > > > The workaround you suggest may be doable, but I am wondering how
>> > much
>> > > > > that
>> > > > > > helps, because the authorization feature would be incomplete
>> like
>> > > that
>> > > > > and
>> > > > > > thus of limited use.
>> > > > > >
>> > > > > > I would also assume that merging it properly and in full use
>> after
>> > > the
>> > > > > 1.2
>> > > > > > release would be a bit better - in general, we have often
>> avoided
>> > > last
>> > > > > > minute additions of sensitive and complex features.
>> > > > > >
>> > > > > > Do you think it is more urgent to have this in Flink?
>> > > > > >
>> > > > > > Best,
>> > > > > > Stephan
>> > > > > >
>> > > > > >
>> > > > > > On Mon, Dec 12, 2016 at 2:49 PM, Vijay
>> > <vijikarthi@yahoo.com.invalid
>> > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Max and Ufuk, I respect your concerns and fully understand the
>> > > > > importance
>> > > > > > > of the network layer stack in Flink code base. Will you be
>> > > > comfortable
>> > > > > to
>> > > > > > > merge the code if I remove the Netty layer changes and leave
>> the
>> > > rest
>> > > > > of
>> > > > > > > the code. We can address the Netty code changes post 1.2
>> release?
>> > > > > > >
>> > > > > > > Regards,
>> > > > > > > Vijay
>> > > > > > >
>> > > > > > > Sent from my iPhone
>> > > > > > >
>> > > > > > > > On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <uc...@apache.org>
>> > wrote:
>> > > > > > > >
>> > > > > > > > On 12 December 2016 at 12:30:31, Maximilian Michels (
>> > > > mxm@apache.org)
>> > > > > > > wrote:
>> > > > > > > >>> It seems like we lack the resources for now to properly to
>> > take
>> > > > > > > >> care
>> > > > > > > >> of your pull request before the release. Unless someone
>> from
>> > > > > > > >> the
>> > > > > > > >> community is really eager to help out here, I would be in
>> > favor
>> > > > > > > >> of
>> > > > > > > >> merging the pull request to the master after the release
>> > branch
>> > > > > > > >> has
>> > > > > > > >> been forked off. We should make sure it gets the attention
>> it
>> > > > > deserves
>> > > > > > > >> then.
>> > > > > > > >
>> > > > > > > > Thanks Max! I fully agree with your reasoning. +1 to not
>> > include
>> > > > this
>> > > > > > in
>> > > > > > > 1.2 now, but look at it afterwards. I hope that OK with you
>> > Vijay.
>> > > > > > > >
>> > > > > > > > - Ufuk
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Robert Metzger <rm...@apache.org>.
Thank you Aljoscha and Fabian for the updates.
I propose *Monday evening (6pm Berlin, 9am US west coast) for feature
freezing Flink 1.2 *then. This means that I'll create a release-1.2 fork
and create a 1.2 RC0 (non-voting) release candidate for testing.
I don't think that I'll create the first (voting) RC until January because
of christmas and new years activities. Most of the committers I know are
out of office during these 1,5 weeks.
On Fri, Dec 16, 2016 at 5:55 PM, Fabian Hueske <fh...@gmail.com> wrote:
> Hi,
>
> I merged the Table API refactoring changes:
>
> - RESOLVED Clean up the packages of the Table API (FLINK-4704)
> - RESOLVED Move Row to flink-core (FLINK-5186)
>
> No blockers left from my side.
>
> Cheers, Fabian
>
> 2016-12-16 17:47 GMT+01:00 Aljoscha Krettek <al...@apache.org>:
>
> > Yes, I'm confident that we can finish the tests until then and merge the
> > code.
> >
> > On Fri, Dec 16, 2016, 17:41 Robert Metzger <rm...@apache.org> wrote:
> >
> > > Thank you for the update. Do you think you get it done until Monday
> > > evening?
> > >
> > > On Fri, Dec 16, 2016 at 5:23 PM, Aljoscha Krettek <aljoscha@apache.org
> >
> > > wrote:
> > >
> > > > Hi,
> > > > we're still working on making the backwards compatibility from 1.1
> > > > savepoints a reality. We have most of the code and some tests now but
> > it
> > > > still needs some work. This is the issue that tracks the progress on
> > the
> > > > operators that we would like to make backwards compatible:
> > > > https://issues.apache.org/jira/browse/FLINK-5292
> > > >
> > > > Cheers,
> > > > Aljoscha
> > > >
> > > > On Tue, 13 Dec 2016 at 11:22 Feng Wang <fe...@outlook.com>
> wrote:
> > > >
> > > > > It will be pretty good if 1.2 branch could be forked off within
> this
> > > > week,
> > > > > and our guys working on FLIP-6 hope FLIP-6 branch could be merged
> > into
> > > > > master as soon as possible.
> > > > >
> > > > > Best Regards,
> > > > >
> > > > > Feng Wang
> > > > >
> > > > > Alibaba
> > > > >
> > > > > ________________________________
> > > > > From: Robert Metzger <rm...@apache.org>
> > > > > Sent: Tuesday, December 13, 2016 4:58 AM
> > > > > To: dev@flink.apache.org
> > > > > Subject: Re: [DISCUSS] Schedule and Scope for Flink 1.2
> > > > >
> > > > > Thank you all for figuring out a solution for the security pull
> > > request.
> > > > >
> > > > >
> > > > > Lets try to get 1.2 feature freezed as fast as possible so that we
> > can
> > > > > "unblock" waiting features like FLIP-6 and the remaining security
> > > > changes.
> > > > >
> > > > > *What do you think about Friday evening (6pm Berlin, 9am US west
> > coast)
> > > > for
> > > > > feature freezing Flink 1.2?* (only bugfixes are allowed in
> > afterwards)
> > > > > I'll then fork-off a "release-1.2" branch and update the version in
> > > > > "master" to 1.3-SNAPSHOT.
> > > > > Please object if you have a bigger change or any other reservations
> > > > > regarding the feature freeze date!
> > > > >
> > > > > This is my current view of things on the release:
> > > > >
> > > > > - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
> > > > > - RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
> > > > > - UNRESOLVED Add Flink 1.1 savepoint backwards compatability
> > > (FLINK-4797)
> > > > > - RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos
> > > (FLINK-1984)
> > > > > - UNDER DISCUSSION Secure Data Access (FLINK-3930)
> > > > > - RESOLVED Queryable State (FLINK-3779)
> > > > > - RESOLVED Metrics in Webinterface (FLINK-4389)
> > > > > - RESOLVED Kafka 0.10 support (FLINK-4035)
> > > > > - RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
> > > > > - RESOLVED Table API: Scalar Functions (FLINK-3097)
> > > > > Added by Stephan:
> > > > > - NON-BLOCKING [Pending PR] Provide support for asynchronous
> > operations
> > > > > over streams (FLINK-4391)
> > > > > - NON-BLOCKING [beginning of next week] Unify Savepoints and
> > > Checkpoints
> > > > > (FLINK-4484)
> > > > > Added by Fabian:
> > > > > - ONGOING [Pending PR] Clean up the packages of the Table API
> > > > (FLINK-4704)
> > > > > Move Row to flink-core (
> > > > > Added by Max:
> > > > > - ONGOING [Pending PR] Change Akka configuration to allow accessing
> > > > actors
> > > > > from different URLs (FLINK-2821)
> > > > >
> > > > >
> > > > > On Mon, Dec 12, 2016 at 5:43 PM, Stephan Ewen <se...@apache.org>
> > > wrote:
> > > > >
> > > > > > Hi Vijay!
> > > > > >
> > > > > > The workaround you suggest may be doable, but I am wondering how
> > much
> > > > > that
> > > > > > helps, because the authorization feature would be incomplete like
> > > that
> > > > > and
> > > > > > thus of limited use.
> > > > > >
> > > > > > I would also assume that merging it properly and in full use
> after
> > > the
> > > > > 1.2
> > > > > > release would be a bit better - in general, we have often avoided
> > > last
> > > > > > minute additions of sensitive and complex features.
> > > > > >
> > > > > > Do you think it is more urgent to have this in Flink?
> > > > > >
> > > > > > Best,
> > > > > > Stephan
> > > > > >
> > > > > >
> > > > > > On Mon, Dec 12, 2016 at 2:49 PM, Vijay
> > <vijikarthi@yahoo.com.invalid
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Max and Ufuk, I respect your concerns and fully understand the
> > > > > importance
> > > > > > > of the network layer stack in Flink code base. Will you be
> > > > comfortable
> > > > > to
> > > > > > > merge the code if I remove the Netty layer changes and leave
> the
> > > rest
> > > > > of
> > > > > > > the code. We can address the Netty code changes post 1.2
> release?
> > > > > > >
> > > > > > > Regards,
> > > > > > > Vijay
> > > > > > >
> > > > > > > Sent from my iPhone
> > > > > > >
> > > > > > > > On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <uc...@apache.org>
> > wrote:
> > > > > > > >
> > > > > > > > On 12 December 2016 at 12:30:31, Maximilian Michels (
> > > > mxm@apache.org)
> > > > > > > wrote:
> > > > > > > >>> It seems like we lack the resources for now to properly to
> > take
> > > > > > > >> care
> > > > > > > >> of your pull request before the release. Unless someone from
> > > > > > > >> the
> > > > > > > >> community is really eager to help out here, I would be in
> > favor
> > > > > > > >> of
> > > > > > > >> merging the pull request to the master after the release
> > branch
> > > > > > > >> has
> > > > > > > >> been forked off. We should make sure it gets the attention
> it
> > > > > deserves
> > > > > > > >> then.
> > > > > > > >
> > > > > > > > Thanks Max! I fully agree with your reasoning. +1 to not
> > include
> > > > this
> > > > > > in
> > > > > > > 1.2 now, but look at it afterwards. I hope that OK with you
> > Vijay.
> > > > > > > >
> > > > > > > > - Ufuk
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
I merged the Table API refactoring changes:
- RESOLVED Clean up the packages of the Table API (FLINK-4704)
- RESOLVED Move Row to flink-core (FLINK-5186)
No blockers left from my side.
Cheers, Fabian
2016-12-16 17:47 GMT+01:00 Aljoscha Krettek <al...@apache.org>:
> Yes, I'm confident that we can finish the tests until then and merge the
> code.
>
> On Fri, Dec 16, 2016, 17:41 Robert Metzger <rm...@apache.org> wrote:
>
> > Thank you for the update. Do you think you get it done until Monday
> > evening?
> >
> > On Fri, Dec 16, 2016 at 5:23 PM, Aljoscha Krettek <al...@apache.org>
> > wrote:
> >
> > > Hi,
> > > we're still working on making the backwards compatibility from 1.1
> > > savepoints a reality. We have most of the code and some tests now but
> it
> > > still needs some work. This is the issue that tracks the progress on
> the
> > > operators that we would like to make backwards compatible:
> > > https://issues.apache.org/jira/browse/FLINK-5292
> > >
> > > Cheers,
> > > Aljoscha
> > >
> > > On Tue, 13 Dec 2016 at 11:22 Feng Wang <fe...@outlook.com> wrote:
> > >
> > > > It will be pretty good if 1.2 branch could be forked off within this
> > > week,
> > > > and our guys working on FLIP-6 hope FLIP-6 branch could be merged
> into
> > > > master as soon as possible.
> > > >
> > > > Best Regards,
> > > >
> > > > Feng Wang
> > > >
> > > > Alibaba
> > > >
> > > > ________________________________
> > > > From: Robert Metzger <rm...@apache.org>
> > > > Sent: Tuesday, December 13, 2016 4:58 AM
> > > > To: dev@flink.apache.org
> > > > Subject: Re: [DISCUSS] Schedule and Scope for Flink 1.2
> > > >
> > > > Thank you all for figuring out a solution for the security pull
> > request.
> > > >
> > > >
> > > > Lets try to get 1.2 feature freezed as fast as possible so that we
> can
> > > > "unblock" waiting features like FLIP-6 and the remaining security
> > > changes.
> > > >
> > > > *What do you think about Friday evening (6pm Berlin, 9am US west
> coast)
> > > for
> > > > feature freezing Flink 1.2?* (only bugfixes are allowed in
> afterwards)
> > > > I'll then fork-off a "release-1.2" branch and update the version in
> > > > "master" to 1.3-SNAPSHOT.
> > > > Please object if you have a bigger change or any other reservations
> > > > regarding the feature freeze date!
> > > >
> > > > This is my current view of things on the release:
> > > >
> > > > - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
> > > > - RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
> > > > - UNRESOLVED Add Flink 1.1 savepoint backwards compatability
> > (FLINK-4797)
> > > > - RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos
> > (FLINK-1984)
> > > > - UNDER DISCUSSION Secure Data Access (FLINK-3930)
> > > > - RESOLVED Queryable State (FLINK-3779)
> > > > - RESOLVED Metrics in Webinterface (FLINK-4389)
> > > > - RESOLVED Kafka 0.10 support (FLINK-4035)
> > > > - RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
> > > > - RESOLVED Table API: Scalar Functions (FLINK-3097)
> > > > Added by Stephan:
> > > > - NON-BLOCKING [Pending PR] Provide support for asynchronous
> operations
> > > > over streams (FLINK-4391)
> > > > - NON-BLOCKING [beginning of next week] Unify Savepoints and
> > Checkpoints
> > > > (FLINK-4484)
> > > > Added by Fabian:
> > > > - ONGOING [Pending PR] Clean up the packages of the Table API
> > > (FLINK-4704)
> > > > Move Row to flink-core (
> > > > Added by Max:
> > > > - ONGOING [Pending PR] Change Akka configuration to allow accessing
> > > actors
> > > > from different URLs (FLINK-2821)
> > > >
> > > >
> > > > On Mon, Dec 12, 2016 at 5:43 PM, Stephan Ewen <se...@apache.org>
> > wrote:
> > > >
> > > > > Hi Vijay!
> > > > >
> > > > > The workaround you suggest may be doable, but I am wondering how
> much
> > > > that
> > > > > helps, because the authorization feature would be incomplete like
> > that
> > > > and
> > > > > thus of limited use.
> > > > >
> > > > > I would also assume that merging it properly and in full use after
> > the
> > > > 1.2
> > > > > release would be a bit better - in general, we have often avoided
> > last
> > > > > minute additions of sensitive and complex features.
> > > > >
> > > > > Do you think it is more urgent to have this in Flink?
> > > > >
> > > > > Best,
> > > > > Stephan
> > > > >
> > > > >
> > > > > On Mon, Dec 12, 2016 at 2:49 PM, Vijay
> <vijikarthi@yahoo.com.invalid
> > >
> > > > > wrote:
> > > > >
> > > > > > Max and Ufuk, I respect your concerns and fully understand the
> > > > importance
> > > > > > of the network layer stack in Flink code base. Will you be
> > > comfortable
> > > > to
> > > > > > merge the code if I remove the Netty layer changes and leave the
> > rest
> > > > of
> > > > > > the code. We can address the Netty code changes post 1.2 release?
> > > > > >
> > > > > > Regards,
> > > > > > Vijay
> > > > > >
> > > > > > Sent from my iPhone
> > > > > >
> > > > > > > On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <uc...@apache.org>
> wrote:
> > > > > > >
> > > > > > > On 12 December 2016 at 12:30:31, Maximilian Michels (
> > > mxm@apache.org)
> > > > > > wrote:
> > > > > > >>> It seems like we lack the resources for now to properly to
> take
> > > > > > >> care
> > > > > > >> of your pull request before the release. Unless someone from
> > > > > > >> the
> > > > > > >> community is really eager to help out here, I would be in
> favor
> > > > > > >> of
> > > > > > >> merging the pull request to the master after the release
> branch
> > > > > > >> has
> > > > > > >> been forked off. We should make sure it gets the attention it
> > > > deserves
> > > > > > >> then.
> > > > > > >
> > > > > > > Thanks Max! I fully agree with your reasoning. +1 to not
> include
> > > this
> > > > > in
> > > > > > 1.2 now, but look at it afterwards. I hope that OK with you
> Vijay.
> > > > > > >
> > > > > > > - Ufuk
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Aljoscha Krettek <al...@apache.org>.
Yes, I'm confident that we can finish the tests until then and merge the
code.
On Fri, Dec 16, 2016, 17:41 Robert Metzger <rm...@apache.org> wrote:
> Thank you for the update. Do you think you get it done until Monday
> evening?
>
> On Fri, Dec 16, 2016 at 5:23 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
> > Hi,
> > we're still working on making the backwards compatibility from 1.1
> > savepoints a reality. We have most of the code and some tests now but it
> > still needs some work. This is the issue that tracks the progress on the
> > operators that we would like to make backwards compatible:
> > https://issues.apache.org/jira/browse/FLINK-5292
> >
> > Cheers,
> > Aljoscha
> >
> > On Tue, 13 Dec 2016 at 11:22 Feng Wang <fe...@outlook.com> wrote:
> >
> > > It will be pretty good if 1.2 branch could be forked off within this
> > week,
> > > and our guys working on FLIP-6 hope FLIP-6 branch could be merged into
> > > master as soon as possible.
> > >
> > > Best Regards,
> > >
> > > Feng Wang
> > >
> > > Alibaba
> > >
> > > ________________________________
> > > From: Robert Metzger <rm...@apache.org>
> > > Sent: Tuesday, December 13, 2016 4:58 AM
> > > To: dev@flink.apache.org
> > > Subject: Re: [DISCUSS] Schedule and Scope for Flink 1.2
> > >
> > > Thank you all for figuring out a solution for the security pull
> request.
> > >
> > >
> > > Lets try to get 1.2 feature freezed as fast as possible so that we can
> > > "unblock" waiting features like FLIP-6 and the remaining security
> > changes.
> > >
> > > *What do you think about Friday evening (6pm Berlin, 9am US west coast)
> > for
> > > feature freezing Flink 1.2?* (only bugfixes are allowed in afterwards)
> > > I'll then fork-off a "release-1.2" branch and update the version in
> > > "master" to 1.3-SNAPSHOT.
> > > Please object if you have a bigger change or any other reservations
> > > regarding the feature freeze date!
> > >
> > > This is my current view of things on the release:
> > >
> > > - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
> > > - RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
> > > - UNRESOLVED Add Flink 1.1 savepoint backwards compatability
> (FLINK-4797)
> > > - RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos
> (FLINK-1984)
> > > - UNDER DISCUSSION Secure Data Access (FLINK-3930)
> > > - RESOLVED Queryable State (FLINK-3779)
> > > - RESOLVED Metrics in Webinterface (FLINK-4389)
> > > - RESOLVED Kafka 0.10 support (FLINK-4035)
> > > - RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
> > > - RESOLVED Table API: Scalar Functions (FLINK-3097)
> > > Added by Stephan:
> > > - NON-BLOCKING [Pending PR] Provide support for asynchronous operations
> > > over streams (FLINK-4391)
> > > - NON-BLOCKING [beginning of next week] Unify Savepoints and
> Checkpoints
> > > (FLINK-4484)
> > > Added by Fabian:
> > > - ONGOING [Pending PR] Clean up the packages of the Table API
> > (FLINK-4704)
> > > Move Row to flink-core (
> > > Added by Max:
> > > - ONGOING [Pending PR] Change Akka configuration to allow accessing
> > actors
> > > from different URLs (FLINK-2821)
> > >
> > >
> > > On Mon, Dec 12, 2016 at 5:43 PM, Stephan Ewen <se...@apache.org>
> wrote:
> > >
> > > > Hi Vijay!
> > > >
> > > > The workaround you suggest may be doable, but I am wondering how much
> > > that
> > > > helps, because the authorization feature would be incomplete like
> that
> > > and
> > > > thus of limited use.
> > > >
> > > > I would also assume that merging it properly and in full use after
> the
> > > 1.2
> > > > release would be a bit better - in general, we have often avoided
> last
> > > > minute additions of sensitive and complex features.
> > > >
> > > > Do you think it is more urgent to have this in Flink?
> > > >
> > > > Best,
> > > > Stephan
> > > >
> > > >
> > > > On Mon, Dec 12, 2016 at 2:49 PM, Vijay <vijikarthi@yahoo.com.invalid
> >
> > > > wrote:
> > > >
> > > > > Max and Ufuk, I respect your concerns and fully understand the
> > > importance
> > > > > of the network layer stack in Flink code base. Will you be
> > comfortable
> > > to
> > > > > merge the code if I remove the Netty layer changes and leave the
> rest
> > > of
> > > > > the code. We can address the Netty code changes post 1.2 release?
> > > > >
> > > > > Regards,
> > > > > Vijay
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <uc...@apache.org> wrote:
> > > > > >
> > > > > > On 12 December 2016 at 12:30:31, Maximilian Michels (
> > mxm@apache.org)
> > > > > wrote:
> > > > > >>> It seems like we lack the resources for now to properly to take
> > > > > >> care
> > > > > >> of your pull request before the release. Unless someone from
> > > > > >> the
> > > > > >> community is really eager to help out here, I would be in favor
> > > > > >> of
> > > > > >> merging the pull request to the master after the release branch
> > > > > >> has
> > > > > >> been forked off. We should make sure it gets the attention it
> > > deserves
> > > > > >> then.
> > > > > >
> > > > > > Thanks Max! I fully agree with your reasoning. +1 to not include
> > this
> > > > in
> > > > > 1.2 now, but look at it afterwards. I hope that OK with you Vijay.
> > > > > >
> > > > > > - Ufuk
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Robert Metzger <rm...@apache.org>.
Thank you for the update. Do you think you get it done until Monday evening?
On Fri, Dec 16, 2016 at 5:23 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> Hi,
> we're still working on making the backwards compatibility from 1.1
> savepoints a reality. We have most of the code and some tests now but it
> still needs some work. This is the issue that tracks the progress on the
> operators that we would like to make backwards compatible:
> https://issues.apache.org/jira/browse/FLINK-5292
>
> Cheers,
> Aljoscha
>
> On Tue, 13 Dec 2016 at 11:22 Feng Wang <fe...@outlook.com> wrote:
>
> > It will be pretty good if 1.2 branch could be forked off within this
> week,
> > and our guys working on FLIP-6 hope FLIP-6 branch could be merged into
> > master as soon as possible.
> >
> > Best Regards,
> >
> > Feng Wang
> >
> > Alibaba
> >
> > ________________________________
> > From: Robert Metzger <rm...@apache.org>
> > Sent: Tuesday, December 13, 2016 4:58 AM
> > To: dev@flink.apache.org
> > Subject: Re: [DISCUSS] Schedule and Scope for Flink 1.2
> >
> > Thank you all for figuring out a solution for the security pull request.
> >
> >
> > Lets try to get 1.2 feature freezed as fast as possible so that we can
> > "unblock" waiting features like FLIP-6 and the remaining security
> changes.
> >
> > *What do you think about Friday evening (6pm Berlin, 9am US west coast)
> for
> > feature freezing Flink 1.2?* (only bugfixes are allowed in afterwards)
> > I'll then fork-off a "release-1.2" branch and update the version in
> > "master" to 1.3-SNAPSHOT.
> > Please object if you have a bigger change or any other reservations
> > regarding the feature freeze date!
> >
> > This is my current view of things on the release:
> >
> > - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
> > - RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
> > - UNRESOLVED Add Flink 1.1 savepoint backwards compatability (FLINK-4797)
> > - RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos (FLINK-1984)
> > - UNDER DISCUSSION Secure Data Access (FLINK-3930)
> > - RESOLVED Queryable State (FLINK-3779)
> > - RESOLVED Metrics in Webinterface (FLINK-4389)
> > - RESOLVED Kafka 0.10 support (FLINK-4035)
> > - RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
> > - RESOLVED Table API: Scalar Functions (FLINK-3097)
> > Added by Stephan:
> > - NON-BLOCKING [Pending PR] Provide support for asynchronous operations
> > over streams (FLINK-4391)
> > - NON-BLOCKING [beginning of next week] Unify Savepoints and Checkpoints
> > (FLINK-4484)
> > Added by Fabian:
> > - ONGOING [Pending PR] Clean up the packages of the Table API
> (FLINK-4704)
> > Move Row to flink-core (
> > Added by Max:
> > - ONGOING [Pending PR] Change Akka configuration to allow accessing
> actors
> > from different URLs (FLINK-2821)
> >
> >
> > On Mon, Dec 12, 2016 at 5:43 PM, Stephan Ewen <se...@apache.org> wrote:
> >
> > > Hi Vijay!
> > >
> > > The workaround you suggest may be doable, but I am wondering how much
> > that
> > > helps, because the authorization feature would be incomplete like that
> > and
> > > thus of limited use.
> > >
> > > I would also assume that merging it properly and in full use after the
> > 1.2
> > > release would be a bit better - in general, we have often avoided last
> > > minute additions of sensitive and complex features.
> > >
> > > Do you think it is more urgent to have this in Flink?
> > >
> > > Best,
> > > Stephan
> > >
> > >
> > > On Mon, Dec 12, 2016 at 2:49 PM, Vijay <vi...@yahoo.com.invalid>
> > > wrote:
> > >
> > > > Max and Ufuk, I respect your concerns and fully understand the
> > importance
> > > > of the network layer stack in Flink code base. Will you be
> comfortable
> > to
> > > > merge the code if I remove the Netty layer changes and leave the rest
> > of
> > > > the code. We can address the Netty code changes post 1.2 release?
> > > >
> > > > Regards,
> > > > Vijay
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <uc...@apache.org> wrote:
> > > > >
> > > > > On 12 December 2016 at 12:30:31, Maximilian Michels (
> mxm@apache.org)
> > > > wrote:
> > > > >>> It seems like we lack the resources for now to properly to take
> > > > >> care
> > > > >> of your pull request before the release. Unless someone from
> > > > >> the
> > > > >> community is really eager to help out here, I would be in favor
> > > > >> of
> > > > >> merging the pull request to the master after the release branch
> > > > >> has
> > > > >> been forked off. We should make sure it gets the attention it
> > deserves
> > > > >> then.
> > > > >
> > > > > Thanks Max! I fully agree with your reasoning. +1 to not include
> this
> > > in
> > > > 1.2 now, but look at it afterwards. I hope that OK with you Vijay.
> > > > >
> > > > > - Ufuk
> > > > >
> > > > >
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
we're still working on making the backwards compatibility from 1.1
savepoints a reality. We have most of the code and some tests now but it
still needs some work. This is the issue that tracks the progress on the
operators that we would like to make backwards compatible:
https://issues.apache.org/jira/browse/FLINK-5292
Cheers,
Aljoscha
On Tue, 13 Dec 2016 at 11:22 Feng Wang <fe...@outlook.com> wrote:
> It will be pretty good if 1.2 branch could be forked off within this week,
> and our guys working on FLIP-6 hope FLIP-6 branch could be merged into
> master as soon as possible.
>
> Best Regards,
>
> Feng Wang
>
> Alibaba
>
> ________________________________
> From: Robert Metzger <rm...@apache.org>
> Sent: Tuesday, December 13, 2016 4:58 AM
> To: dev@flink.apache.org
> Subject: Re: [DISCUSS] Schedule and Scope for Flink 1.2
>
> Thank you all for figuring out a solution for the security pull request.
>
>
> Lets try to get 1.2 feature freezed as fast as possible so that we can
> "unblock" waiting features like FLIP-6 and the remaining security changes.
>
> *What do you think about Friday evening (6pm Berlin, 9am US west coast) for
> feature freezing Flink 1.2?* (only bugfixes are allowed in afterwards)
> I'll then fork-off a "release-1.2" branch and update the version in
> "master" to 1.3-SNAPSHOT.
> Please object if you have a bigger change or any other reservations
> regarding the feature freeze date!
>
> This is my current view of things on the release:
>
> - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
> - RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
> - UNRESOLVED Add Flink 1.1 savepoint backwards compatability (FLINK-4797)
> - RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos (FLINK-1984)
> - UNDER DISCUSSION Secure Data Access (FLINK-3930)
> - RESOLVED Queryable State (FLINK-3779)
> - RESOLVED Metrics in Webinterface (FLINK-4389)
> - RESOLVED Kafka 0.10 support (FLINK-4035)
> - RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
> - RESOLVED Table API: Scalar Functions (FLINK-3097)
> Added by Stephan:
> - NON-BLOCKING [Pending PR] Provide support for asynchronous operations
> over streams (FLINK-4391)
> - NON-BLOCKING [beginning of next week] Unify Savepoints and Checkpoints
> (FLINK-4484)
> Added by Fabian:
> - ONGOING [Pending PR] Clean up the packages of the Table API (FLINK-4704)
> Move Row to flink-core (
> Added by Max:
> - ONGOING [Pending PR] Change Akka configuration to allow accessing actors
> from different URLs (FLINK-2821)
>
>
> On Mon, Dec 12, 2016 at 5:43 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > Hi Vijay!
> >
> > The workaround you suggest may be doable, but I am wondering how much
> that
> > helps, because the authorization feature would be incomplete like that
> and
> > thus of limited use.
> >
> > I would also assume that merging it properly and in full use after the
> 1.2
> > release would be a bit better - in general, we have often avoided last
> > minute additions of sensitive and complex features.
> >
> > Do you think it is more urgent to have this in Flink?
> >
> > Best,
> > Stephan
> >
> >
> > On Mon, Dec 12, 2016 at 2:49 PM, Vijay <vi...@yahoo.com.invalid>
> > wrote:
> >
> > > Max and Ufuk, I respect your concerns and fully understand the
> importance
> > > of the network layer stack in Flink code base. Will you be comfortable
> to
> > > merge the code if I remove the Netty layer changes and leave the rest
> of
> > > the code. We can address the Netty code changes post 1.2 release?
> > >
> > > Regards,
> > > Vijay
> > >
> > > Sent from my iPhone
> > >
> > > > On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <uc...@apache.org> wrote:
> > > >
> > > > On 12 December 2016 at 12:30:31, Maximilian Michels (mxm@apache.org)
> > > wrote:
> > > >>> It seems like we lack the resources for now to properly to take
> > > >> care
> > > >> of your pull request before the release. Unless someone from
> > > >> the
> > > >> community is really eager to help out here, I would be in favor
> > > >> of
> > > >> merging the pull request to the master after the release branch
> > > >> has
> > > >> been forked off. We should make sure it gets the attention it
> deserves
> > > >> then.
> > > >
> > > > Thanks Max! I fully agree with your reasoning. +1 to not include this
> > in
> > > 1.2 now, but look at it afterwards. I hope that OK with you Vijay.
> > > >
> > > > - Ufuk
> > > >
> > > >
> > >
> > >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Feng Wang <fe...@outlook.com>.
It will be pretty good if 1.2 branch could be forked off within this week, and our guys working on FLIP-6 hope FLIP-6 branch could be merged into master as soon as possible.
Best Regards,
Feng Wang
Alibaba
________________________________
From: Robert Metzger <rm...@apache.org>
Sent: Tuesday, December 13, 2016 4:58 AM
To: dev@flink.apache.org
Subject: Re: [DISCUSS] Schedule and Scope for Flink 1.2
Thank you all for figuring out a solution for the security pull request.
Lets try to get 1.2 feature freezed as fast as possible so that we can
"unblock" waiting features like FLIP-6 and the remaining security changes.
*What do you think about Friday evening (6pm Berlin, 9am US west coast) for
feature freezing Flink 1.2?* (only bugfixes are allowed in afterwards)
I'll then fork-off a "release-1.2" branch and update the version in
"master" to 1.3-SNAPSHOT.
Please object if you have a bigger change or any other reservations
regarding the feature freeze date!
This is my current view of things on the release:
- RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
- RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
- UNRESOLVED Add Flink 1.1 savepoint backwards compatability (FLINK-4797)
- RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos (FLINK-1984)
- UNDER DISCUSSION Secure Data Access (FLINK-3930)
- RESOLVED Queryable State (FLINK-3779)
- RESOLVED Metrics in Webinterface (FLINK-4389)
- RESOLVED Kafka 0.10 support (FLINK-4035)
- RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
- RESOLVED Table API: Scalar Functions (FLINK-3097)
Added by Stephan:
- NON-BLOCKING [Pending PR] Provide support for asynchronous operations
over streams (FLINK-4391)
- NON-BLOCKING [beginning of next week] Unify Savepoints and Checkpoints
(FLINK-4484)
Added by Fabian:
- ONGOING [Pending PR] Clean up the packages of the Table API (FLINK-4704)
Move Row to flink-core (
Added by Max:
- ONGOING [Pending PR] Change Akka configuration to allow accessing actors
from different URLs (FLINK-2821)
On Mon, Dec 12, 2016 at 5:43 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi Vijay!
>
> The workaround you suggest may be doable, but I am wondering how much that
> helps, because the authorization feature would be incomplete like that and
> thus of limited use.
>
> I would also assume that merging it properly and in full use after the 1.2
> release would be a bit better - in general, we have often avoided last
> minute additions of sensitive and complex features.
>
> Do you think it is more urgent to have this in Flink?
>
> Best,
> Stephan
>
>
> On Mon, Dec 12, 2016 at 2:49 PM, Vijay <vi...@yahoo.com.invalid>
> wrote:
>
> > Max and Ufuk, I respect your concerns and fully understand the importance
> > of the network layer stack in Flink code base. Will you be comfortable to
> > merge the code if I remove the Netty layer changes and leave the rest of
> > the code. We can address the Netty code changes post 1.2 release?
> >
> > Regards,
> > Vijay
> >
> > Sent from my iPhone
> >
> > > On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <uc...@apache.org> wrote:
> > >
> > > On 12 December 2016 at 12:30:31, Maximilian Michels (mxm@apache.org)
> > wrote:
> > >>> It seems like we lack the resources for now to properly to take
> > >> care
> > >> of your pull request before the release. Unless someone from
> > >> the
> > >> community is really eager to help out here, I would be in favor
> > >> of
> > >> merging the pull request to the master after the release branch
> > >> has
> > >> been forked off. We should make sure it gets the attention it deserves
> > >> then.
> > >
> > > Thanks Max! I fully agree with your reasoning. +1 to not include this
> in
> > 1.2 now, but look at it afterwards. I hope that OK with you Vijay.
> > >
> > > - Ufuk
> > >
> > >
> >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Robert Metzger <rm...@apache.org>.
Thank you all for figuring out a solution for the security pull request.
Lets try to get 1.2 feature freezed as fast as possible so that we can
"unblock" waiting features like FLIP-6 and the remaining security changes.
*What do you think about Friday evening (6pm Berlin, 9am US west coast) for
feature freezing Flink 1.2?* (only bugfixes are allowed in afterwards)
I'll then fork-off a "release-1.2" branch and update the version in
"master" to 1.3-SNAPSHOT.
Please object if you have a bigger change or any other reservations
regarding the feature freeze date!
This is my current view of things on the release:
- RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
- RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
- UNRESOLVED Add Flink 1.1 savepoint backwards compatability (FLINK-4797)
- RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos (FLINK-1984)
- UNDER DISCUSSION Secure Data Access (FLINK-3930)
- RESOLVED Queryable State (FLINK-3779)
- RESOLVED Metrics in Webinterface (FLINK-4389)
- RESOLVED Kafka 0.10 support (FLINK-4035)
- RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
- RESOLVED Table API: Scalar Functions (FLINK-3097)
Added by Stephan:
- NON-BLOCKING [Pending PR] Provide support for asynchronous operations
over streams (FLINK-4391)
- NON-BLOCKING [beginning of next week] Unify Savepoints and Checkpoints
(FLINK-4484)
Added by Fabian:
- ONGOING [Pending PR] Clean up the packages of the Table API (FLINK-4704)
Move Row to flink-core (
Added by Max:
- ONGOING [Pending PR] Change Akka configuration to allow accessing actors
from different URLs (FLINK-2821)
On Mon, Dec 12, 2016 at 5:43 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi Vijay!
>
> The workaround you suggest may be doable, but I am wondering how much that
> helps, because the authorization feature would be incomplete like that and
> thus of limited use.
>
> I would also assume that merging it properly and in full use after the 1.2
> release would be a bit better - in general, we have often avoided last
> minute additions of sensitive and complex features.
>
> Do you think it is more urgent to have this in Flink?
>
> Best,
> Stephan
>
>
> On Mon, Dec 12, 2016 at 2:49 PM, Vijay <vi...@yahoo.com.invalid>
> wrote:
>
> > Max and Ufuk, I respect your concerns and fully understand the importance
> > of the network layer stack in Flink code base. Will you be comfortable to
> > merge the code if I remove the Netty layer changes and leave the rest of
> > the code. We can address the Netty code changes post 1.2 release?
> >
> > Regards,
> > Vijay
> >
> > Sent from my iPhone
> >
> > > On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <uc...@apache.org> wrote:
> > >
> > > On 12 December 2016 at 12:30:31, Maximilian Michels (mxm@apache.org)
> > wrote:
> > >>> It seems like we lack the resources for now to properly to take
> > >> care
> > >> of your pull request before the release. Unless someone from
> > >> the
> > >> community is really eager to help out here, I would be in favor
> > >> of
> > >> merging the pull request to the master after the release branch
> > >> has
> > >> been forked off. We should make sure it gets the attention it deserves
> > >> then.
> > >
> > > Thanks Max! I fully agree with your reasoning. +1 to not include this
> in
> > 1.2 now, but look at it afterwards. I hope that OK with you Vijay.
> > >
> > > – Ufuk
> > >
> > >
> >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Stephan Ewen <se...@apache.org>.
Hi Vijay!
The workaround you suggest may be doable, but I am wondering how much that
helps, because the authorization feature would be incomplete like that and
thus of limited use.
I would also assume that merging it properly and in full use after the 1.2
release would be a bit better - in general, we have often avoided last
minute additions of sensitive and complex features.
Do you think it is more urgent to have this in Flink?
Best,
Stephan
On Mon, Dec 12, 2016 at 2:49 PM, Vijay <vi...@yahoo.com.invalid> wrote:
> Max and Ufuk, I respect your concerns and fully understand the importance
> of the network layer stack in Flink code base. Will you be comfortable to
> merge the code if I remove the Netty layer changes and leave the rest of
> the code. We can address the Netty code changes post 1.2 release?
>
> Regards,
> Vijay
>
> Sent from my iPhone
>
> > On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <uc...@apache.org> wrote:
> >
> > On 12 December 2016 at 12:30:31, Maximilian Michels (mxm@apache.org)
> wrote:
> >>> It seems like we lack the resources for now to properly to take
> >> care
> >> of your pull request before the release. Unless someone from
> >> the
> >> community is really eager to help out here, I would be in favor
> >> of
> >> merging the pull request to the master after the release branch
> >> has
> >> been forked off. We should make sure it gets the attention it deserves
> >> then.
> >
> > Thanks Max! I fully agree with your reasoning. +1 to not include this in
> 1.2 now, but look at it afterwards. I hope that OK with you Vijay.
> >
> > – Ufuk
> >
> >
>
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Vijay <vi...@yahoo.com.INVALID>.
Max and Ufuk, I respect your concerns and fully understand the importance of the network layer stack in Flink code base. Will you be comfortable to merge the code if I remove the Netty layer changes and leave the rest of the code. We can address the Netty code changes post 1.2 release?
Regards,
Vijay
Sent from my iPhone
> On Dec 12, 2016, at 3:38 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
> On 12 December 2016 at 12:30:31, Maximilian Michels (mxm@apache.org) wrote:
>>> It seems like we lack the resources for now to properly to take
>> care
>> of your pull request before the release. Unless someone from
>> the
>> community is really eager to help out here, I would be in favor
>> of
>> merging the pull request to the master after the release branch
>> has
>> been forked off. We should make sure it gets the attention it deserves
>> then.
>
> Thanks Max! I fully agree with your reasoning. +1 to not include this in 1.2 now, but look at it afterwards. I hope that OK with you Vijay.
>
> – Ufuk
>
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Ufuk Celebi <uc...@apache.org>.
On 12 December 2016 at 12:30:31, Maximilian Michels (mxm@apache.org) wrote:
> > It seems like we lack the resources for now to properly to take
> care
> of your pull request before the release. Unless someone from
> the
> community is really eager to help out here, I would be in favor
> of
> merging the pull request to the master after the release branch
> has
> been forked off. We should make sure it gets the attention it deserves
> then.
Thanks Max! I fully agree with your reasoning. +1 to not include this in 1.2 now, but look at it afterwards. I hope that OK with you Vijay.
– Ufuk
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Maximilian Michels <mx...@apache.org>.
Hi Vijay,
Thank you for updating the pull request. I appreciate your work in the
security realm of Flink and value your contributions so far. It would
be great to merge the authorization pull request for the release.
However, I don't feel comfortable about the network stack (i.e. Netty)
related changes because they touch a very critical part of the Flink
engine. While the code has already been iterated over in the lifetime
of the pull request, I still think somebody extremely familiar with
Netty and the Flink network layer should check out the changes
(thinking about Ufuk here). Apart from that, I would like to further
test the changes to harden the code.
It seems like we lack the resources for now to properly to take care
of your pull request before the release. Unless someone from the
community is really eager to help out here, I would be in favor of
merging the pull request to the master after the release branch has
been forked off. We should make sure it gets the attention it deserves
then.
Thanks,
Max
On Mon, Dec 12, 2016 at 8:09 AM, Vijay <vi...@yahoo.com> wrote:
> On FLINK-3930, almost all of the feedback has been addressed. The only pending review is Netty cookie authorization part which I have moved the cookie validation from message level to a separate channel handler. I have just rebased the code with master for final review.
>
> Regards,
> Vijay
>
> Sent from my iPhone
>
>> On Dec 8, 2016, at 1:17 AM, Robert Metzger <rm...@apache.org> wrote:
>>
>> Thank you for your responses Max and Vijay.
>> So I understand that Mesos is basically ready for the 1.2 release.
>>
>> Regarding the security changes: Having Hadoop, Kafka and Zookeeper
>> integration is a big improvement and a much requested feature. I'm super
>> excited to have that in :)
>> Are all the other security changes useless without authorization, or could
>> we consider releasing 1.2 without it? (Another way to think about it: How
>> close is the PR to being merged. If its just a final review & we are done,
>> I would actually try to get it in. But if there's a lot of uncertainty, I
>> would prefer to move it to the next release)
>>
>> I agree regarding FLINK-2821, that's important for many deployments.
>>
>>
>> The updated list:
>> - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
>> - RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
>> - RESOLVED [Split for 1.3] Add Flink 1.1 savepoint backwards compatability
>> (FLINK-4797)
>> - RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos (FLINK-1984)
>> - UNRESOLVED Secure Data Access (FLINK-3930)
>> - RESOLVED Queryable State (FLINK-3779)
>> - RESOLVED Metrics in Webinterface (FLINK-4389)
>> - RESOLVED Kafka 0.10 support (FLINK-4035)
>> - RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
>> - RESOLVED Table API: Scalar Functions (FLINK-3097)
>> Added by Stephan:
>> - NON-BLOCKING [Pending PR] Provide support for asynchronous operations
>> over streams (FLINK-4391)
>> - ONGOING [beginning of next week] Unify Savepoints and Checkpoints
>> (FLINK-4484)
>> Added by Fabian:
>> - ONGOING [Pending PR] Clean up the packages of the Table API (FLINK-4704)
>> - UNRESOLVED Move Row to flink-core (FLINK-5186)
>> Added by Max:
>> - ONGOING [Pending PR] Change Akka configuration to allow accessing actors
>> from different URLs (FLINK-2821)
>>
>>
>> On Wed, Dec 7, 2016 at 12:40 PM, Maximilian Michels <mx...@apache.org> wrote:
>>
>>>> - UNRESOLVED Integrate Flink with Apache Mesos (FLINK-1984)
>>>
>>> The initial integration is already completed with the last issues
>>> being resolved in the Mesos component:
>>> https://issues.apache.org/jira/browse/FLINK/component/12331068/ The
>>> implementation will be further refined after the next release and with
>>> the merge of FLIP-6. We're missing documentation on how to deploy a
>>> Flink Mesos cluster.
>>>
>>>> - UNRESOLVED Secure Data Access (FLINK-3930)
>>>
>>> We have support for Kerberos authentication with Haddop, Kafka,
>>> Zookeper, and all services supporting JAAS. Additionally, we
>>> implemented SSL encryption for all communications paths, i.e. web
>>> interface, Akka, Netty, BlobServer. We still lack support for
>>> authorization: Vijay's PR is blocked because we haven't found time to
>>> properly review the sensitive network changes.
>>>
>>> I'd like to add the Akka changes for containered environments which
>>> should be ready by the end of the week:
>>> https://issues.apache.org/jira/browse/FLINK-2821
>>>
>>> -Max
>>>
>>> On Tue, Dec 6, 2016 at 8:57 PM, Vijay <vi...@yahoo.com.invalid>
>>> wrote:
>>>>>> Secure Data Access (FLINK-3930)
>>>>
>>>> The PR for the work is still under review and I hope this could be
>>> included in the release.
>>>>
>>>> Regards,
>>>> Vijay
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On Dec 6, 2016, at 11:51 AM, Robert Metzger <rm...@apache.org>
>>> wrote:
>>>>>
>>>>> UNRESOLVED Secure Data Access (FLINK-3930)
>>>>
>>>
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Vijay <vi...@yahoo.com.INVALID>.
On FLINK-3930, almost all of the feedback has been addressed. The only pending review is Netty cookie authorization part which I have moved the cookie validation from message level to a separate channel handler. I have just rebased the code with master for final review.
Regards,
Vijay
Sent from my iPhone
> On Dec 8, 2016, at 1:17 AM, Robert Metzger <rm...@apache.org> wrote:
>
> Thank you for your responses Max and Vijay.
> So I understand that Mesos is basically ready for the 1.2 release.
>
> Regarding the security changes: Having Hadoop, Kafka and Zookeeper
> integration is a big improvement and a much requested feature. I'm super
> excited to have that in :)
> Are all the other security changes useless without authorization, or could
> we consider releasing 1.2 without it? (Another way to think about it: How
> close is the PR to being merged. If its just a final review & we are done,
> I would actually try to get it in. But if there's a lot of uncertainty, I
> would prefer to move it to the next release)
>
> I agree regarding FLINK-2821, that's important for many deployments.
>
>
> The updated list:
> - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
> - RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
> - RESOLVED [Split for 1.3] Add Flink 1.1 savepoint backwards compatability
> (FLINK-4797)
> - RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos (FLINK-1984)
> - UNRESOLVED Secure Data Access (FLINK-3930)
> - RESOLVED Queryable State (FLINK-3779)
> - RESOLVED Metrics in Webinterface (FLINK-4389)
> - RESOLVED Kafka 0.10 support (FLINK-4035)
> - RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
> - RESOLVED Table API: Scalar Functions (FLINK-3097)
> Added by Stephan:
> - NON-BLOCKING [Pending PR] Provide support for asynchronous operations
> over streams (FLINK-4391)
> - ONGOING [beginning of next week] Unify Savepoints and Checkpoints
> (FLINK-4484)
> Added by Fabian:
> - ONGOING [Pending PR] Clean up the packages of the Table API (FLINK-4704)
> - UNRESOLVED Move Row to flink-core (FLINK-5186)
> Added by Max:
> - ONGOING [Pending PR] Change Akka configuration to allow accessing actors
> from different URLs (FLINK-2821)
>
>
> On Wed, Dec 7, 2016 at 12:40 PM, Maximilian Michels <mx...@apache.org> wrote:
>
>>> - UNRESOLVED Integrate Flink with Apache Mesos (FLINK-1984)
>>
>> The initial integration is already completed with the last issues
>> being resolved in the Mesos component:
>> https://issues.apache.org/jira/browse/FLINK/component/12331068/ The
>> implementation will be further refined after the next release and with
>> the merge of FLIP-6. We're missing documentation on how to deploy a
>> Flink Mesos cluster.
>>
>>> - UNRESOLVED Secure Data Access (FLINK-3930)
>>
>> We have support for Kerberos authentication with Haddop, Kafka,
>> Zookeper, and all services supporting JAAS. Additionally, we
>> implemented SSL encryption for all communications paths, i.e. web
>> interface, Akka, Netty, BlobServer. We still lack support for
>> authorization: Vijay's PR is blocked because we haven't found time to
>> properly review the sensitive network changes.
>>
>> I'd like to add the Akka changes for containered environments which
>> should be ready by the end of the week:
>> https://issues.apache.org/jira/browse/FLINK-2821
>>
>> -Max
>>
>> On Tue, Dec 6, 2016 at 8:57 PM, Vijay <vi...@yahoo.com.invalid>
>> wrote:
>>>>> Secure Data Access (FLINK-3930)
>>>
>>> The PR for the work is still under review and I hope this could be
>> included in the release.
>>>
>>> Regards,
>>> Vijay
>>>
>>> Sent from my iPhone
>>>
>>>> On Dec 6, 2016, at 11:51 AM, Robert Metzger <rm...@apache.org>
>> wrote:
>>>>
>>>> UNRESOLVED Secure Data Access (FLINK-3930)
>>>
>>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Robert Metzger <rm...@apache.org>.
Thank you for your responses Max and Vijay.
So I understand that Mesos is basically ready for the 1.2 release.
Regarding the security changes: Having Hadoop, Kafka and Zookeeper
integration is a big improvement and a much requested feature. I'm super
excited to have that in :)
Are all the other security changes useless without authorization, or could
we consider releasing 1.2 without it? (Another way to think about it: How
close is the PR to being merged. If its just a final review & we are done,
I would actually try to get it in. But if there's a lot of uncertainty, I
would prefer to move it to the next release)
I agree regarding FLINK-2821, that's important for many deployments.
The updated list:
- RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
- RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
- RESOLVED [Split for 1.3] Add Flink 1.1 savepoint backwards compatability
(FLINK-4797)
- RESOLVED [Split for 1.3] Integrate Flink with Apache Mesos (FLINK-1984)
- UNRESOLVED Secure Data Access (FLINK-3930)
- RESOLVED Queryable State (FLINK-3779)
- RESOLVED Metrics in Webinterface (FLINK-4389)
- RESOLVED Kafka 0.10 support (FLINK-4035)
- RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
- RESOLVED Table API: Scalar Functions (FLINK-3097)
Added by Stephan:
- NON-BLOCKING [Pending PR] Provide support for asynchronous operations
over streams (FLINK-4391)
- ONGOING [beginning of next week] Unify Savepoints and Checkpoints
(FLINK-4484)
Added by Fabian:
- ONGOING [Pending PR] Clean up the packages of the Table API (FLINK-4704)
- UNRESOLVED Move Row to flink-core (FLINK-5186)
Added by Max:
- ONGOING [Pending PR] Change Akka configuration to allow accessing actors
from different URLs (FLINK-2821)
On Wed, Dec 7, 2016 at 12:40 PM, Maximilian Michels <mx...@apache.org> wrote:
> > - UNRESOLVED Integrate Flink with Apache Mesos (FLINK-1984)
>
> The initial integration is already completed with the last issues
> being resolved in the Mesos component:
> https://issues.apache.org/jira/browse/FLINK/component/12331068/ The
> implementation will be further refined after the next release and with
> the merge of FLIP-6. We're missing documentation on how to deploy a
> Flink Mesos cluster.
>
> > - UNRESOLVED Secure Data Access (FLINK-3930)
>
> We have support for Kerberos authentication with Haddop, Kafka,
> Zookeper, and all services supporting JAAS. Additionally, we
> implemented SSL encryption for all communications paths, i.e. web
> interface, Akka, Netty, BlobServer. We still lack support for
> authorization: Vijay's PR is blocked because we haven't found time to
> properly review the sensitive network changes.
>
> I'd like to add the Akka changes for containered environments which
> should be ready by the end of the week:
> https://issues.apache.org/jira/browse/FLINK-2821
>
> -Max
>
> On Tue, Dec 6, 2016 at 8:57 PM, Vijay <vi...@yahoo.com.invalid>
> wrote:
> >>>Secure Data Access (FLINK-3930)
> >
> > The PR for the work is still under review and I hope this could be
> included in the release.
> >
> > Regards,
> > Vijay
> >
> > Sent from my iPhone
> >
> >> On Dec 6, 2016, at 11:51 AM, Robert Metzger <rm...@apache.org>
> wrote:
> >>
> >> UNRESOLVED Secure Data Access (FLINK-3930)
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Maximilian Michels <mx...@apache.org>.
> - UNRESOLVED Integrate Flink with Apache Mesos (FLINK-1984)
The initial integration is already completed with the last issues
being resolved in the Mesos component:
https://issues.apache.org/jira/browse/FLINK/component/12331068/ The
implementation will be further refined after the next release and with
the merge of FLIP-6. We're missing documentation on how to deploy a
Flink Mesos cluster.
> - UNRESOLVED Secure Data Access (FLINK-3930)
We have support for Kerberos authentication with Haddop, Kafka,
Zookeper, and all services supporting JAAS. Additionally, we
implemented SSL encryption for all communications paths, i.e. web
interface, Akka, Netty, BlobServer. We still lack support for
authorization: Vijay's PR is blocked because we haven't found time to
properly review the sensitive network changes.
I'd like to add the Akka changes for containered environments which
should be ready by the end of the week:
https://issues.apache.org/jira/browse/FLINK-2821
-Max
On Tue, Dec 6, 2016 at 8:57 PM, Vijay <vi...@yahoo.com.invalid> wrote:
>>>Secure Data Access (FLINK-3930)
>
> The PR for the work is still under review and I hope this could be included in the release.
>
> Regards,
> Vijay
>
> Sent from my iPhone
>
>> On Dec 6, 2016, at 11:51 AM, Robert Metzger <rm...@apache.org> wrote:
>>
>> UNRESOLVED Secure Data Access (FLINK-3930)
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Vijay <vi...@yahoo.com.INVALID>.
>>Secure Data Access (FLINK-3930)
The PR for the work is still under review and I hope this could be included in the release.
Regards,
Vijay
Sent from my iPhone
> On Dec 6, 2016, at 11:51 AM, Robert Metzger <rm...@apache.org> wrote:
>
> UNRESOLVED Secure Data Access (FLINK-3930)
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Robert Metzger <rm...@apache.org>.
Thank you all for your responses so far. The situation is better than I
expected :)
Let me summarize, and please object if you disagree:
- RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
- RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
- RESOLVED [Split for 1.3] Add Flink 1.1 savepoint backwards compatability
(FLINK-4797)
- UNRESOLVED Integrate Flink with Apache Mesos (FLINK-1984)
- UNRESOLVED Secure Data Access (FLINK-3930)
- RESOLVED Queryable State (FLINK-3779)
- RESOLVED Metrics in Webinterface (FLINK-4389)
- RESOLVED Kafka 0.10 support (FLINK-4035)
- RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
- RESOLVED Table API: Scalar Functions (FLINK-3097)
Added by Stephan:
- NON-BLOCKING [Pending PR] Provide support for asynchronous operations
over streams (FLINK-4391)
- ONGOING [beginning of next week] Unify Savepoints and Checkpoints
(FLINK-4484)
Added by Fabian:
- UNRESOLVED Clean up the packages of the Table API (FLINK-4704)
- UNRESOLVED Move Row to flink-core (FLINK-5186)
On Tue, Dec 6, 2016 at 5:55 PM, Kostas Kloudas <k....@data-artisans.com>
wrote:
> Hi Robert,
>
> I agree with Stefan on the estimation about 4.
> As for the testing, this can already start, as some basic operators have
> already been ported.
>
> Kostas
>
> > On Dec 6, 2016, at 5:50 PM, Stefan Richter <s....@data-artisans.com>
> wrote:
> >
> > e already restored a job with general window operator and kafka source
> in an end-to-en
>
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Kostas Kloudas <k....@data-artisans.com>.
Hi Robert,
I agree with Stefan on the estimation about 4.
As for the testing, this can already start, as some basic operators have already been ported.
Kostas
> On Dec 6, 2016, at 5:50 PM, Stefan Richter <s....@data-artisans.com> wrote:
>
> e already restored a job with general window operator and kafka source in an end-to-en
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Stefan Richter <s....@data-artisans.com>.
Hi Robert,
The state of (FLINK-4797) Add Flink 1.1 savepoint backwards compatibility is as follows:
We have identified 5 major steps:
1) Convert savepoint format, task state, state handles, etc.
2) Convert binary format of keyed state backends (Rocks, Heap, Fs)
3) Introduce a first step for for being able to upgrade type serializers in the future, in particular versioning and making them part of a check/savepoint
4) Ensure current operators are compatible with their old format
5) Intensive testing
1) and 2) are finished and reviewed, they will be merged today or tomorrow.
3) Is almost done, then must be reviewed. My estimate is end of this week.
4) Kostas is working on this, but my feeling is that we are pretty close to completion. We already restored a job with general window operator and kafka source in an end-to-end test. My estimate for this would be end of this week or beginning of next week.
5) While we already tested a lo during development, I think this needs some thorough end-to-end testing after steps 1-4 are completed.
Best,
Stefan
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Fabian Hueske <fh...@gmail.com>.
Thanks Robert for reactivating the discussion.
There are a few Table API related refactorings that I would like to have in
1.2 because they are API breaking:
- FLINK-4704 Clean up the packages of the Table API and move it to the
org.apache.flink.table package space. This is an API breaking change that
we should do rather earlier than later.
- FLINK-5186 Move Row to flink-core. Row is used by other modules as a
flexible and wider Tuple (e.g., flink-jdbc, RowCsvInputFormat). Several
users requested to move it to flink-core, also to avoid Calcite which comes
with flink-table.
Both changes are mostly refactoring and should be solvable with rather
little effort.
@Flavio: I completely agree that we need a fix for FLINK-5122 in 1.2.0.
Since this is not a new feature but rather a bugfix it won't be affected by
the feature freeze, IMO.
It makes sense to fix it for RC1 though.
Best,
Fabian
2016-12-05 18:35 GMT+01:00 Flavio Pompermaier <po...@okkam.it>:
> Hi Robert,
> our pull request was closed in favor of
> https://github.com/apache/flink/pull/2861.
> However that PR still need some review I think (I also added some comment
> in the JIRA issue[1]).
> it will be great to have that fix in the next release :)
>
> [1] https://issues.apache.org/jira/browse/FLINK-5122
>
> Best,
> Flavio
>
> On Mon, Dec 5, 2016 at 6:28 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
> > Hi,
> >
> > I would like to reactivate this old email thread. The initial plan was to
> > release 1.2 before Christmas, have RC1 by end of Nov / December and a
> > feature freeze mid-November.
> > Today (December 5), we did not yet do the feature freeze. I would like to
> > get an overview again of the remaining blockers and what we want to
> include
> > into the release.
> >
> > From the issues Fabian posted in the beginning, they have the following
> > status:
> > - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
> > - RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
> > - UNRESOLVED Add Flink 1.1 savepoint backwards compatability (FLINK-4797)
> > - UNRESOLVED Integrate Flink with Apache Mesos (FLINK-1984)
> > - UNRESOLVED Secure Data Access (FLINK-3930)
> > - RESOLVED Queryable State (FLINK-3779)
> > - RESOLVED Metrics in Webinterface (FLINK-4389)
> > - RESOLVED Kafka 0.10 support (FLINK-4035)
> > - RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
> > - RESOLVED Table API: Scalar Functions (FLINK-3097)
> > Added by Stephan:
> > - UNRESOLVED Provide support for asynchronous operations over streams
> > (FLINK-4391)
> > - UNRESOLVED Unify Savepoints and Checkpoints (FLINK-4484)
> >
> > I've also reflected the status in the Wiki:
> > https://cwiki.apache.org/confluence/display/FLINK/1.2+Release
> >
> > @Flavio: I assume PR #2790 is resolved and not a release blocker.
> >
> > Given that we are already behind the schedule, I'm wondering what we can
> do
> > regarding the remaining UNRESOLVED issues.
> > I would also be willing to release some of them as undocumented,
> > work-in-progress features. We have so many good things like dynamic
> scaling
> > or Mesos in the release that we should get it out rather soon.
> >
> > It would be nice if those committers / contributors working on the
> > UNRESOLVED issues could give a more detailed status update what they
> think
> > release-wise.
> > Also, is there anything else we need to consider before forking off the
> > release-1.2 branch from master?
> >
> > Once we've collected the detailed status updates from the last unresolved
> > features, we can discuss a new date for the feature freeze.
> > I'll then also create a RC0 testing release candidate without a VOTE, to
> > have a starting point for the first big release testing. (And yes, I'm
> > volunteering to be the release manager for the 1.2 release ;) )
> >
> >
> > Regards,
> > Robert
> >
> >
> >
> >
> > On Mon, Nov 21, 2016 at 3:03 PM, Flavio Pompermaier <
> pompermaier@okkam.it>
> > wrote:
> >
> > > Hi to all,
> > > we've completed the PR #2790 <https://github.com/apache/
> flink/pull/2790>
> > > to
> > > implement FLINK-4491 <https://issues.apache.org/jira/browse/FLINK-4491
> >:
> > > we've added the comment that @zentol asked for a week ago and we also
> > fixed
> > > a potential annoying problem we faced during indexing (if a malformed
> > > document generates an error during indexing, the close() method throws
> an
> > > exception forcing the entire job to fail, killing other threads that
> are
> > > still indexing). I hope this PR could hep other people working with
> > > Elasticsearch, right from the next release ;)
> > >
> > > On Mon, Nov 14, 2016 at 2:47 PM, Flavio Pompermaier <
> > pompermaier@okkam.it>
> > > wrote:
> > >
> > > > Hi to all,
> > > > for the 1.2 release do you think it will be possible to include also
> a
> > PR
> > > > to implement FLINK-4491 <https://issues.apache.org/
> > > jira/browse/FLINK-4491>
> > > > ?
> > > > The PR is waiting for a review at https://github.com/apache/
> > > > flink/pull/2790.
> > > >
> > > > Best,
> > > > Flavio
> > > >
> > > > On Fri, Oct 28, 2016 at 5:59 PM, Fabian Hueske <fh...@gmail.com>
> > > wrote:
> > > >
> > > >> Hi Jark,
> > > >>
> > > >> FLINK-4469: I agree, UDTFs would be a great feature. I don't think
> we
> > > >> should make it a blocker but I will help to get it the master before
> > the
> > > >> feature freeze.
> > > >> FLINK-4704: Regarding the renaming of the namespace, we have two
> > > positive
> > > >> votes in JIRA, but we should also start a discussion on the dev and
> > user
> > > >> list to collect some more input. The actual change should be simple.
> > > >>
> > > >> @Everybody: I created a wiki page for the 1.2 release [1] and added
> > all
> > > >> features discussed in this thread so far.
> > > >> Please have a look at the page and update it if you know the JIRAs
> > > and/or
> > > >> state of the features.
> > > >>
> > > >> Cheers, Fabian
> > > >>
> > > >> [1] https://cwiki.apache.org/confluence/display/FLINK/1.2+Release
> > > >>
> > > >> 2016-10-28 15:28 GMT+02:00 Kostas Kloudas <
> > k.kloudas@data-artisans.com
> > > >:
> > > >>
> > > >> > Hello,
> > > >> >
> > > >> > As Aljoshca said, the TriggerDSL depends on another PR, before it
> > can
> > > be
> > > >> > merged.
> > > >> > In addition, this will change also the whole structure of how we
> > test
> > > >> > window/trigger
> > > >> > related functionality, so some refactoring will be required.
> > > >> >
> > > >> > Kostas
> > > >> >
> > > >> > > On Oct 28, 2016, at 3:06 PM, Aljoscha Krettek <
> > aljoscha@apache.org>
> > > >> > wrote:
> > > >> > >
> > > >> > > @Till: For the TriggerDSL we have to merge my open PR [1] for
> > > >> refactoring
> > > >> > > the WindowOperator/Trigger tests and then refactor the tests of
> > the
> > > >> > > TriggerDSL to use the new test facilities introduced there. Plus
> > > then
> > > >> > maybe
> > > >> > > some fixups.
> > > >> > >
> > > >> > > On Fri, 28 Oct 2016 at 11:17 Jark Wu <
> wuchong.wc@alibaba-inc.com>
> > > >> wrote:
> > > >> > >
> > > >> > >> +1 to the schedule
> > > >> > >>
> > > >> > >> I hope to add
> > > >> > >>
> > > >> > >> 1) Table API: User defined table function (FLINK-4469 <
> > > >> > >> https://issues.apache.org/jira/browse/FLINK-4469>)
> > > >> > >> 2) Table API: Move Table API to org.apache.flink.table
> > (FLINK-4704
> > > <
> > > >> > >> https://issues.apache.org/jira/browse/FLINK-4704>)
> > > >> > >>
> > > >> > >> The 1) has been discussed for a long time and a pull request
> has
> > > >> been on
> > > >> > >> github. The 2) would be a large change, but I hope it could be
> > > >> included
> > > >> > in
> > > >> > >> 1.2 release, otherwise it will be more difficult to fix it.
> > > >> > >>
> > > >> > >> - Jark Wu
> > > >> > >>
> > > >> > >>> 在 2016年10月26日，下午8:09，Till Rohrmann <tr...@apache.org> 写道：
> > > >> > >>>
> > > >> > >>> +1 for the features to include.
> > > >> > >>>
> > > >> > >>> What is the state of the Trigger DSL? How much is left to be
> > done
> > > >> > before
> > > >> > >>> merging?
> > > >> > >>>
> > > >> > >>> Cheers,
> > > >> > >>> Till
> > > >> > >>>
> > > >> > >>> On Tue, Oct 25, 2016 at 4:32 PM, Aljoscha Krettek <
> > > >> aljoscha@apache.org
> > > >> > >
> > > >> > >>> wrote:
> > > >> > >>>
> > > >> > >>>> +1 the schedule proposed so far.
> > > >> > >>>>
> > > >> > >>>> Do we also want to get in the "Trigger DSL" that we've had
> > > brewing
> > > >> > for a
> > > >> > >>>> while now?
> > > >> > >>>>
> > > >> > >>>> On Mon, 17 Oct 2016 at 16:17 Stephan Ewen <se...@apache.org>
> > > >> wrote:
> > > >> > >>>>
> > > >> > >>>>> I think this sounds very reasonable, +1 to the schedule.
> > > >> > >>>>>
> > > >> > >>>>> I would definitely add
> > > >> > >>>>> - FLIP-10 (unify checkpoints and savepoints)
> > > >> > >>>>> - FLIP-7 (metrics in web UI)
> > > >> > >>>>> - FLIP-12 (async request operators)
> > > >> > >>>>>
> > > >> > >>>>> Those should be all safe bets, as they are basically done.
> > > >> > >>>>>
> > > >> > >>>>> Let's see what else is in shape until the feature freeze
> day.
> > > >> > >>>>>
> > > >> > >>>>>
> > > >> > >>>>>
> > > >> > >>>>> On Mon, Oct 17, 2016 at 2:18 PM, Fabian Hueske <
> > > fhueske@gmail.com
> > > >> >
> > > >> > >>>> wrote:
> > > >> > >>>>>
> > > >> > >>>>>> Hi everybody,
> > > >> > >>>>>>
> > > >> > >>>>>> Flink 1.1.0 was released in August and I think it is time
> to
> > > kick
> > > >> > off
> > > >> > >> a
> > > >> > >>>>>> discussion about the schedule and scope of Flink 1.2.0.
> > > >> > >>>>>>
> > > >> > >>>>>> == Scope
> > > >> > >>>>>>
> > > >> > >>>>>> We started to collect features for Flink 1.2.0 in the Flink
> > > >> Release
> > > >> > >>>> wiki
> > > >> > >>>>>> page [1].
> > > >> > >>>>>> I copy the feature list for convenience:
> > > >> > >>>>>>
> > > >> > >>>>>> - Dynamic Scaling / Key Groups (FLINK-3755
> > > >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3755>)
> (Design
> > > Doc
> > > >> > >>>>>> <https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-
> > > >> > >>>>>> LuBCyPUWyFd9l3T9WyssQ63w/edit#heading=h.2suhu1fjxmp4>
> > > >> > >>>>>> )
> > > >> > >>>>>> - Rescalable Non-Partitioned State (FLIP-8
> > > >> > >>>>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-
> > > >> > >>>>>> 8%3A+Rescalable+Non-Partitioned+State>
> > > >> > >>>>>> )
> > > >> > >>>>>> - Backwards compatible savepoints (FLINK-4797
> > > >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-4797>)
> > > >> > >>>>>> -
> > > >> > >>>>>>
> > > >> > >>>>>> Integrate Flink with Apache Mesos (FLINK-1984
> > > >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-1984>)
> (Design
> > > Doc
> > > >> > >>>>>> <https://docs.google.com/document/d/
> > 1WItafBmGbjlaBbP8Of5PAFO
> > > >> H9GUJQ
> > > >> > >>>>>> xf5S4hjEuPchuU/edit#heading=h.33uxp8abvuf0>
> > > >> > >>>>>> )
> > > >> > >>>>>> - Secure Data Access (FLINK-3930
> > > >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3930>)
> (Design
> > > Doc
> > > >> > >>>>>> <https://docs.google.com/document/d/1-
> > > >> > >>>> GQB6uVOyoaXGwtqwqLV8BHDxWiMO2W
> > > >> > >>>>>> nVzBoJ8oPaAs/edit#heading=h.dr22iualfm0h>
> > > >> > >>>>>> )
> > > >> > >>>>>> - Queryable State (FLINK-3779
> > > >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3779>)
> > > >> > >>>>>> - Metrics in Webinterface (FLIP-7
> > > >> > >>>>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-
> > > >> > >>>>>> 7%3A+Expose+metrics+to+WebInterface>
> > > >> > >>>>>> )
> > > >> > >>>>>> - Kafka 0.10 support (FLINK-4035
> > > >> > >>>>>> <http://issues.apache.org/jira/browse/FLINK-4035>)
> > > >> > >>>>>> - Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
> > > >> > >>>>>> - Table API: Scalar Functions (FLINK-3097)
> > > >> > >>>>>>
> > > >> > >>>>>>
> > > >> > >>>>>> Is a feature missing or something too ambitious?
> > > >> > >>>>>>
> > > >> > >>>>>> == Schedule
> > > >> > >>>>>>
> > > >> > >>>>>> In my opinion we should try to release 1.2.0 before
> > Christmas.
> > > >> That
> > > >> > >>>>> means,
> > > >> > >>>>>> we should have a RC1 by end November / early December and a
> > > >> feature
> > > >> > >>>>> freeze
> > > >> > >>>>>> two weeks earlier in mid of November. Does that sound
> > > >> appropriate or
> > > >> > >>>>> should
> > > >> > >>>>>> we aim for an earlier or later release?
> > > >> > >>>>>>
> > > >> > >>>>>> What do others think about the scope and schedule?
> > > >> > >>>>>> Please discuss.
> > > >> > >>>>>>
> > > >> > >>>>>> Best,
> > > >> > >>>>>> Fabian
> > > >> > >>>>>>
> > > >> > >>>>>> [1]
> > > >> > >>>>>> https://cwiki.apache.org/confluence/display/FLINK/
> > > >> > >>>>>> Flink+Release+and+Feature+Plan
> > > >> > >>>>>>
> > > >> > >>>>>
> > > >> > >>>>
> > > >> > >>
> > > >> > >>
> > > >> >
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Flavio Pompermaier <po...@okkam.it>.
Hi Robert,
our pull request was closed in favor of
https://github.com/apache/flink/pull/2861.
However that PR still need some review I think (I also added some comment
in the JIRA issue[1]).
it will be great to have that fix in the next release :)
[1] https://issues.apache.org/jira/browse/FLINK-5122
Best,
Flavio
On Mon, Dec 5, 2016 at 6:28 PM, Robert Metzger <rm...@apache.org> wrote:
> Hi,
>
> I would like to reactivate this old email thread. The initial plan was to
> release 1.2 before Christmas, have RC1 by end of Nov / December and a
> feature freeze mid-November.
> Today (December 5), we did not yet do the feature freeze. I would like to
> get an overview again of the remaining blockers and what we want to include
> into the release.
>
> From the issues Fabian posted in the beginning, they have the following
> status:
> - RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
> - RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
> - UNRESOLVED Add Flink 1.1 savepoint backwards compatability (FLINK-4797)
> - UNRESOLVED Integrate Flink with Apache Mesos (FLINK-1984)
> - UNRESOLVED Secure Data Access (FLINK-3930)
> - RESOLVED Queryable State (FLINK-3779)
> - RESOLVED Metrics in Webinterface (FLINK-4389)
> - RESOLVED Kafka 0.10 support (FLINK-4035)
> - RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
> - RESOLVED Table API: Scalar Functions (FLINK-3097)
> Added by Stephan:
> - UNRESOLVED Provide support for asynchronous operations over streams
> (FLINK-4391)
> - UNRESOLVED Unify Savepoints and Checkpoints (FLINK-4484)
>
> I've also reflected the status in the Wiki:
> https://cwiki.apache.org/confluence/display/FLINK/1.2+Release
>
> @Flavio: I assume PR #2790 is resolved and not a release blocker.
>
> Given that we are already behind the schedule, I'm wondering what we can do
> regarding the remaining UNRESOLVED issues.
> I would also be willing to release some of them as undocumented,
> work-in-progress features. We have so many good things like dynamic scaling
> or Mesos in the release that we should get it out rather soon.
>
> It would be nice if those committers / contributors working on the
> UNRESOLVED issues could give a more detailed status update what they think
> release-wise.
> Also, is there anything else we need to consider before forking off the
> release-1.2 branch from master?
>
> Once we've collected the detailed status updates from the last unresolved
> features, we can discuss a new date for the feature freeze.
> I'll then also create a RC0 testing release candidate without a VOTE, to
> have a starting point for the first big release testing. (And yes, I'm
> volunteering to be the release manager for the 1.2 release ;) )
>
>
> Regards,
> Robert
>
>
>
>
> On Mon, Nov 21, 2016 at 3:03 PM, Flavio Pompermaier <po...@okkam.it>
> wrote:
>
> > Hi to all,
> > we've completed the PR #2790 <https://github.com/apache/flink/pull/2790>
> > to
> > implement FLINK-4491 <https://issues.apache.org/jira/browse/FLINK-4491>:
> > we've added the comment that @zentol asked for a week ago and we also
> fixed
> > a potential annoying problem we faced during indexing (if a malformed
> > document generates an error during indexing, the close() method throws an
> > exception forcing the entire job to fail, killing other threads that are
> > still indexing). I hope this PR could hep other people working with
> > Elasticsearch, right from the next release ;)
> >
> > On Mon, Nov 14, 2016 at 2:47 PM, Flavio Pompermaier <
> pompermaier@okkam.it>
> > wrote:
> >
> > > Hi to all,
> > > for the 1.2 release do you think it will be possible to include also a
> PR
> > > to implement FLINK-4491 <https://issues.apache.org/
> > jira/browse/FLINK-4491>
> > > ?
> > > The PR is waiting for a review at https://github.com/apache/
> > > flink/pull/2790.
> > >
> > > Best,
> > > Flavio
> > >
> > > On Fri, Oct 28, 2016 at 5:59 PM, Fabian Hueske <fh...@gmail.com>
> > wrote:
> > >
> > >> Hi Jark,
> > >>
> > >> FLINK-4469: I agree, UDTFs would be a great feature. I don't think we
> > >> should make it a blocker but I will help to get it the master before
> the
> > >> feature freeze.
> > >> FLINK-4704: Regarding the renaming of the namespace, we have two
> > positive
> > >> votes in JIRA, but we should also start a discussion on the dev and
> user
> > >> list to collect some more input. The actual change should be simple.
> > >>
> > >> @Everybody: I created a wiki page for the 1.2 release [1] and added
> all
> > >> features discussed in this thread so far.
> > >> Please have a look at the page and update it if you know the JIRAs
> > and/or
> > >> state of the features.
> > >>
> > >> Cheers, Fabian
> > >>
> > >> [1] https://cwiki.apache.org/confluence/display/FLINK/1.2+Release
> > >>
> > >> 2016-10-28 15:28 GMT+02:00 Kostas Kloudas <
> k.kloudas@data-artisans.com
> > >:
> > >>
> > >> > Hello,
> > >> >
> > >> > As Aljoshca said, the TriggerDSL depends on another PR, before it
> can
> > be
> > >> > merged.
> > >> > In addition, this will change also the whole structure of how we
> test
> > >> > window/trigger
> > >> > related functionality, so some refactoring will be required.
> > >> >
> > >> > Kostas
> > >> >
> > >> > > On Oct 28, 2016, at 3:06 PM, Aljoscha Krettek <
> aljoscha@apache.org>
> > >> > wrote:
> > >> > >
> > >> > > @Till: For the TriggerDSL we have to merge my open PR [1] for
> > >> refactoring
> > >> > > the WindowOperator/Trigger tests and then refactor the tests of
> the
> > >> > > TriggerDSL to use the new test facilities introduced there. Plus
> > then
> > >> > maybe
> > >> > > some fixups.
> > >> > >
> > >> > > On Fri, 28 Oct 2016 at 11:17 Jark Wu <wu...@alibaba-inc.com>
> > >> wrote:
> > >> > >
> > >> > >> +1 to the schedule
> > >> > >>
> > >> > >> I hope to add
> > >> > >>
> > >> > >> 1) Table API: User defined table function (FLINK-4469 <
> > >> > >> https://issues.apache.org/jira/browse/FLINK-4469>)
> > >> > >> 2) Table API: Move Table API to org.apache.flink.table
> (FLINK-4704
> > <
> > >> > >> https://issues.apache.org/jira/browse/FLINK-4704>)
> > >> > >>
> > >> > >> The 1) has been discussed for a long time and a pull request has
> > >> been on
> > >> > >> github. The 2) would be a large change, but I hope it could be
> > >> included
> > >> > in
> > >> > >> 1.2 release, otherwise it will be more difficult to fix it.
> > >> > >>
> > >> > >> - Jark Wu
> > >> > >>
> > >> > >>> 在 2016年10月26日，下午8:09，Till Rohrmann <tr...@apache.org> 写道：
> > >> > >>>
> > >> > >>> +1 for the features to include.
> > >> > >>>
> > >> > >>> What is the state of the Trigger DSL? How much is left to be
> done
> > >> > before
> > >> > >>> merging?
> > >> > >>>
> > >> > >>> Cheers,
> > >> > >>> Till
> > >> > >>>
> > >> > >>> On Tue, Oct 25, 2016 at 4:32 PM, Aljoscha Krettek <
> > >> aljoscha@apache.org
> > >> > >
> > >> > >>> wrote:
> > >> > >>>
> > >> > >>>> +1 the schedule proposed so far.
> > >> > >>>>
> > >> > >>>> Do we also want to get in the "Trigger DSL" that we've had
> > brewing
> > >> > for a
> > >> > >>>> while now?
> > >> > >>>>
> > >> > >>>> On Mon, 17 Oct 2016 at 16:17 Stephan Ewen <se...@apache.org>
> > >> wrote:
> > >> > >>>>
> > >> > >>>>> I think this sounds very reasonable, +1 to the schedule.
> > >> > >>>>>
> > >> > >>>>> I would definitely add
> > >> > >>>>> - FLIP-10 (unify checkpoints and savepoints)
> > >> > >>>>> - FLIP-7 (metrics in web UI)
> > >> > >>>>> - FLIP-12 (async request operators)
> > >> > >>>>>
> > >> > >>>>> Those should be all safe bets, as they are basically done.
> > >> > >>>>>
> > >> > >>>>> Let's see what else is in shape until the feature freeze day.
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>> On Mon, Oct 17, 2016 at 2:18 PM, Fabian Hueske <
> > fhueske@gmail.com
> > >> >
> > >> > >>>> wrote:
> > >> > >>>>>
> > >> > >>>>>> Hi everybody,
> > >> > >>>>>>
> > >> > >>>>>> Flink 1.1.0 was released in August and I think it is time to
> > kick
> > >> > off
> > >> > >> a
> > >> > >>>>>> discussion about the schedule and scope of Flink 1.2.0.
> > >> > >>>>>>
> > >> > >>>>>> == Scope
> > >> > >>>>>>
> > >> > >>>>>> We started to collect features for Flink 1.2.0 in the Flink
> > >> Release
> > >> > >>>> wiki
> > >> > >>>>>> page [1].
> > >> > >>>>>> I copy the feature list for convenience:
> > >> > >>>>>>
> > >> > >>>>>> - Dynamic Scaling / Key Groups (FLINK-3755
> > >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3755>) (Design
> > Doc
> > >> > >>>>>> <https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-
> > >> > >>>>>> LuBCyPUWyFd9l3T9WyssQ63w/edit#heading=h.2suhu1fjxmp4>
> > >> > >>>>>> )
> > >> > >>>>>> - Rescalable Non-Partitioned State (FLIP-8
> > >> > >>>>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-
> > >> > >>>>>> 8%3A+Rescalable+Non-Partitioned+State>
> > >> > >>>>>> )
> > >> > >>>>>> - Backwards compatible savepoints (FLINK-4797
> > >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-4797>)
> > >> > >>>>>> -
> > >> > >>>>>>
> > >> > >>>>>> Integrate Flink with Apache Mesos (FLINK-1984
> > >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-1984>) (Design
> > Doc
> > >> > >>>>>> <https://docs.google.com/document/d/
> 1WItafBmGbjlaBbP8Of5PAFO
> > >> H9GUJQ
> > >> > >>>>>> xf5S4hjEuPchuU/edit#heading=h.33uxp8abvuf0>
> > >> > >>>>>> )
> > >> > >>>>>> - Secure Data Access (FLINK-3930
> > >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3930>) (Design
> > Doc
> > >> > >>>>>> <https://docs.google.com/document/d/1-
> > >> > >>>> GQB6uVOyoaXGwtqwqLV8BHDxWiMO2W
> > >> > >>>>>> nVzBoJ8oPaAs/edit#heading=h.dr22iualfm0h>
> > >> > >>>>>> )
> > >> > >>>>>> - Queryable State (FLINK-3779
> > >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3779>)
> > >> > >>>>>> - Metrics in Webinterface (FLIP-7
> > >> > >>>>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-
> > >> > >>>>>> 7%3A+Expose+metrics+to+WebInterface>
> > >> > >>>>>> )
> > >> > >>>>>> - Kafka 0.10 support (FLINK-4035
> > >> > >>>>>> <http://issues.apache.org/jira/browse/FLINK-4035>)
> > >> > >>>>>> - Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
> > >> > >>>>>> - Table API: Scalar Functions (FLINK-3097)
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> Is a feature missing or something too ambitious?
> > >> > >>>>>>
> > >> > >>>>>> == Schedule
> > >> > >>>>>>
> > >> > >>>>>> In my opinion we should try to release 1.2.0 before
> Christmas.
> > >> That
> > >> > >>>>> means,
> > >> > >>>>>> we should have a RC1 by end November / early December and a
> > >> feature
> > >> > >>>>> freeze
> > >> > >>>>>> two weeks earlier in mid of November. Does that sound
> > >> appropriate or
> > >> > >>>>> should
> > >> > >>>>>> we aim for an earlier or later release?
> > >> > >>>>>>
> > >> > >>>>>> What do others think about the scope and schedule?
> > >> > >>>>>> Please discuss.
> > >> > >>>>>>
> > >> > >>>>>> Best,
> > >> > >>>>>> Fabian
> > >> > >>>>>>
> > >> > >>>>>> [1]
> > >> > >>>>>> https://cwiki.apache.org/confluence/display/FLINK/
> > >> > >>>>>> Flink+Release+and+Feature+Plan
> > >> > >>>>>>
> > >> > >>>>>
> > >> > >>>>
> > >> > >>
> > >> > >>
> > >> >
> > >> >
> > >>
> > >
> > >
> > >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Stephan Ewen <se...@apache.org>.
Thanks Robert for bringing this up again.
Would be great to come to that
@use: I'll take a look at https://github.com/apache/flink/pull/2752
Considering "UNRESOLVED Provide support for asynchronous operations over
streams (FLINK-4391)" - that is currently in the hands of the contributor.
I would like it if it went into the release, but would not block the
release on that, if it is not ready in time.
Best,
Stephan
On Tue, Dec 6, 2016 at 2:32 PM, Ufuk Celebi <uc...@apache.org> wrote:
> Hey Robert,
>
> thanks for bringing the discussion up again.
>
> On Mon, Dec 5, 2016 at 6:28 PM, Robert Metzger <rm...@apache.org>
> wrote:
> > - UNRESOLVED Unify Savepoints and Checkpoints (FLINK-4484)
>
> This issue/FLIP contained two parts:
> 1. Write periodic checkpoints out to external storage
> 2. Allow periodic scheduling of savepoints
>
> (1) is done. (2) is aimed for after 1.2.
>
> There are two small non-blocking PRs for (1) that would be nice to get
> in before 1.2:
> https://github.com/apache/flink/pull/2750
> https://github.com/apache/flink/pull/2752
>
> – Ufuk
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Ufuk Celebi <uc...@apache.org>.
Hey Robert,
thanks for bringing the discussion up again.
On Mon, Dec 5, 2016 at 6:28 PM, Robert Metzger <rm...@apache.org> wrote:
> - UNRESOLVED Unify Savepoints and Checkpoints (FLINK-4484)
This issue/FLIP contained two parts:
1. Write periodic checkpoints out to external storage
2. Allow periodic scheduling of savepoints
(1) is done. (2) is aimed for after 1.2.
There are two small non-blocking PRs for (1) that would be nice to get
in before 1.2:
https://github.com/apache/flink/pull/2750
https://github.com/apache/flink/pull/2752
– Ufuk
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Robert Metzger <rm...@apache.org>.
Hi,
I would like to reactivate this old email thread. The initial plan was to
release 1.2 before Christmas, have RC1 by end of Nov / December and a
feature freeze mid-November.
Today (December 5), we did not yet do the feature freeze. I would like to
get an overview again of the remaining blockers and what we want to include
into the release.
From the issues Fabian posted in the beginning, they have the following
status:
- RESOLVED dynamic Scaling / Key Groups (FLINK-3755)
- RESOLVED Add Rescalable Non-Partitioned State (FLINK-4379)
- UNRESOLVED Add Flink 1.1 savepoint backwards compatability (FLINK-4797)
- UNRESOLVED Integrate Flink with Apache Mesos (FLINK-1984)
- UNRESOLVED Secure Data Access (FLINK-3930)
- RESOLVED Queryable State (FLINK-3779)
- RESOLVED Metrics in Webinterface (FLINK-4389)
- RESOLVED Kafka 0.10 support (FLINK-4035)
- RESOLVED Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
- RESOLVED Table API: Scalar Functions (FLINK-3097)
Added by Stephan:
- UNRESOLVED Provide support for asynchronous operations over streams
(FLINK-4391)
- UNRESOLVED Unify Savepoints and Checkpoints (FLINK-4484)
I've also reflected the status in the Wiki:
https://cwiki.apache.org/confluence/display/FLINK/1.2+Release
@Flavio: I assume PR #2790 is resolved and not a release blocker.
Given that we are already behind the schedule, I'm wondering what we can do
regarding the remaining UNRESOLVED issues.
I would also be willing to release some of them as undocumented,
work-in-progress features. We have so many good things like dynamic scaling
or Mesos in the release that we should get it out rather soon.
It would be nice if those committers / contributors working on the
UNRESOLVED issues could give a more detailed status update what they think
release-wise.
Also, is there anything else we need to consider before forking off the
release-1.2 branch from master?
Once we've collected the detailed status updates from the last unresolved
features, we can discuss a new date for the feature freeze.
I'll then also create a RC0 testing release candidate without a VOTE, to
have a starting point for the first big release testing. (And yes, I'm
volunteering to be the release manager for the 1.2 release ;) )
Regards,
Robert
On Mon, Nov 21, 2016 at 3:03 PM, Flavio Pompermaier <po...@okkam.it>
wrote:
> Hi to all,
> we've completed the PR #2790 <https://github.com/apache/flink/pull/2790>
> to
> implement FLINK-4491 <https://issues.apache.org/jira/browse/FLINK-4491>:
> we've added the comment that @zentol asked for a week ago and we also fixed
> a potential annoying problem we faced during indexing (if a malformed
> document generates an error during indexing, the close() method throws an
> exception forcing the entire job to fail, killing other threads that are
> still indexing). I hope this PR could hep other people working with
> Elasticsearch, right from the next release ;)
>
> On Mon, Nov 14, 2016 at 2:47 PM, Flavio Pompermaier <po...@okkam.it>
> wrote:
>
> > Hi to all,
> > for the 1.2 release do you think it will be possible to include also a PR
> > to implement FLINK-4491 <https://issues.apache.org/
> jira/browse/FLINK-4491>
> > ?
> > The PR is waiting for a review at https://github.com/apache/
> > flink/pull/2790.
> >
> > Best,
> > Flavio
> >
> > On Fri, Oct 28, 2016 at 5:59 PM, Fabian Hueske <fh...@gmail.com>
> wrote:
> >
> >> Hi Jark,
> >>
> >> FLINK-4469: I agree, UDTFs would be a great feature. I don't think we
> >> should make it a blocker but I will help to get it the master before the
> >> feature freeze.
> >> FLINK-4704: Regarding the renaming of the namespace, we have two
> positive
> >> votes in JIRA, but we should also start a discussion on the dev and user
> >> list to collect some more input. The actual change should be simple.
> >>
> >> @Everybody: I created a wiki page for the 1.2 release [1] and added all
> >> features discussed in this thread so far.
> >> Please have a look at the page and update it if you know the JIRAs
> and/or
> >> state of the features.
> >>
> >> Cheers, Fabian
> >>
> >> [1] https://cwiki.apache.org/confluence/display/FLINK/1.2+Release
> >>
> >> 2016-10-28 15:28 GMT+02:00 Kostas Kloudas <k.kloudas@data-artisans.com
> >:
> >>
> >> > Hello,
> >> >
> >> > As Aljoshca said, the TriggerDSL depends on another PR, before it can
> be
> >> > merged.
> >> > In addition, this will change also the whole structure of how we test
> >> > window/trigger
> >> > related functionality, so some refactoring will be required.
> >> >
> >> > Kostas
> >> >
> >> > > On Oct 28, 2016, at 3:06 PM, Aljoscha Krettek <al...@apache.org>
> >> > wrote:
> >> > >
> >> > > @Till: For the TriggerDSL we have to merge my open PR [1] for
> >> refactoring
> >> > > the WindowOperator/Trigger tests and then refactor the tests of the
> >> > > TriggerDSL to use the new test facilities introduced there. Plus
> then
> >> > maybe
> >> > > some fixups.
> >> > >
> >> > > On Fri, 28 Oct 2016 at 11:17 Jark Wu <wu...@alibaba-inc.com>
> >> wrote:
> >> > >
> >> > >> +1 to the schedule
> >> > >>
> >> > >> I hope to add
> >> > >>
> >> > >> 1) Table API: User defined table function (FLINK-4469 <
> >> > >> https://issues.apache.org/jira/browse/FLINK-4469>)
> >> > >> 2) Table API: Move Table API to org.apache.flink.table (FLINK-4704
> <
> >> > >> https://issues.apache.org/jira/browse/FLINK-4704>)
> >> > >>
> >> > >> The 1) has been discussed for a long time and a pull request has
> >> been on
> >> > >> github. The 2) would be a large change, but I hope it could be
> >> included
> >> > in
> >> > >> 1.2 release, otherwise it will be more difficult to fix it.
> >> > >>
> >> > >> - Jark Wu
> >> > >>
> >> > >>> 在 2016年10月26日，下午8:09，Till Rohrmann <tr...@apache.org> 写道：
> >> > >>>
> >> > >>> +1 for the features to include.
> >> > >>>
> >> > >>> What is the state of the Trigger DSL? How much is left to be done
> >> > before
> >> > >>> merging?
> >> > >>>
> >> > >>> Cheers,
> >> > >>> Till
> >> > >>>
> >> > >>> On Tue, Oct 25, 2016 at 4:32 PM, Aljoscha Krettek <
> >> aljoscha@apache.org
> >> > >
> >> > >>> wrote:
> >> > >>>
> >> > >>>> +1 the schedule proposed so far.
> >> > >>>>
> >> > >>>> Do we also want to get in the "Trigger DSL" that we've had
> brewing
> >> > for a
> >> > >>>> while now?
> >> > >>>>
> >> > >>>> On Mon, 17 Oct 2016 at 16:17 Stephan Ewen <se...@apache.org>
> >> wrote:
> >> > >>>>
> >> > >>>>> I think this sounds very reasonable, +1 to the schedule.
> >> > >>>>>
> >> > >>>>> I would definitely add
> >> > >>>>> - FLIP-10 (unify checkpoints and savepoints)
> >> > >>>>> - FLIP-7 (metrics in web UI)
> >> > >>>>> - FLIP-12 (async request operators)
> >> > >>>>>
> >> > >>>>> Those should be all safe bets, as they are basically done.
> >> > >>>>>
> >> > >>>>> Let's see what else is in shape until the feature freeze day.
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> On Mon, Oct 17, 2016 at 2:18 PM, Fabian Hueske <
> fhueske@gmail.com
> >> >
> >> > >>>> wrote:
> >> > >>>>>
> >> > >>>>>> Hi everybody,
> >> > >>>>>>
> >> > >>>>>> Flink 1.1.0 was released in August and I think it is time to
> kick
> >> > off
> >> > >> a
> >> > >>>>>> discussion about the schedule and scope of Flink 1.2.0.
> >> > >>>>>>
> >> > >>>>>> == Scope
> >> > >>>>>>
> >> > >>>>>> We started to collect features for Flink 1.2.0 in the Flink
> >> Release
> >> > >>>> wiki
> >> > >>>>>> page [1].
> >> > >>>>>> I copy the feature list for convenience:
> >> > >>>>>>
> >> > >>>>>> - Dynamic Scaling / Key Groups (FLINK-3755
> >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3755>) (Design
> Doc
> >> > >>>>>> <https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-
> >> > >>>>>> LuBCyPUWyFd9l3T9WyssQ63w/edit#heading=h.2suhu1fjxmp4>
> >> > >>>>>> )
> >> > >>>>>> - Rescalable Non-Partitioned State (FLIP-8
> >> > >>>>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-
> >> > >>>>>> 8%3A+Rescalable+Non-Partitioned+State>
> >> > >>>>>> )
> >> > >>>>>> - Backwards compatible savepoints (FLINK-4797
> >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-4797>)
> >> > >>>>>> -
> >> > >>>>>>
> >> > >>>>>> Integrate Flink with Apache Mesos (FLINK-1984
> >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-1984>) (Design
> Doc
> >> > >>>>>> <https://docs.google.com/document/d/1WItafBmGbjlaBbP8Of5PAFO
> >> H9GUJQ
> >> > >>>>>> xf5S4hjEuPchuU/edit#heading=h.33uxp8abvuf0>
> >> > >>>>>> )
> >> > >>>>>> - Secure Data Access (FLINK-3930
> >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3930>) (Design
> Doc
> >> > >>>>>> <https://docs.google.com/document/d/1-
> >> > >>>> GQB6uVOyoaXGwtqwqLV8BHDxWiMO2W
> >> > >>>>>> nVzBoJ8oPaAs/edit#heading=h.dr22iualfm0h>
> >> > >>>>>> )
> >> > >>>>>> - Queryable State (FLINK-3779
> >> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3779>)
> >> > >>>>>> - Metrics in Webinterface (FLIP-7
> >> > >>>>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-
> >> > >>>>>> 7%3A+Expose+metrics+to+WebInterface>
> >> > >>>>>> )
> >> > >>>>>> - Kafka 0.10 support (FLINK-4035
> >> > >>>>>> <http://issues.apache.org/jira/browse/FLINK-4035>)
> >> > >>>>>> - Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
> >> > >>>>>> - Table API: Scalar Functions (FLINK-3097)
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> Is a feature missing or something too ambitious?
> >> > >>>>>>
> >> > >>>>>> == Schedule
> >> > >>>>>>
> >> > >>>>>> In my opinion we should try to release 1.2.0 before Christmas.
> >> That
> >> > >>>>> means,
> >> > >>>>>> we should have a RC1 by end November / early December and a
> >> feature
> >> > >>>>> freeze
> >> > >>>>>> two weeks earlier in mid of November. Does that sound
> >> appropriate or
> >> > >>>>> should
> >> > >>>>>> we aim for an earlier or later release?
> >> > >>>>>>
> >> > >>>>>> What do others think about the scope and schedule?
> >> > >>>>>> Please discuss.
> >> > >>>>>>
> >> > >>>>>> Best,
> >> > >>>>>> Fabian
> >> > >>>>>>
> >> > >>>>>> [1]
> >> > >>>>>> https://cwiki.apache.org/confluence/display/FLINK/
> >> > >>>>>> Flink+Release+and+Feature+Plan
> >> > >>>>>>
> >> > >>>>>
> >> > >>>>
> >> > >>
> >> > >>
> >> >
> >> >
> >>
> >
> >
> >
>
Re: [DISCUSS] Schedule and Scope for Flink 1.2
Posted by Flavio Pompermaier <po...@okkam.it>.
Hi to all,
we've completed the PR #2790 <https://github.com/apache/flink/pull/2790> to
implement FLINK-4491 <https://issues.apache.org/jira/browse/FLINK-4491>:
we've added the comment that @zentol asked for a week ago and we also fixed
a potential annoying problem we faced during indexing (if a malformed
document generates an error during indexing, the close() method throws an
exception forcing the entire job to fail, killing other threads that are
still indexing). I hope this PR could hep other people working with
Elasticsearch, right from the next release ;)
On Mon, Nov 14, 2016 at 2:47 PM, Flavio Pompermaier <po...@okkam.it>
wrote:
> Hi to all,
> for the 1.2 release do you think it will be possible to include also a PR
> to implement FLINK-4491 <https://issues.apache.org/jira/browse/FLINK-4491>
> ?
> The PR is waiting for a review at https://github.com/apache/
> flink/pull/2790.
>
> Best,
> Flavio
>
> On Fri, Oct 28, 2016 at 5:59 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
>> Hi Jark,
>>
>> FLINK-4469: I agree, UDTFs would be a great feature. I don't think we
>> should make it a blocker but I will help to get it the master before the
>> feature freeze.
>> FLINK-4704: Regarding the renaming of the namespace, we have two positive
>> votes in JIRA, but we should also start a discussion on the dev and user
>> list to collect some more input. The actual change should be simple.
>>
>> @Everybody: I created a wiki page for the 1.2 release [1] and added all
>> features discussed in this thread so far.
>> Please have a look at the page and update it if you know the JIRAs and/or
>> state of the features.
>>
>> Cheers, Fabian
>>
>> [1] https://cwiki.apache.org/confluence/display/FLINK/1.2+Release
>>
>> 2016-10-28 15:28 GMT+02:00 Kostas Kloudas <k....@data-artisans.com>:
>>
>> > Hello,
>> >
>> > As Aljoshca said, the TriggerDSL depends on another PR, before it can be
>> > merged.
>> > In addition, this will change also the whole structure of how we test
>> > window/trigger
>> > related functionality, so some refactoring will be required.
>> >
>> > Kostas
>> >
>> > > On Oct 28, 2016, at 3:06 PM, Aljoscha Krettek <al...@apache.org>
>> > wrote:
>> > >
>> > > @Till: For the TriggerDSL we have to merge my open PR [1] for
>> refactoring
>> > > the WindowOperator/Trigger tests and then refactor the tests of the
>> > > TriggerDSL to use the new test facilities introduced there. Plus then
>> > maybe
>> > > some fixups.
>> > >
>> > > On Fri, 28 Oct 2016 at 11:17 Jark Wu <wu...@alibaba-inc.com>
>> wrote:
>> > >
>> > >> +1 to the schedule
>> > >>
>> > >> I hope to add
>> > >>
>> > >> 1) Table API: User defined table function (FLINK-4469 <
>> > >> https://issues.apache.org/jira/browse/FLINK-4469>)
>> > >> 2) Table API: Move Table API to org.apache.flink.table (FLINK-4704 <
>> > >> https://issues.apache.org/jira/browse/FLINK-4704>)
>> > >>
>> > >> The 1) has been discussed for a long time and a pull request has
>> been on
>> > >> github. The 2) would be a large change, but I hope it could be
>> included
>> > in
>> > >> 1.2 release, otherwise it will be more difficult to fix it.
>> > >>
>> > >> - Jark Wu
>> > >>
>> > >>> 在 2016年10月26日，下午8:09，Till Rohrmann <tr...@apache.org> 写道：
>> > >>>
>> > >>> +1 for the features to include.
>> > >>>
>> > >>> What is the state of the Trigger DSL? How much is left to be done
>> > before
>> > >>> merging?
>> > >>>
>> > >>> Cheers,
>> > >>> Till
>> > >>>
>> > >>> On Tue, Oct 25, 2016 at 4:32 PM, Aljoscha Krettek <
>> aljoscha@apache.org
>> > >
>> > >>> wrote:
>> > >>>
>> > >>>> +1 the schedule proposed so far.
>> > >>>>
>> > >>>> Do we also want to get in the "Trigger DSL" that we've had brewing
>> > for a
>> > >>>> while now?
>> > >>>>
>> > >>>> On Mon, 17 Oct 2016 at 16:17 Stephan Ewen <se...@apache.org>
>> wrote:
>> > >>>>
>> > >>>>> I think this sounds very reasonable, +1 to the schedule.
>> > >>>>>
>> > >>>>> I would definitely add
>> > >>>>> - FLIP-10 (unify checkpoints and savepoints)
>> > >>>>> - FLIP-7 (metrics in web UI)
>> > >>>>> - FLIP-12 (async request operators)
>> > >>>>>
>> > >>>>> Those should be all safe bets, as they are basically done.
>> > >>>>>
>> > >>>>> Let's see what else is in shape until the feature freeze day.
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> On Mon, Oct 17, 2016 at 2:18 PM, Fabian Hueske <fhueske@gmail.com
>> >
>> > >>>> wrote:
>> > >>>>>
>> > >>>>>> Hi everybody,
>> > >>>>>>
>> > >>>>>> Flink 1.1.0 was released in August and I think it is time to kick
>> > off
>> > >> a
>> > >>>>>> discussion about the schedule and scope of Flink 1.2.0.
>> > >>>>>>
>> > >>>>>> == Scope
>> > >>>>>>
>> > >>>>>> We started to collect features for Flink 1.2.0 in the Flink
>> Release
>> > >>>> wiki
>> > >>>>>> page [1].
>> > >>>>>> I copy the feature list for convenience:
>> > >>>>>>
>> > >>>>>> - Dynamic Scaling / Key Groups (FLINK-3755
>> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3755>) (Design Doc
>> > >>>>>> <https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-
>> > >>>>>> LuBCyPUWyFd9l3T9WyssQ63w/edit#heading=h.2suhu1fjxmp4>
>> > >>>>>> )
>> > >>>>>> - Rescalable Non-Partitioned State (FLIP-8
>> > >>>>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-
>> > >>>>>> 8%3A+Rescalable+Non-Partitioned+State>
>> > >>>>>> )
>> > >>>>>> - Backwards compatible savepoints (FLINK-4797
>> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-4797>)
>> > >>>>>> -
>> > >>>>>>
>> > >>>>>> Integrate Flink with Apache Mesos (FLINK-1984
>> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-1984>) (Design Doc
>> > >>>>>> <https://docs.google.com/document/d/1WItafBmGbjlaBbP8Of5PAFO
>> H9GUJQ
>> > >>>>>> xf5S4hjEuPchuU/edit#heading=h.33uxp8abvuf0>
>> > >>>>>> )
>> > >>>>>> - Secure Data Access (FLINK-3930
>> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3930>) (Design Doc
>> > >>>>>> <https://docs.google.com/document/d/1-
>> > >>>> GQB6uVOyoaXGwtqwqLV8BHDxWiMO2W
>> > >>>>>> nVzBoJ8oPaAs/edit#heading=h.dr22iualfm0h>
>> > >>>>>> )
>> > >>>>>> - Queryable State (FLINK-3779
>> > >>>>>> <https://issues.apache.org/jira/browse/FLINK-3779>)
>> > >>>>>> - Metrics in Webinterface (FLIP-7
>> > >>>>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-
>> > >>>>>> 7%3A+Expose+metrics+to+WebInterface>
>> > >>>>>> )
>> > >>>>>> - Kafka 0.10 support (FLINK-4035
>> > >>>>>> <http://issues.apache.org/jira/browse/FLINK-4035>)
>> > >>>>>> - Table API: Group Window Aggregates (FLINK-4691, FLIP-11)
>> > >>>>>> - Table API: Scalar Functions (FLINK-3097)
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Is a feature missing or something too ambitious?
>> > >>>>>>
>> > >>>>>> == Schedule
>> > >>>>>>
>> > >>>>>> In my opinion we should try to release 1.2.0 before Christmas.
>> That
>> > >>>>> means,
>> > >>>>>> we should have a RC1 by end November / early December and a
>> feature
>> > >>>>> freeze
>> > >>>>>> two weeks earlier in mid of November. Does that sound
>> appropriate or
>> > >>>>> should
>> > >>>>>> we aim for an earlier or later release?
>> > >>>>>>
>> > >>>>>> What do others think about the scope and schedule?
>> > >>>>>> Please discuss.
>> > >>>>>>
>> > >>>>>> Best,
>> > >>>>>> Fabian
>> > >>>>>>
>> > >>>>>> [1]
>> > >>>>>> https://cwiki.apache.org/confluence/display/FLINK/
>> > >>>>>> Flink+Release+and+Feature+Plan
>> > >>>>>>
>> > >>>>>
>> > >>>>
>> > >>
>> > >>
>> >
>> >
>>
>
>
>