You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Robert Metzger <rm...@apache.org> on 2020/07/22 12:10:40 UTC
[DISCUSS] Planning Flink 1.12
Hi all,
Now that the 1.11 release is out, it is time to plan for the next major
Flink release.
Some items:
1.
Dian Fu and me volunteer to be the release managers for Flink 1.12.
1.
Timeline: We propose to stick to our approximate 4 month release cycle,
thus the release should be done by late October. Given that there’s a
holiday week in China at the beginning of October, I propose to do the
feature freeze on master by late September.
2.
Collecting features: It would be good to have a rough overview of the
features that will likely be ready to be merged by late September, and that
we want in the release.
Based on the discussion, we will update the Roadmap on the Flink website
again!
1.
Test instabilities and blockers: I would like to avoid a situation where
we have many blocking issues or build instabilities at the time of the
feature freeze. To achieve that, we will try to check every build
instability within a week, to decide if it is a blocker (make sure to use
the “test-stability” label for those tickets!)
Blocker issues will need to have somebody assigned (responsible) within
a week, and we want to see progress on all blocker issues (downgrade,
resolution, a good plan how to proceed if it is more complicated)
2.
Quality and stability of new features: In order to have a short feature
freeze phase, we encourage developers to only merge well-tested and
documented features. In our experience, the feature freeze works best if
new features are complete, and the community can focus fully on addressing
newly found bugs and voting the release.
By having a smooth release process, the next merge-window for the next
release will come sooner.
Let me know what you think about our items, and share which features you
want in Flink 1.12.
Best,
Robert & Dian
Re: [DISCUSS] Planning Flink 1.12
Posted by Seth Wiesman <sj...@gmail.com>.
@David Anderson <da...@alpinegizmo.com>
I think we can get that into 1.12. I have a branch that just needs to be
cleaned up.
+1 for https://issues.apache.org/jira/browse/FLINK-13095
On Mon, Aug 17, 2020 at 2:03 AM Till Rohrmann <tr...@apache.org> wrote:
> Hi Rodrigo,
>
> FLINK-10407 has not been up to date with the actual progress on the
> feature. We will update it soon.
>
> I think that the community won't actively work on FLINK-12002 in this
> release. However, we will work on FLINK-16430 [1] and once this is done
> continue with the clean up of the legacy scheduler code paths.
>
> As for the scope of the 1.12 release, we hope to finish FLINK-16430 and
> make considerable progress with FLINK-10407. However, I don't think that
> FLINK-10407 will be fully functional yet.
>
> [1] https://issues.apache.org/jira/browse/FLINK-16430
>
> Cheers,
> Till
>
> On Sat, Aug 15, 2020 at 6:43 PM rodrigobrochado <
> rodrigo.brochado@predito.com.br> wrote:
>
> > Thanks Dian!
> >
> > About the wiki page, I think that the "Reactive-scaling mode" by Till has
> > an
> > open issue on FLINK-10407 [1].
> >
> > Still about scaling, what about the adaptive parallelism of jobs [2]?
> This
> > is somehow related to Prasanna and Harold's comments above. It seems to
> > depend on finishing the old "Redesign Flink Scheduling" [3], FLIP-119
> > (already on wiki list), and FLINK-15626 (Remove of legacy scheduler) [4].
> > Would they be achievable until 1.12?
> >
> > If I could add one more, the python UDF in docker mode would be awesome
> > [5].
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-10407
> > [2] https://issues.apache.org/jira/browse/FLINK-12002
> > [3] https://issues.apache.org/jira/browse/FLINK-10429
> > [4] https://issues.apache.org/jira/browse/FLINK-15626
> > [5] https://issues.apache.org/jira/browse/FLINK-14025
> >
> > Thanks,
> > Rodrigo
> >
> >
> >
> > --
> > Sent from:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
> >
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Till Rohrmann <tr...@apache.org>.
Hi Rodrigo,
FLINK-10407 has not been up to date with the actual progress on the
feature. We will update it soon.
I think that the community won't actively work on FLINK-12002 in this
release. However, we will work on FLINK-16430 [1] and once this is done
continue with the clean up of the legacy scheduler code paths.
As for the scope of the 1.12 release, we hope to finish FLINK-16430 and
make considerable progress with FLINK-10407. However, I don't think that
FLINK-10407 will be fully functional yet.
[1] https://issues.apache.org/jira/browse/FLINK-16430
Cheers,
Till
On Sat, Aug 15, 2020 at 6:43 PM rodrigobrochado <
rodrigo.brochado@predito.com.br> wrote:
> Thanks Dian!
>
> About the wiki page, I think that the "Reactive-scaling mode" by Till has
> an
> open issue on FLINK-10407 [1].
>
> Still about scaling, what about the adaptive parallelism of jobs [2]? This
> is somehow related to Prasanna and Harold's comments above. It seems to
> depend on finishing the old "Redesign Flink Scheduling" [3], FLIP-119
> (already on wiki list), and FLINK-15626 (Remove of legacy scheduler) [4].
> Would they be achievable until 1.12?
>
> If I could add one more, the python UDF in docker mode would be awesome
> [5].
>
> [1] https://issues.apache.org/jira/browse/FLINK-10407
> [2] https://issues.apache.org/jira/browse/FLINK-12002
> [3] https://issues.apache.org/jira/browse/FLINK-10429
> [4] https://issues.apache.org/jira/browse/FLINK-15626
> [5] https://issues.apache.org/jira/browse/FLINK-14025
>
> Thanks,
> Rodrigo
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>
Re: [DISCUSS] Planning Flink 1.12
Posted by rodrigobrochado <ro...@predito.com.br>.
Thanks Dian!
About the wiki page, I think that the "Reactive-scaling mode" by Till has an
open issue on FLINK-10407 [1].
Still about scaling, what about the adaptive parallelism of jobs [2]? This
is somehow related to Prasanna and Harold's comments above. It seems to
depend on finishing the old "Redesign Flink Scheduling" [3], FLIP-119
(already on wiki list), and FLINK-15626 (Remove of legacy scheduler) [4].
Would they be achievable until 1.12?
If I could add one more, the python UDF in docker mode would be awesome [5].
[1] https://issues.apache.org/jira/browse/FLINK-10407
[2] https://issues.apache.org/jira/browse/FLINK-12002
[3] https://issues.apache.org/jira/browse/FLINK-10429
[4] https://issues.apache.org/jira/browse/FLINK-15626
[5] https://issues.apache.org/jira/browse/FLINK-14025
Thanks,
Rodrigo
--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
Re: [DISCUSS] Planning Flink 1.12
Posted by David Anderson <da...@alpinegizmo.com>.
What about filling in at least the most critical holes in the State
Processor API? As it stands now, many of its supposed benefits (e.g.,
migrate between state backends, change max parallelism) can't be realized
for applications that use windows.
https://issues.apache.org/jira/browse/FLINK-13095 - Provide an easy way to
read / bootstrap window state
I realize that some rework will also be needed soon to move the
implementation away from DataSet, but I hope we can see some usability
improvements in 1.12.
Regards,
David
On Fri, Aug 14, 2020 at 8:37 PM Steven Wu <st...@gmail.com> wrote:
> What about the work of migrating some Flink sources to the new FLIP-27
> source interface? They are not listed in the 1.12 release wiki page.
>
> On Thu, Aug 13, 2020 at 6:51 PM Dian Fu <di...@gmail.com> wrote:
>
> > Hi Rodrigo,
> >
> > Both FLIP-130 and FLIP-133 will be in the list of 1.12. Besides, there
> are
> > also some other features from PyFlink side in 1.12. More details could be
> > found in the wiki page(
> > https://cwiki.apache.org/confluence/display/FLINK/1.12+Release <
> > https://cwiki.apache.org/confluence/display/FLINK/1.12+Release>).
> >
> > Regards,
> > Dian
> >
> > > 在 2020年8月14日，上午9:37，rodrigobrochado <ro...@predito.com.br>
> > 写道：
> > >
> > > Hi,
> > >
> > > I hope it's not too late to ask, but would FLIP-130 [1] and FLIP-133
> [2]
> > be
> > > considered? I think that it would be nice to have some details of
> pyFlink
> > > Datastreams API (FLIP-130) on the roadmap, giving us (users) more
> > insights
> > > into what we can expect from pyFlink in the near future.
> > >
> > >
> > > [1]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-130-Support-for-Python-DataStream-API-Stateless-Part-td43035.html
> > > [2]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-133-Rework-PyFlink-Documentation-tt43570.html
> > >
> > >
> > > Thanks,
> > > Rodrigo
> > >
> > >
> > >
> > > --
> > > Sent from:
> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
> >
> >
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Steven Wu <st...@gmail.com>.
What about the work of migrating some Flink sources to the new FLIP-27
source interface? They are not listed in the 1.12 release wiki page.
On Thu, Aug 13, 2020 at 6:51 PM Dian Fu <di...@gmail.com> wrote:
> Hi Rodrigo,
>
> Both FLIP-130 and FLIP-133 will be in the list of 1.12. Besides, there are
> also some other features from PyFlink side in 1.12. More details could be
> found in the wiki page(
> https://cwiki.apache.org/confluence/display/FLINK/1.12+Release <
> https://cwiki.apache.org/confluence/display/FLINK/1.12+Release>).
>
> Regards,
> Dian
>
> > 在 2020年8月14日，上午9:37，rodrigobrochado <ro...@predito.com.br>
> 写道：
> >
> > Hi,
> >
> > I hope it's not too late to ask, but would FLIP-130 [1] and FLIP-133 [2]
> be
> > considered? I think that it would be nice to have some details of pyFlink
> > Datastreams API (FLIP-130) on the roadmap, giving us (users) more
> insights
> > into what we can expect from pyFlink in the near future.
> >
> >
> > [1]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-130-Support-for-Python-DataStream-API-Stateless-Part-td43035.html
> > [2]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-133-Rework-PyFlink-Documentation-tt43570.html
> >
> >
> > Thanks,
> > Rodrigo
> >
> >
> >
> > --
> > Sent from:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Dian Fu <di...@gmail.com>.
Hi Rodrigo,
Both FLIP-130 and FLIP-133 will be in the list of 1.12. Besides, there are also some other features from PyFlink side in 1.12. More details could be found in the wiki page(https://cwiki.apache.org/confluence/display/FLINK/1.12+Release <https://cwiki.apache.org/confluence/display/FLINK/1.12+Release>).
Regards,
Dian
> 在 2020年8月14日，上午9:37，rodrigobrochado <ro...@predito.com.br> 写道：
>
> Hi,
>
> I hope it's not too late to ask, but would FLIP-130 [1] and FLIP-133 [2] be
> considered? I think that it would be nice to have some details of pyFlink
> Datastreams API (FLIP-130) on the roadmap, giving us (users) more insights
> into what we can expect from pyFlink in the near future.
>
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-130-Support-for-Python-DataStream-API-Stateless-Part-td43035.html
> [2]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-133-Rework-PyFlink-Documentation-tt43570.html
>
>
> Thanks,
> Rodrigo
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
Re: [DISCUSS] Planning Flink 1.12
Posted by rodrigobrochado <ro...@predito.com.br>.
Hi,
I hope it's not too late to ask, but would FLIP-130 [1] and FLIP-133 [2] be
considered? I think that it would be nice to have some details of pyFlink
Datastreams API (FLIP-130) on the roadmap, giving us (users) more insights
into what we can expect from pyFlink in the near future.
[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-130-Support-for-Python-DataStream-API-Stateless-Part-td43035.html
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-133-Rework-PyFlink-Documentation-tt43570.html
Thanks,
Rodrigo
--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
Re: [DISCUSS] Planning Flink 1.12
Posted by Robert Metzger <rm...@apache.org>.
I updated the release date in the Wiki page.
On Sun, Aug 9, 2020 at 8:18 PM Yun Tang <my...@live.com> wrote:
> +1 for extending the feature freeze due date.
> ________________________________
> From: Zhijiang <wa...@aliyun.com.INVALID>
> Sent: Thursday, August 6, 2020 17:05
> To: dev <de...@flink.apache.org>
> Subject: Re: [DISCUSS] Planning Flink 1.12
>
> +1 on my side for feature freeze date by the end of Oct.
>
>
> ------------------------------------------------------------------
> From:Yuan Mei <yu...@gmail.com>
> Send Time:2020年8月6日(星期四) 14:54
> To:dev <de...@flink.apache.org>
> Subject:Re: [DISCUSS] Planning Flink 1.12
>
> +1
>
> > +1 for extending the feature freeze date to the end of October.
>
>
>
> On Thu, Aug 6, 2020 at 12:08 PM Yu Li <ca...@gmail.com> wrote:
>
> > +1 for extending feature freeze date to end of October.
> >
> > Feature development in the master branch could be unblocked through
> > creating the release branch, but every coin has its two sides (smile)
> >
> > Best Regards,
> > Yu
> >
> >
> > On Wed, 5 Aug 2020 at 20:12, Robert Metzger <rm...@apache.org> wrote:
> >
> > > Thanks all for your opinion.
> > >
> > > @Chesnay: That is a risk, but I hope the people responsible for
> > individual
> > > FLIPs plan accordingly. Extending the time till the feature freeze
> should
> > > not mean that we are extending the scope of the release.
> > > Ideally, features are done before FF, and they use the time till the
> > freeze
> > > for additional testing and documentation polishing.
> > > This FF will be virtual, there should be less disruption than a
> physical
> > > conference with all the travelling.
> > > Do you have a different proposal for the timing?
> > >
> > >
> > > I'm currently considering splitting the feature freeze and the release
> > > branch creation. Similar to the Linux kernel development, we could
> have a
> > > "merge window" and a stabilization phase. At the end of the
> stabilization
> > > phase, we cut the release branch and open the next merge window (I'll
> > start
> > > a separate thread regarding this towards the end of this release cycle,
> > if
> > > I still like the idea then)
> > >
> > >
> > > On Wed, Aug 5, 2020 at 12:04 PM Chesnay Schepler <ch...@apache.org>
> > > wrote:
> > >
> > > > I'm a bit concerned about end of October, because it means we have
> > Flink
> > > > forward, which usually means at least 1 week of little-to-no
> activity,
> > > > and then 1 week until feature-freeze.
> > > >
> > > > On 05/08/2020 11:56, jincheng sun wrote:
> > > > > +1 for end of October from me as well.
> > > > >
> > > > > Best,
> > > > > Jincheng
> > > > >
> > > > >
> > > > > Kostas Kloudas <kk...@gmail.com> 于2020年8月5日周三 下午4:59写道：
> > > > >
> > > > >> +1 for end of October from me as well.
> > > > >>
> > > > >> Cheers,
> > > > >> Kostas
> > > > >>
> > > > >> On Wed, Aug 5, 2020 at 9:59 AM Till Rohrmann <
> trohrmann@apache.org>
> > > > wrote:
> > > > >>
> > > > >>> +1 for end of October from my side as well.
> > > > >>>
> > > > >>> Cheers,
> > > > >>> Till
> > > > >>>
> > > > >>> On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen <se...@apache.org>
> > > wrote:
> > > > >>>
> > > > >>>> The end of October sounds good from my side, unless it collides
> > with
> > > > >> some
> > > > >>>> holidays that affect many committers.
> > > > >>>>
> > > > >>>> Feature-wise, I believe we can definitely make good use of the
> > time
> > > to
> > > > >>> wrap
> > > > >>>> up some critical threads (like finishing the FLIP-27 source
> > > efforts).
> > > > >>>>
> > > > >>>> So +1 to the end of October from my side.
> > > > >>>>
> > > > >>>> Best,
> > > > >>>> Stephan
> > > > >>>>
> > > > >>>>
> > > > >>>> On Tue, Aug 4, 2020 at 8:59 AM Robert Metzger <
> > rmetzger@apache.org>
> > > > >>> wrote:
> > > > >>>>> Thanks a lot for commenting on the feature freeze date.
> > > > >>>>>
> > > > >>>>> You are raising a few good points on the timing.
> > > > >>>>> If we have already (2 months before) concerns regarding the
> > > deadline,
> > > > >>>> then
> > > > >>>>> I agree that we should move it till the end of October.
> > > > >>>>>
> > > > >>>>> We then just need to be careful not to run into the Christmas
> > > season
> > > > >> at
> > > > >>>> the
> > > > >>>>> end of December.
> > > > >>>>>
> > > > >>>>> If nobody objects within a few days, I'll update the feature
> > freeze
> > > > >>> date
> > > > >>>> in
> > > > >>>>> the Wiki.
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <yk...@gmail.com>
> > > wrote:
> > > > >>>>>
> > > > >>>>>> Regarding setting the feature freeze date to late September, I
> > > have
> > > > >>>> some
> > > > >>>>>> concern that it might make
> > > > >>>>>> the development time of 1.12 too short.
> > > > >>>>>>
> > > > >>>>>> One reason for this is we took too much time (about 1.5 month,
> > > from
> > > > >>> mid
> > > > >>>>> of
> > > > >>>>>> May to beginning of July)
> > > > >>>>>> for testing 1.11. It's not ideal but further squeeze the
> > > > >> development
> > > > >>>> time
> > > > >>>>>> of 1.12 won't make this better.
> > > > >>>>>> Besides, AFAIK July & August is also a popular vacation
> season
> > > for
> > > > >>>>>> European. Given the fact most
> > > > >>>>>> committers of Flink come from Europe, I think we should also
> > > take
> > > > >>> this
> > > > >>>>>> into consideration.
> > > > >>>>>>
> > > > >>>>>> It's also true that the first week of October is the national
> > > > >> holiday
> > > > >>>> of
> > > > >>>>>> China, so I'm wondering whether the
> > > > >>>>>> end of October could be a candidate feature freeze date.
> > > > >>>>>>
> > > > >>>>>> Best,
> > > > >>>>>> Kurt
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <
> > > > >> rmetzger@apache.org>
> > > > >>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hi all,
> > > > >>>>>>>
> > > > >>>>>>> Thanks a lot for the responses so far. I've put them into
> this
> > > > >> Wiki
> > > > >>>>> page:
> > > > >>>>>>>
> https://cwiki.apache.org/confluence/display/FLINK/1.12+Release
> > > > >> to
> > > > >>>> keep
> > > > >>>>>>> track of them. Ideally, post JIRA tickets for your feature,
> > then
> > > > >>> the
> > > > >>>>>> status
> > > > >>>>>>> will update automatically in the wiki :)
> > > > >>>>>>>
> > > > >>>>>>> Please keep posting features here, or add them to the Wiki
> > > > >> yourself
> > > > >>>> 🙏
> > > > >>>>>>> @Prasanna kumar <pr...@gmail.com>: Dynamic
> Auto
> > > > >>>> Scaling
> > > > >>>>>> is a
> > > > >>>>>>> feature request the community is well-aware of. Till has
> posted
> > > > >>>>>>> "Reactive-scaling mode" as a feature he's working on for the
> > 1.12
> > > > >>>>>> release.
> > > > >>>>>>> This work will introduce the basic building blocks and
> partial
> > > > >>>> support
> > > > >>>>>> for
> > > > >>>>>>> the feature you are requesting.
> > > > >>>>>>> Proper support for dynamic scaling, while maintaining Flink's
> > > > >> high
> > > > >>>>>>> performance (throughout, low latency) and correctness is a
> > > > >>> difficult
> > > > >>>>> task
> > > > >>>>>>> that needs a lot of work. It will probably take a little bit
> of
> > > > >>> time
> > > > >>>>> till
> > > > >>>>>>> this is fully available.
> > > > >>>>>>>
> > > > >>>>>>> Cheers,
> > > > >>>>>>> Robert
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <
> > > > >>> trohrmann@apache.org>
> > > > >>>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Thanks for being our release managers for the 1.12 release
> > > > >> Dian &
> > > > >>>>>> Robert!
> > > > >>>>>>>> Here are some features I would like to work on for this
> > > > >> release:
> > > > >>>>>>>> # Features
> > > > >>>>>>>>
> > > > >>>>>>>> ## Finishing pipelined region scheduling (
> > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-16430)
> > > > >>>>>>>> With the pipelined region scheduler we want to implement a
> > > > >>>> scheduler
> > > > >>>>>>> which
> > > > >>>>>>>> can serve streaming as well as batch workloads alike while
> > > > >> being
> > > > >>>> able
> > > > >>>>>> to
> > > > >>>>>>>> run jobs under constrained resources. The latter is
> > > > >> particularly
> > > > >>>>>>> important
> > > > >>>>>>>> for bounded streaming jobs which, currently, are not well
> > > > >>>> supported.
> > > > >>>>>>>> ## Reactive-scaling mode
> > > > >>>>>>>> Being able to react to newly available resources and
> rescaling
> > > > >> a
> > > > >>>>>> running
> > > > >>>>>>>> job accordingly will make Flink's operation much easier
> > because
> > > > >>>>>> resources
> > > > >>>>>>>> can then be controlled by an external tool (e.g. GCP
> > > > >> autoscaling,
> > > > >>>> K8s
> > > > >>>>>>>> horizontal pod scaler, etc.). In this release we want to
> make
> > a
> > > > >>> big
> > > > >>>>>> step
> > > > >>>>>>>> towards this direction. As a first step we want to support
> the
> > > > >>>>>> execution
> > > > >>>>>>> of
> > > > >>>>>>>> jobs with a parallelism which is lower than the specified
> > > > >>>> parallelism
> > > > >>>>>> in
> > > > >>>>>>>> case that Flink lost a TaskManager or could not acquire
> enough
> > > > >>>>>> resources.
> > > > >>>>>>>> # Maintenance/Stability
> > > > >>>>>>>>
> > > > >>>>>>>> ## JM / TM finished task reconciliation (
> > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-17075)
> > > > >>>>>>>> This prevents the system from going out of sync if a task
> > state
> > > > >>>>> change
> > > > >>>>>>> from
> > > > >>>>>>>> the TM to the JM is lost.
> > > > >>>>>>>>
> > > > >>>>>>>> ## Make metrics services work with Kubernetes deployments (
> > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-11127)
> > > > >>>>>>>> Invert the direction in which the MetricFetcher connects to
> > the
> > > > >>>>>>>> MetricQueryFetchers. That way it will no longer be necessary
> > to
> > > > >>>>> expose
> > > > >>>>>> on
> > > > >>>>>>>> K8s for every TaskManager a port on which the
> > > > >> MetricQueryFetcher
> > > > >>>>> runs.
> > > > >>>>>>> This
> > > > >>>>>>>> will then make the deployment of Flink clusters on K8s
> easier.
> > > > >>>>>>>>
> > > > >>>>>>>> ## Handle long-blocking operations during job submission
> > > > >>> (savepoint
> > > > >>>>>>>> restore) (https://issues.apache.org/jira/browse/FLINK-16866
> )
> > > > >>>>>>>> Submitting a Flink job can involve the interaction with
> > > > >> external
> > > > >>>>>> systems
> > > > >>>>>>>> (blocking operations). Depending on the job the interactions
> > > > >> can
> > > > >>>> take
> > > > >>>>>> so
> > > > >>>>>>>> long that it exceeds the submission timeout which reports a
> > > > >>> failure
> > > > >>>>> on
> > > > >>>>>>> the
> > > > >>>>>>>> client side even though the actual submission succeeded. By
> > > > >>>>> decoupling
> > > > >>>>>>> the
> > > > >>>>>>>> creation of the ExecutionGraph from the job submission, we
> can
> > > > >>> make
> > > > >>>>> the
> > > > >>>>>>> job
> > > > >>>>>>>> submission non-blocking which will solve this problem.
> > > > >>>>>>>>
> > > > >>>>>>>> ## Make IDs more intuitive to ease debugging (FLIP-118) (
> > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-15679)
> > > > >>>>>>>> By making the internal Flink IDs compositional or logging
> how
> > > > >>> they
> > > > >>>>>> belong
> > > > >>>>>>>> together, we can make the debugging of Flink's operations
> much
> > > > >>>>> easier.
> > > > >>>>>>>> Cheers,
> > > > >>>>>>>> Till
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <
> > > > >>>> felixzhengcb@gmail.com
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Hi All,
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks for bring-up this discussion, Robert!
> > > > >>>>>>>>> Congratulations on becoming the release manager of 1.12，
> Dian
> > > > >>> and
> > > > >>>>>>> Robert
> > > > >>>>>>>> !
> > > > >>>>>>>>> ----------
> > > > >>>>>>>>> Here are some of my thoughts of the features for native
> > > > >>>> integration
> > > > >>>>>>> with
> > > > >>>>>>>>> Kubernetes in Flink 1.12:
> > > > >>>>>>>>>
> > > > >>>>>>>>> 1. Support user-specified pod templates
> > > > >>>>>>>>> Description:
> > > > >>>>>>>>> The current approach of introducing new configuration
> > > > >>> options
> > > > >>>>> for
> > > > >>>>>>>> each
> > > > >>>>>>>>> aspect of pod specification a user might wish is becoming
> > > > >>>> unwieldy,
> > > > >>>>>> we
> > > > >>>>>>>> have
> > > > >>>>>>>>> to maintain more and more Flink side Kubernetes
> configuration
> > > > >>>>> options
> > > > >>>>>>> and
> > > > >>>>>>>>> users have to learn the gap between the declarative model
> > > > >> used
> > > > >>> by
> > > > >>>>>>>>> Kubernetes and the configuration model used by Flink. It's
> a
> > > > >>>> great
> > > > >>>>>>>>> improvement to allow users to specify pod templates as
> > > > >> central
> > > > >>>>> places
> > > > >>>>>>> for
> > > > >>>>>>>>> all customization needs for the jobmanager and taskmanager
> > > > >>> pods.
> > > > >>>>>>>>> Benefits:
> > > > >>>>>>>>> Users can leverage many of the advanced K8s features
> > that
> > > > >>> the
> > > > >>>>>> Flink
> > > > >>>>>>>>> community does not support explicitly, such as volume
> > > > >> mounting,
> > > > >>>> DNS
> > > > >>>>>>>>> configuration, pod affinity/anti-affinity setting, etc.
> > > > >>>>>>>>>
> > > > >>>>>>>>> 2. Support running PyFlink on Kubernetes
> > > > >>>>>>>>> Description:
> > > > >>>>>>>>> Support running PyFlink on Kubernetes, including
> session
> > > > >>>>> cluster
> > > > >>>>>>> and
> > > > >>>>>>>>> application cluster.
> > > > >>>>>>>>> Benefits:
> > > > >>>>>>>>> Running python application in a containerized
> > > > >> environment.
> > > > >>>>>>>>> 3. Support built-in init-Container
> > > > >>>>>>>>> Description:
> > > > >>>>>>>>> We need a built-in init-Container to help solve
> > > > >> dependency
> > > > >>>>>>> management
> > > > >>>>>>>>> in a containerized environment, especially in the
> application
> > > > >>>> mode.
> > > > >>>>>>>>> Benefits:
> > > > >>>>>>>>> Separate the base Flink image from dynamic
> dependencies.
> > > > >>>>>>>>>
> > > > >>>>>>>>> 4. Support accessing secured services via K8s secrets
> > > > >>>>>>>>> Description:
> > > > >>>>>>>>> Kubernetes Secrets
> > > > >>>>>>>>> <https://kubernetes.io/docs/concepts/configuration/secret/
> >
> > > > >>> can
> > > > >>>> be
> > > > >>>>>>> used
> > > > >>>>>>>> to
> > > > >>>>>>>>> provide credentials for a Flink application to access
> secured
> > > > >>>>>> services.
> > > > >>>>>>>> It
> > > > >>>>>>>>> helps people who want to use a user-specified K8s Secret
> > > > >>> through
> > > > >>>> an
> > > > >>>>>>>>> environment variable.
> > > > >>>>>>>>> Benefits:
> > > > >>>>>>>>> Improve user experience.
> > > > >>>>>>>>>
> > > > >>>>>>>>> 5. Support configuring replica of JobManager Deployment in
> > > > >>>>> ZooKeeper
> > > > >>>>>> HA
> > > > >>>>>>>>> setups
> > > > >>>>>>>>> Description:
> > > > >>>>>>>>> Make the *replica* of Deployment configurable in the
> > > > >>>> ZooKeeper
> > > > >>>>> HA
> > > > >>>>>>>>> setups.
> > > > >>>>>>>>> Benefits:
> > > > >>>>>>>>> Achieve faster failover.
> > > > >>>>>>>>>
> > > > >>>>>>>>> 6. Support to configure limit for CPU requirement
> > > > >>>>>>>>> Description:
> > > > >>>>>>>>> To leverage the Kubernetes feature of container
> > > > >>> request/limit
> > > > >>>>>> CPU.
> > > > >>>>>>>>> Benefits:
> > > > >>>>>>>>> Reduce cost.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Regards,
> > > > >>>>>>>>> Canbin Zheng
> > > > >>>>>>>>>
> > > > >>>>>>>>> Harold.Miao <mi...@gmail.com> 于2020年7月23日周四
> 下午12:44写道：
> > > > >>>>>>>>>
> > > > >>>>>>>>>> I'm excited to hear about this feature, very, very, very
> > > > >>>> highly
> > > > >>>>>>>>> encouraged
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Prasanna kumar <pr...@gmail.com>
> > > > >> 于2020年7月23日周四
> > > > >>>>>>>> 上午12:10写道：
> > > > >>>>>>>>>>> Hi Flink Dev Team,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Dynamic AutoScaling Based on the incoming data load would
> > > > >>> be
> > > > >>>> a
> > > > >>>>>>> great
> > > > >>>>>>>>>>> feature.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> We should be able have some rule say If the load
> > > > >> increased
> > > > >>> by
> > > > >>>>>> 20% ,
> > > > >>>>>>>> add
> > > > >>>>>>>>>>> extra resource should be added.
> > > > >>>>>>>>>>> Or time based say during these peak hours the pipeline
> > > > >>> should
> > > > >>>>>> scale
> > > > >>>>>>>>>>> automatically by 50%.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> This will help a lot in cost reduction.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> EMR cluster provides a similar feature for SPARK based
> > > > >>>>>> application.
> > > > >>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>> Prasanna.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <
> > > > >>>>>>> rmetzger@apache.org>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Hi all,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Now that the 1.11 release is out, it is time to plan
> > > > >> for
> > > > >>>> the
> > > > >>>>>> next
> > > > >>>>>>>>> major
> > > > >>>>>>>>>>>> Flink release.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Some items:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> 1.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Dian Fu and me volunteer to be the release managers
> > > > >>> for
> > > > >>>>>> Flink
> > > > >>>>>>>>> 1.12.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> 1.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Timeline: We propose to stick to our approximate 4
> > > > >>> month
> > > > >>>>>>> release
> > > > >>>>>>>>>>> cycle,
> > > > >>>>>>>>>>>> thus the release should be done by late October.
> > > > >> Given
> > > > >>>>> that
> > > > >>>>>>>>> there’s
> > > > >>>>>>>>>> a
> > > > >>>>>>>>>>>> holiday week in China at the beginning of October, I
> > > > >>>>> propose
> > > > >>>>>>> to
> > > > >>>>>>>> do
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>>> feature freeze on master by late September.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> 2.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Collecting features: It would be good to have a
> > > > >> rough
> > > > >>>>>> overview
> > > > >>>>>>>> of
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>>> features that will likely be ready to be merged by
> > > > >>> late
> > > > >>>>>>>> September,
> > > > >>>>>>>>>> and
> > > > >>>>>>>>>>>> that
> > > > >>>>>>>>>>>> we want in the release.
> > > > >>>>>>>>>>>> Based on the discussion, we will update the Roadmap
> > > > >> on
> > > > >>>> the
> > > > >>>>>>> Flink
> > > > >>>>>>>>>>> website
> > > > >>>>>>>>>>>> again!
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> 1.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Test instabilities and blockers: I would like to
> > > > >>> avoid a
> > > > >>>>>>>> situation
> > > > >>>>>>>>>>> where
> > > > >>>>>>>>>>>> we have many blocking issues or build instabilities
> > > > >> at
> > > > >>>> the
> > > > >>>>>>> time
> > > > >>>>>>>> of
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>>> feature freeze. To achieve that, we will try to
> > > > >> check
> > > > >>>>> every
> > > > >>>>>>>> build
> > > > >>>>>>>>>>>> instability within a week, to decide if it is a
> > > > >>> blocker
> > > > >>>>>> (make
> > > > >>>>>>>> sure
> > > > >>>>>>>>>> to
> > > > >>>>>>>>>>>> use
> > > > >>>>>>>>>>>> the “test-stability” label for those tickets!)
> > > > >>>>>>>>>>>> Blocker issues will need to have somebody assigned
> > > > >>>>>>> (responsible)
> > > > >>>>>>>>>>> within
> > > > >>>>>>>>>>>> a week, and we want to see progress on all blocker
> > > > >>>> issues
> > > > >>>>>>>>>> (downgrade,
> > > > >>>>>>>>>>>> resolution, a good plan how to proceed if it is more
> > > > >>>>>>>> complicated)
> > > > >>>>>>>>>>>> 2.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Quality and stability of new features: In order to
> > > > >>> have
> > > > >>>> a
> > > > >>>>>>> short
> > > > >>>>>>>>>>> feature
> > > > >>>>>>>>>>>> freeze phase, we encourage developers to only merge
> > > > >>>>>>> well-tested
> > > > >>>>>>>>> and
> > > > >>>>>>>>>>>> documented features. In our experience, the feature
> > > > >>>> freeze
> > > > >>>>>>> works
> > > > >>>>>>>>>> best
> > > > >>>>>>>>>>> if
> > > > >>>>>>>>>>>> new features are complete, and the community can
> > > > >> focus
> > > > >>>>> fully
> > > > >>>>>>> on
> > > > >>>>>>>>>>>> addressing
> > > > >>>>>>>>>>>> newly found bugs and voting the release.
> > > > >>>>>>>>>>>> By having a smooth release process, the next
> > > > >>>> merge-window
> > > > >>>>>> for
> > > > >>>>>>>> the
> > > > >>>>>>>>>> next
> > > > >>>>>>>>>>>> release will come sooner.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Let me know what you think about our items, and share
> > > > >>> which
> > > > >>>>>>>> features
> > > > >>>>>>>>>> you
> > > > >>>>>>>>>>>> want in Flink 1.12.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Robert & Dian
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> --
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Best Regards,
> > > > >>>>>>>>>> Harold Miao
> > > > >>>>>>>>>>
> > > >
> > > >
> > >
> >
>
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Yun Tang <my...@live.com>.
+1 for extending the feature freeze due date.
________________________________
From: Zhijiang <wa...@aliyun.com.INVALID>
Sent: Thursday, August 6, 2020 17:05
To: dev <de...@flink.apache.org>
Subject: Re: [DISCUSS] Planning Flink 1.12
+1 on my side for feature freeze date by the end of Oct.
------------------------------------------------------------------
From:Yuan Mei <yu...@gmail.com>
Send Time:2020年8月6日(星期四) 14:54
To:dev <de...@flink.apache.org>
Subject:Re: [DISCUSS] Planning Flink 1.12
+1
> +1 for extending the feature freeze date to the end of October.
On Thu, Aug 6, 2020 at 12:08 PM Yu Li <ca...@gmail.com> wrote:
> +1 for extending feature freeze date to end of October.
>
> Feature development in the master branch could be unblocked through
> creating the release branch, but every coin has its two sides (smile)
>
> Best Regards,
> Yu
>
>
> On Wed, 5 Aug 2020 at 20:12, Robert Metzger <rm...@apache.org> wrote:
>
> > Thanks all for your opinion.
> >
> > @Chesnay: That is a risk, but I hope the people responsible for
> individual
> > FLIPs plan accordingly. Extending the time till the feature freeze should
> > not mean that we are extending the scope of the release.
> > Ideally, features are done before FF, and they use the time till the
> freeze
> > for additional testing and documentation polishing.
> > This FF will be virtual, there should be less disruption than a physical
> > conference with all the travelling.
> > Do you have a different proposal for the timing?
> >
> >
> > I'm currently considering splitting the feature freeze and the release
> > branch creation. Similar to the Linux kernel development, we could have a
> > "merge window" and a stabilization phase. At the end of the stabilization
> > phase, we cut the release branch and open the next merge window (I'll
> start
> > a separate thread regarding this towards the end of this release cycle,
> if
> > I still like the idea then)
> >
> >
> > On Wed, Aug 5, 2020 at 12:04 PM Chesnay Schepler <ch...@apache.org>
> > wrote:
> >
> > > I'm a bit concerned about end of October, because it means we have
> Flink
> > > forward, which usually means at least 1 week of little-to-no activity,
> > > and then 1 week until feature-freeze.
> > >
> > > On 05/08/2020 11:56, jincheng sun wrote:
> > > > +1 for end of October from me as well.
> > > >
> > > > Best,
> > > > Jincheng
> > > >
> > > >
> > > > Kostas Kloudas <kk...@gmail.com> 于2020年8月5日周三 下午4:59写道：
> > > >
> > > >> +1 for end of October from me as well.
> > > >>
> > > >> Cheers,
> > > >> Kostas
> > > >>
> > > >> On Wed, Aug 5, 2020 at 9:59 AM Till Rohrmann <tr...@apache.org>
> > > wrote:
> > > >>
> > > >>> +1 for end of October from my side as well.
> > > >>>
> > > >>> Cheers,
> > > >>> Till
> > > >>>
> > > >>> On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen <se...@apache.org>
> > wrote:
> > > >>>
> > > >>>> The end of October sounds good from my side, unless it collides
> with
> > > >> some
> > > >>>> holidays that affect many committers.
> > > >>>>
> > > >>>> Feature-wise, I believe we can definitely make good use of the
> time
> > to
> > > >>> wrap
> > > >>>> up some critical threads (like finishing the FLIP-27 source
> > efforts).
> > > >>>>
> > > >>>> So +1 to the end of October from my side.
> > > >>>>
> > > >>>> Best,
> > > >>>> Stephan
> > > >>>>
> > > >>>>
> > > >>>> On Tue, Aug 4, 2020 at 8:59 AM Robert Metzger <
> rmetzger@apache.org>
> > > >>> wrote:
> > > >>>>> Thanks a lot for commenting on the feature freeze date.
> > > >>>>>
> > > >>>>> You are raising a few good points on the timing.
> > > >>>>> If we have already (2 months before) concerns regarding the
> > deadline,
> > > >>>> then
> > > >>>>> I agree that we should move it till the end of October.
> > > >>>>>
> > > >>>>> We then just need to be careful not to run into the Christmas
> > season
> > > >> at
> > > >>>> the
> > > >>>>> end of December.
> > > >>>>>
> > > >>>>> If nobody objects within a few days, I'll update the feature
> freeze
> > > >>> date
> > > >>>> in
> > > >>>>> the Wiki.
> > > >>>>>
> > > >>>>>
> > > >>>>> On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <yk...@gmail.com>
> > wrote:
> > > >>>>>
> > > >>>>>> Regarding setting the feature freeze date to late September, I
> > have
> > > >>>> some
> > > >>>>>> concern that it might make
> > > >>>>>> the development time of 1.12 too short.
> > > >>>>>>
> > > >>>>>> One reason for this is we took too much time (about 1.5 month,
> > from
> > > >>> mid
> > > >>>>> of
> > > >>>>>> May to beginning of July)
> > > >>>>>> for testing 1.11. It's not ideal but further squeeze the
> > > >> development
> > > >>>> time
> > > >>>>>> of 1.12 won't make this better.
> > > >>>>>> Besides, AFAIK July & August is also a popular vacation season
> > for
> > > >>>>>> European. Given the fact most
> > > >>>>>> committers of Flink come from Europe, I think we should also
> > take
> > > >>> this
> > > >>>>>> into consideration.
> > > >>>>>>
> > > >>>>>> It's also true that the first week of October is the national
> > > >> holiday
> > > >>>> of
> > > >>>>>> China, so I'm wondering whether the
> > > >>>>>> end of October could be a candidate feature freeze date.
> > > >>>>>>
> > > >>>>>> Best,
> > > >>>>>> Kurt
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <
> > > >> rmetzger@apache.org>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hi all,
> > > >>>>>>>
> > > >>>>>>> Thanks a lot for the responses so far. I've put them into this
> > > >> Wiki
> > > >>>>> page:
> > > >>>>>>> https://cwiki.apache.org/confluence/display/FLINK/1.12+Release
> > > >> to
> > > >>>> keep
> > > >>>>>>> track of them. Ideally, post JIRA tickets for your feature,
> then
> > > >>> the
> > > >>>>>> status
> > > >>>>>>> will update automatically in the wiki :)
> > > >>>>>>>
> > > >>>>>>> Please keep posting features here, or add them to the Wiki
> > > >> yourself
> > > >>>> 🙏
> > > >>>>>>> @Prasanna kumar <pr...@gmail.com>: Dynamic Auto
> > > >>>> Scaling
> > > >>>>>> is a
> > > >>>>>>> feature request the community is well-aware of. Till has posted
> > > >>>>>>> "Reactive-scaling mode" as a feature he's working on for the
> 1.12
> > > >>>>>> release.
> > > >>>>>>> This work will introduce the basic building blocks and partial
> > > >>>> support
> > > >>>>>> for
> > > >>>>>>> the feature you are requesting.
> > > >>>>>>> Proper support for dynamic scaling, while maintaining Flink's
> > > >> high
> > > >>>>>>> performance (throughout, low latency) and correctness is a
> > > >>> difficult
> > > >>>>> task
> > > >>>>>>> that needs a lot of work. It will probably take a little bit of
> > > >>> time
> > > >>>>> till
> > > >>>>>>> this is fully available.
> > > >>>>>>>
> > > >>>>>>> Cheers,
> > > >>>>>>> Robert
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <
> > > >>> trohrmann@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Thanks for being our release managers for the 1.12 release
> > > >> Dian &
> > > >>>>>> Robert!
> > > >>>>>>>> Here are some features I would like to work on for this
> > > >> release:
> > > >>>>>>>> # Features
> > > >>>>>>>>
> > > >>>>>>>> ## Finishing pipelined region scheduling (
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-16430)
> > > >>>>>>>> With the pipelined region scheduler we want to implement a
> > > >>>> scheduler
> > > >>>>>>> which
> > > >>>>>>>> can serve streaming as well as batch workloads alike while
> > > >> being
> > > >>>> able
> > > >>>>>> to
> > > >>>>>>>> run jobs under constrained resources. The latter is
> > > >> particularly
> > > >>>>>>> important
> > > >>>>>>>> for bounded streaming jobs which, currently, are not well
> > > >>>> supported.
> > > >>>>>>>> ## Reactive-scaling mode
> > > >>>>>>>> Being able to react to newly available resources and rescaling
> > > >> a
> > > >>>>>> running
> > > >>>>>>>> job accordingly will make Flink's operation much easier
> because
> > > >>>>>> resources
> > > >>>>>>>> can then be controlled by an external tool (e.g. GCP
> > > >> autoscaling,
> > > >>>> K8s
> > > >>>>>>>> horizontal pod scaler, etc.). In this release we want to make
> a
> > > >>> big
> > > >>>>>> step
> > > >>>>>>>> towards this direction. As a first step we want to support the
> > > >>>>>> execution
> > > >>>>>>> of
> > > >>>>>>>> jobs with a parallelism which is lower than the specified
> > > >>>> parallelism
> > > >>>>>> in
> > > >>>>>>>> case that Flink lost a TaskManager or could not acquire enough
> > > >>>>>> resources.
> > > >>>>>>>> # Maintenance/Stability
> > > >>>>>>>>
> > > >>>>>>>> ## JM / TM finished task reconciliation (
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-17075)
> > > >>>>>>>> This prevents the system from going out of sync if a task
> state
> > > >>>>> change
> > > >>>>>>> from
> > > >>>>>>>> the TM to the JM is lost.
> > > >>>>>>>>
> > > >>>>>>>> ## Make metrics services work with Kubernetes deployments (
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-11127)
> > > >>>>>>>> Invert the direction in which the MetricFetcher connects to
> the
> > > >>>>>>>> MetricQueryFetchers. That way it will no longer be necessary
> to
> > > >>>>> expose
> > > >>>>>> on
> > > >>>>>>>> K8s for every TaskManager a port on which the
> > > >> MetricQueryFetcher
> > > >>>>> runs.
> > > >>>>>>> This
> > > >>>>>>>> will then make the deployment of Flink clusters on K8s easier.
> > > >>>>>>>>
> > > >>>>>>>> ## Handle long-blocking operations during job submission
> > > >>> (savepoint
> > > >>>>>>>> restore) (https://issues.apache.org/jira/browse/FLINK-16866)
> > > >>>>>>>> Submitting a Flink job can involve the interaction with
> > > >> external
> > > >>>>>> systems
> > > >>>>>>>> (blocking operations). Depending on the job the interactions
> > > >> can
> > > >>>> take
> > > >>>>>> so
> > > >>>>>>>> long that it exceeds the submission timeout which reports a
> > > >>> failure
> > > >>>>> on
> > > >>>>>>> the
> > > >>>>>>>> client side even though the actual submission succeeded. By
> > > >>>>> decoupling
> > > >>>>>>> the
> > > >>>>>>>> creation of the ExecutionGraph from the job submission, we can
> > > >>> make
> > > >>>>> the
> > > >>>>>>> job
> > > >>>>>>>> submission non-blocking which will solve this problem.
> > > >>>>>>>>
> > > >>>>>>>> ## Make IDs more intuitive to ease debugging (FLIP-118) (
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-15679)
> > > >>>>>>>> By making the internal Flink IDs compositional or logging how
> > > >>> they
> > > >>>>>> belong
> > > >>>>>>>> together, we can make the debugging of Flink's operations much
> > > >>>>> easier.
> > > >>>>>>>> Cheers,
> > > >>>>>>>> Till
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <
> > > >>>> felixzhengcb@gmail.com
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi All,
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks for bring-up this discussion, Robert!
> > > >>>>>>>>> Congratulations on becoming the release manager of 1.12， Dian
> > > >>> and
> > > >>>>>>> Robert
> > > >>>>>>>> !
> > > >>>>>>>>> ----------
> > > >>>>>>>>> Here are some of my thoughts of the features for native
> > > >>>> integration
> > > >>>>>>> with
> > > >>>>>>>>> Kubernetes in Flink 1.12:
> > > >>>>>>>>>
> > > >>>>>>>>> 1. Support user-specified pod templates
> > > >>>>>>>>> Description:
> > > >>>>>>>>> The current approach of introducing new configuration
> > > >>> options
> > > >>>>> for
> > > >>>>>>>> each
> > > >>>>>>>>> aspect of pod specification a user might wish is becoming
> > > >>>> unwieldy,
> > > >>>>>> we
> > > >>>>>>>> have
> > > >>>>>>>>> to maintain more and more Flink side Kubernetes configuration
> > > >>>>> options
> > > >>>>>>> and
> > > >>>>>>>>> users have to learn the gap between the declarative model
> > > >> used
> > > >>> by
> > > >>>>>>>>> Kubernetes and the configuration model used by Flink. It's a
> > > >>>> great
> > > >>>>>>>>> improvement to allow users to specify pod templates as
> > > >> central
> > > >>>>> places
> > > >>>>>>> for
> > > >>>>>>>>> all customization needs for the jobmanager and taskmanager
> > > >>> pods.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Users can leverage many of the advanced K8s features
> that
> > > >>> the
> > > >>>>>> Flink
> > > >>>>>>>>> community does not support explicitly, such as volume
> > > >> mounting,
> > > >>>> DNS
> > > >>>>>>>>> configuration, pod affinity/anti-affinity setting, etc.
> > > >>>>>>>>>
> > > >>>>>>>>> 2. Support running PyFlink on Kubernetes
> > > >>>>>>>>> Description:
> > > >>>>>>>>> Support running PyFlink on Kubernetes, including session
> > > >>>>> cluster
> > > >>>>>>> and
> > > >>>>>>>>> application cluster.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Running python application in a containerized
> > > >> environment.
> > > >>>>>>>>> 3. Support built-in init-Container
> > > >>>>>>>>> Description:
> > > >>>>>>>>> We need a built-in init-Container to help solve
> > > >> dependency
> > > >>>>>>> management
> > > >>>>>>>>> in a containerized environment, especially in the application
> > > >>>> mode.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Separate the base Flink image from dynamic dependencies.
> > > >>>>>>>>>
> > > >>>>>>>>> 4. Support accessing secured services via K8s secrets
> > > >>>>>>>>> Description:
> > > >>>>>>>>> Kubernetes Secrets
> > > >>>>>>>>> <https://kubernetes.io/docs/concepts/configuration/secret/>
> > > >>> can
> > > >>>> be
> > > >>>>>>> used
> > > >>>>>>>> to
> > > >>>>>>>>> provide credentials for a Flink application to access secured
> > > >>>>>> services.
> > > >>>>>>>> It
> > > >>>>>>>>> helps people who want to use a user-specified K8s Secret
> > > >>> through
> > > >>>> an
> > > >>>>>>>>> environment variable.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Improve user experience.
> > > >>>>>>>>>
> > > >>>>>>>>> 5. Support configuring replica of JobManager Deployment in
> > > >>>>> ZooKeeper
> > > >>>>>> HA
> > > >>>>>>>>> setups
> > > >>>>>>>>> Description:
> > > >>>>>>>>> Make the *replica* of Deployment configurable in the
> > > >>>> ZooKeeper
> > > >>>>> HA
> > > >>>>>>>>> setups.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Achieve faster failover.
> > > >>>>>>>>>
> > > >>>>>>>>> 6. Support to configure limit for CPU requirement
> > > >>>>>>>>> Description:
> > > >>>>>>>>> To leverage the Kubernetes feature of container
> > > >>> request/limit
> > > >>>>>> CPU.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Reduce cost.
> > > >>>>>>>>>
> > > >>>>>>>>> Regards,
> > > >>>>>>>>> Canbin Zheng
> > > >>>>>>>>>
> > > >>>>>>>>> Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
> > > >>>>>>>>>
> > > >>>>>>>>>> I'm excited to hear about this feature, very, very, very
> > > >>>> highly
> > > >>>>>>>>> encouraged
> > > >>>>>>>>>>
> > > >>>>>>>>>> Prasanna kumar <pr...@gmail.com>
> > > >> 于2020年7月23日周四
> > > >>>>>>>> 上午12:10写道：
> > > >>>>>>>>>>> Hi Flink Dev Team,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Dynamic AutoScaling Based on the incoming data load would
> > > >>> be
> > > >>>> a
> > > >>>>>>> great
> > > >>>>>>>>>>> feature.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> We should be able have some rule say If the load
> > > >> increased
> > > >>> by
> > > >>>>>> 20% ,
> > > >>>>>>>> add
> > > >>>>>>>>>>> extra resource should be added.
> > > >>>>>>>>>>> Or time based say during these peak hours the pipeline
> > > >>> should
> > > >>>>>> scale
> > > >>>>>>>>>>> automatically by 50%.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> This will help a lot in cost reduction.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> EMR cluster provides a similar feature for SPARK based
> > > >>>>>> application.
> > > >>>>>>>>>>> Thanks,
> > > >>>>>>>>>>> Prasanna.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <
> > > >>>>>>> rmetzger@apache.org>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Now that the 1.11 release is out, it is time to plan
> > > >> for
> > > >>>> the
> > > >>>>>> next
> > > >>>>>>>>> major
> > > >>>>>>>>>>>> Flink release.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Some items:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Dian Fu and me volunteer to be the release managers
> > > >>> for
> > > >>>>>> Flink
> > > >>>>>>>>> 1.12.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Timeline: We propose to stick to our approximate 4
> > > >>> month
> > > >>>>>>> release
> > > >>>>>>>>>>> cycle,
> > > >>>>>>>>>>>> thus the release should be done by late October.
> > > >> Given
> > > >>>>> that
> > > >>>>>>>>> there’s
> > > >>>>>>>>>> a
> > > >>>>>>>>>>>> holiday week in China at the beginning of October, I
> > > >>>>> propose
> > > >>>>>>> to
> > > >>>>>>>> do
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> feature freeze on master by late September.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 2.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Collecting features: It would be good to have a
> > > >> rough
> > > >>>>>> overview
> > > >>>>>>>> of
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> features that will likely be ready to be merged by
> > > >>> late
> > > >>>>>>>> September,
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>> that
> > > >>>>>>>>>>>> we want in the release.
> > > >>>>>>>>>>>> Based on the discussion, we will update the Roadmap
> > > >> on
> > > >>>> the
> > > >>>>>>> Flink
> > > >>>>>>>>>>> website
> > > >>>>>>>>>>>> again!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Test instabilities and blockers: I would like to
> > > >>> avoid a
> > > >>>>>>>> situation
> > > >>>>>>>>>>> where
> > > >>>>>>>>>>>> we have many blocking issues or build instabilities
> > > >> at
> > > >>>> the
> > > >>>>>>> time
> > > >>>>>>>> of
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> feature freeze. To achieve that, we will try to
> > > >> check
> > > >>>>> every
> > > >>>>>>>> build
> > > >>>>>>>>>>>> instability within a week, to decide if it is a
> > > >>> blocker
> > > >>>>>> (make
> > > >>>>>>>> sure
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>> use
> > > >>>>>>>>>>>> the “test-stability” label for those tickets!)
> > > >>>>>>>>>>>> Blocker issues will need to have somebody assigned
> > > >>>>>>> (responsible)
> > > >>>>>>>>>>> within
> > > >>>>>>>>>>>> a week, and we want to see progress on all blocker
> > > >>>> issues
> > > >>>>>>>>>> (downgrade,
> > > >>>>>>>>>>>> resolution, a good plan how to proceed if it is more
> > > >>>>>>>> complicated)
> > > >>>>>>>>>>>> 2.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Quality and stability of new features: In order to
> > > >>> have
> > > >>>> a
> > > >>>>>>> short
> > > >>>>>>>>>>> feature
> > > >>>>>>>>>>>> freeze phase, we encourage developers to only merge
> > > >>>>>>> well-tested
> > > >>>>>>>>> and
> > > >>>>>>>>>>>> documented features. In our experience, the feature
> > > >>>> freeze
> > > >>>>>>> works
> > > >>>>>>>>>> best
> > > >>>>>>>>>>> if
> > > >>>>>>>>>>>> new features are complete, and the community can
> > > >> focus
> > > >>>>> fully
> > > >>>>>>> on
> > > >>>>>>>>>>>> addressing
> > > >>>>>>>>>>>> newly found bugs and voting the release.
> > > >>>>>>>>>>>> By having a smooth release process, the next
> > > >>>> merge-window
> > > >>>>>> for
> > > >>>>>>>> the
> > > >>>>>>>>>> next
> > > >>>>>>>>>>>> release will come sooner.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Let me know what you think about our items, and share
> > > >>> which
> > > >>>>>>>> features
> > > >>>>>>>>>> you
> > > >>>>>>>>>>>> want in Flink 1.12.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Robert & Dian
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> --
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best Regards,
> > > >>>>>>>>>> Harold Miao
> > > >>>>>>>>>>
> > >
> > >
> >
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Zhijiang <wa...@aliyun.com.INVALID>.
+1 on my side for feature freeze date by the end of Oct.
------------------------------------------------------------------
From:Yuan Mei <yu...@gmail.com>
Send Time:2020年8月6日(星期四) 14:54
To:dev <de...@flink.apache.org>
Subject:Re: [DISCUSS] Planning Flink 1.12
+1
> +1 for extending the feature freeze date to the end of October.
On Thu, Aug 6, 2020 at 12:08 PM Yu Li <ca...@gmail.com> wrote:
> +1 for extending feature freeze date to end of October.
>
> Feature development in the master branch could be unblocked through
> creating the release branch, but every coin has its two sides (smile)
>
> Best Regards,
> Yu
>
>
> On Wed, 5 Aug 2020 at 20:12, Robert Metzger <rm...@apache.org> wrote:
>
> > Thanks all for your opinion.
> >
> > @Chesnay: That is a risk, but I hope the people responsible for
> individual
> > FLIPs plan accordingly. Extending the time till the feature freeze should
> > not mean that we are extending the scope of the release.
> > Ideally, features are done before FF, and they use the time till the
> freeze
> > for additional testing and documentation polishing.
> > This FF will be virtual, there should be less disruption than a physical
> > conference with all the travelling.
> > Do you have a different proposal for the timing?
> >
> >
> > I'm currently considering splitting the feature freeze and the release
> > branch creation. Similar to the Linux kernel development, we could have a
> > "merge window" and a stabilization phase. At the end of the stabilization
> > phase, we cut the release branch and open the next merge window (I'll
> start
> > a separate thread regarding this towards the end of this release cycle,
> if
> > I still like the idea then)
> >
> >
> > On Wed, Aug 5, 2020 at 12:04 PM Chesnay Schepler <ch...@apache.org>
> > wrote:
> >
> > > I'm a bit concerned about end of October, because it means we have
> Flink
> > > forward, which usually means at least 1 week of little-to-no activity,
> > > and then 1 week until feature-freeze.
> > >
> > > On 05/08/2020 11:56, jincheng sun wrote:
> > > > +1 for end of October from me as well.
> > > >
> > > > Best,
> > > > Jincheng
> > > >
> > > >
> > > > Kostas Kloudas <kk...@gmail.com> 于2020年8月5日周三 下午4:59写道：
> > > >
> > > >> +1 for end of October from me as well.
> > > >>
> > > >> Cheers,
> > > >> Kostas
> > > >>
> > > >> On Wed, Aug 5, 2020 at 9:59 AM Till Rohrmann <tr...@apache.org>
> > > wrote:
> > > >>
> > > >>> +1 for end of October from my side as well.
> > > >>>
> > > >>> Cheers,
> > > >>> Till
> > > >>>
> > > >>> On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen <se...@apache.org>
> > wrote:
> > > >>>
> > > >>>> The end of October sounds good from my side, unless it collides
> with
> > > >> some
> > > >>>> holidays that affect many committers.
> > > >>>>
> > > >>>> Feature-wise, I believe we can definitely make good use of the
> time
> > to
> > > >>> wrap
> > > >>>> up some critical threads (like finishing the FLIP-27 source
> > efforts).
> > > >>>>
> > > >>>> So +1 to the end of October from my side.
> > > >>>>
> > > >>>> Best,
> > > >>>> Stephan
> > > >>>>
> > > >>>>
> > > >>>> On Tue, Aug 4, 2020 at 8:59 AM Robert Metzger <
> rmetzger@apache.org>
> > > >>> wrote:
> > > >>>>> Thanks a lot for commenting on the feature freeze date.
> > > >>>>>
> > > >>>>> You are raising a few good points on the timing.
> > > >>>>> If we have already (2 months before) concerns regarding the
> > deadline,
> > > >>>> then
> > > >>>>> I agree that we should move it till the end of October.
> > > >>>>>
> > > >>>>> We then just need to be careful not to run into the Christmas
> > season
> > > >> at
> > > >>>> the
> > > >>>>> end of December.
> > > >>>>>
> > > >>>>> If nobody objects within a few days, I'll update the feature
> freeze
> > > >>> date
> > > >>>> in
> > > >>>>> the Wiki.
> > > >>>>>
> > > >>>>>
> > > >>>>> On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <yk...@gmail.com>
> > wrote:
> > > >>>>>
> > > >>>>>> Regarding setting the feature freeze date to late September, I
> > have
> > > >>>> some
> > > >>>>>> concern that it might make
> > > >>>>>> the development time of 1.12 too short.
> > > >>>>>>
> > > >>>>>> One reason for this is we took too much time (about 1.5 month,
> > from
> > > >>> mid
> > > >>>>> of
> > > >>>>>> May to beginning of July)
> > > >>>>>> for testing 1.11. It's not ideal but further squeeze the
> > > >> development
> > > >>>> time
> > > >>>>>> of 1.12 won't make this better.
> > > >>>>>> Besides, AFAIK July & August is also a popular vacation season
> > for
> > > >>>>>> European. Given the fact most
> > > >>>>>> committers of Flink come from Europe, I think we should also
> > take
> > > >>> this
> > > >>>>>> into consideration.
> > > >>>>>>
> > > >>>>>> It's also true that the first week of October is the national
> > > >> holiday
> > > >>>> of
> > > >>>>>> China, so I'm wondering whether the
> > > >>>>>> end of October could be a candidate feature freeze date.
> > > >>>>>>
> > > >>>>>> Best,
> > > >>>>>> Kurt
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <
> > > >> rmetzger@apache.org>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hi all,
> > > >>>>>>>
> > > >>>>>>> Thanks a lot for the responses so far. I've put them into this
> > > >> Wiki
> > > >>>>> page:
> > > >>>>>>> https://cwiki.apache.org/confluence/display/FLINK/1.12+Release
> > > >> to
> > > >>>> keep
> > > >>>>>>> track of them. Ideally, post JIRA tickets for your feature,
> then
> > > >>> the
> > > >>>>>> status
> > > >>>>>>> will update automatically in the wiki :)
> > > >>>>>>>
> > > >>>>>>> Please keep posting features here, or add them to the Wiki
> > > >> yourself
> > > >>>> 🙏
> > > >>>>>>> @Prasanna kumar <pr...@gmail.com>: Dynamic Auto
> > > >>>> Scaling
> > > >>>>>> is a
> > > >>>>>>> feature request the community is well-aware of. Till has posted
> > > >>>>>>> "Reactive-scaling mode" as a feature he's working on for the
> 1.12
> > > >>>>>> release.
> > > >>>>>>> This work will introduce the basic building blocks and partial
> > > >>>> support
> > > >>>>>> for
> > > >>>>>>> the feature you are requesting.
> > > >>>>>>> Proper support for dynamic scaling, while maintaining Flink's
> > > >> high
> > > >>>>>>> performance (throughout, low latency) and correctness is a
> > > >>> difficult
> > > >>>>> task
> > > >>>>>>> that needs a lot of work. It will probably take a little bit of
> > > >>> time
> > > >>>>> till
> > > >>>>>>> this is fully available.
> > > >>>>>>>
> > > >>>>>>> Cheers,
> > > >>>>>>> Robert
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <
> > > >>> trohrmann@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Thanks for being our release managers for the 1.12 release
> > > >> Dian &
> > > >>>>>> Robert!
> > > >>>>>>>> Here are some features I would like to work on for this
> > > >> release:
> > > >>>>>>>> # Features
> > > >>>>>>>>
> > > >>>>>>>> ## Finishing pipelined region scheduling (
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-16430)
> > > >>>>>>>> With the pipelined region scheduler we want to implement a
> > > >>>> scheduler
> > > >>>>>>> which
> > > >>>>>>>> can serve streaming as well as batch workloads alike while
> > > >> being
> > > >>>> able
> > > >>>>>> to
> > > >>>>>>>> run jobs under constrained resources. The latter is
> > > >> particularly
> > > >>>>>>> important
> > > >>>>>>>> for bounded streaming jobs which, currently, are not well
> > > >>>> supported.
> > > >>>>>>>> ## Reactive-scaling mode
> > > >>>>>>>> Being able to react to newly available resources and rescaling
> > > >> a
> > > >>>>>> running
> > > >>>>>>>> job accordingly will make Flink's operation much easier
> because
> > > >>>>>> resources
> > > >>>>>>>> can then be controlled by an external tool (e.g. GCP
> > > >> autoscaling,
> > > >>>> K8s
> > > >>>>>>>> horizontal pod scaler, etc.). In this release we want to make
> a
> > > >>> big
> > > >>>>>> step
> > > >>>>>>>> towards this direction. As a first step we want to support the
> > > >>>>>> execution
> > > >>>>>>> of
> > > >>>>>>>> jobs with a parallelism which is lower than the specified
> > > >>>> parallelism
> > > >>>>>> in
> > > >>>>>>>> case that Flink lost a TaskManager or could not acquire enough
> > > >>>>>> resources.
> > > >>>>>>>> # Maintenance/Stability
> > > >>>>>>>>
> > > >>>>>>>> ## JM / TM finished task reconciliation (
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-17075)
> > > >>>>>>>> This prevents the system from going out of sync if a task
> state
> > > >>>>> change
> > > >>>>>>> from
> > > >>>>>>>> the TM to the JM is lost.
> > > >>>>>>>>
> > > >>>>>>>> ## Make metrics services work with Kubernetes deployments (
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-11127)
> > > >>>>>>>> Invert the direction in which the MetricFetcher connects to
> the
> > > >>>>>>>> MetricQueryFetchers. That way it will no longer be necessary
> to
> > > >>>>> expose
> > > >>>>>> on
> > > >>>>>>>> K8s for every TaskManager a port on which the
> > > >> MetricQueryFetcher
> > > >>>>> runs.
> > > >>>>>>> This
> > > >>>>>>>> will then make the deployment of Flink clusters on K8s easier.
> > > >>>>>>>>
> > > >>>>>>>> ## Handle long-blocking operations during job submission
> > > >>> (savepoint
> > > >>>>>>>> restore) (https://issues.apache.org/jira/browse/FLINK-16866)
> > > >>>>>>>> Submitting a Flink job can involve the interaction with
> > > >> external
> > > >>>>>> systems
> > > >>>>>>>> (blocking operations). Depending on the job the interactions
> > > >> can
> > > >>>> take
> > > >>>>>> so
> > > >>>>>>>> long that it exceeds the submission timeout which reports a
> > > >>> failure
> > > >>>>> on
> > > >>>>>>> the
> > > >>>>>>>> client side even though the actual submission succeeded. By
> > > >>>>> decoupling
> > > >>>>>>> the
> > > >>>>>>>> creation of the ExecutionGraph from the job submission, we can
> > > >>> make
> > > >>>>> the
> > > >>>>>>> job
> > > >>>>>>>> submission non-blocking which will solve this problem.
> > > >>>>>>>>
> > > >>>>>>>> ## Make IDs more intuitive to ease debugging (FLIP-118) (
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-15679)
> > > >>>>>>>> By making the internal Flink IDs compositional or logging how
> > > >>> they
> > > >>>>>> belong
> > > >>>>>>>> together, we can make the debugging of Flink's operations much
> > > >>>>> easier.
> > > >>>>>>>> Cheers,
> > > >>>>>>>> Till
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <
> > > >>>> felixzhengcb@gmail.com
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi All,
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks for bring-up this discussion, Robert!
> > > >>>>>>>>> Congratulations on becoming the release manager of 1.12， Dian
> > > >>> and
> > > >>>>>>> Robert
> > > >>>>>>>> !
> > > >>>>>>>>> ----------
> > > >>>>>>>>> Here are some of my thoughts of the features for native
> > > >>>> integration
> > > >>>>>>> with
> > > >>>>>>>>> Kubernetes in Flink 1.12:
> > > >>>>>>>>>
> > > >>>>>>>>> 1. Support user-specified pod templates
> > > >>>>>>>>> Description:
> > > >>>>>>>>> The current approach of introducing new configuration
> > > >>> options
> > > >>>>> for
> > > >>>>>>>> each
> > > >>>>>>>>> aspect of pod specification a user might wish is becoming
> > > >>>> unwieldy,
> > > >>>>>> we
> > > >>>>>>>> have
> > > >>>>>>>>> to maintain more and more Flink side Kubernetes configuration
> > > >>>>> options
> > > >>>>>>> and
> > > >>>>>>>>> users have to learn the gap between the declarative model
> > > >> used
> > > >>> by
> > > >>>>>>>>> Kubernetes and the configuration model used by Flink. It's a
> > > >>>> great
> > > >>>>>>>>> improvement to allow users to specify pod templates as
> > > >> central
> > > >>>>> places
> > > >>>>>>> for
> > > >>>>>>>>> all customization needs for the jobmanager and taskmanager
> > > >>> pods.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Users can leverage many of the advanced K8s features
> that
> > > >>> the
> > > >>>>>> Flink
> > > >>>>>>>>> community does not support explicitly, such as volume
> > > >> mounting,
> > > >>>> DNS
> > > >>>>>>>>> configuration, pod affinity/anti-affinity setting, etc.
> > > >>>>>>>>>
> > > >>>>>>>>> 2. Support running PyFlink on Kubernetes
> > > >>>>>>>>> Description:
> > > >>>>>>>>> Support running PyFlink on Kubernetes, including session
> > > >>>>> cluster
> > > >>>>>>> and
> > > >>>>>>>>> application cluster.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Running python application in a containerized
> > > >> environment.
> > > >>>>>>>>> 3. Support built-in init-Container
> > > >>>>>>>>> Description:
> > > >>>>>>>>> We need a built-in init-Container to help solve
> > > >> dependency
> > > >>>>>>> management
> > > >>>>>>>>> in a containerized environment, especially in the application
> > > >>>> mode.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Separate the base Flink image from dynamic dependencies.
> > > >>>>>>>>>
> > > >>>>>>>>> 4. Support accessing secured services via K8s secrets
> > > >>>>>>>>> Description:
> > > >>>>>>>>> Kubernetes Secrets
> > > >>>>>>>>> <https://kubernetes.io/docs/concepts/configuration/secret/>
> > > >>> can
> > > >>>> be
> > > >>>>>>> used
> > > >>>>>>>> to
> > > >>>>>>>>> provide credentials for a Flink application to access secured
> > > >>>>>> services.
> > > >>>>>>>> It
> > > >>>>>>>>> helps people who want to use a user-specified K8s Secret
> > > >>> through
> > > >>>> an
> > > >>>>>>>>> environment variable.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Improve user experience.
> > > >>>>>>>>>
> > > >>>>>>>>> 5. Support configuring replica of JobManager Deployment in
> > > >>>>> ZooKeeper
> > > >>>>>> HA
> > > >>>>>>>>> setups
> > > >>>>>>>>> Description:
> > > >>>>>>>>> Make the *replica* of Deployment configurable in the
> > > >>>> ZooKeeper
> > > >>>>> HA
> > > >>>>>>>>> setups.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Achieve faster failover.
> > > >>>>>>>>>
> > > >>>>>>>>> 6. Support to configure limit for CPU requirement
> > > >>>>>>>>> Description:
> > > >>>>>>>>> To leverage the Kubernetes feature of container
> > > >>> request/limit
> > > >>>>>> CPU.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Reduce cost.
> > > >>>>>>>>>
> > > >>>>>>>>> Regards,
> > > >>>>>>>>> Canbin Zheng
> > > >>>>>>>>>
> > > >>>>>>>>> Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
> > > >>>>>>>>>
> > > >>>>>>>>>> I'm excited to hear about this feature, very, very, very
> > > >>>> highly
> > > >>>>>>>>> encouraged
> > > >>>>>>>>>>
> > > >>>>>>>>>> Prasanna kumar <pr...@gmail.com>
> > > >> 于2020年7月23日周四
> > > >>>>>>>> 上午12:10写道：
> > > >>>>>>>>>>> Hi Flink Dev Team,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Dynamic AutoScaling Based on the incoming data load would
> > > >>> be
> > > >>>> a
> > > >>>>>>> great
> > > >>>>>>>>>>> feature.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> We should be able have some rule say If the load
> > > >> increased
> > > >>> by
> > > >>>>>> 20% ,
> > > >>>>>>>> add
> > > >>>>>>>>>>> extra resource should be added.
> > > >>>>>>>>>>> Or time based say during these peak hours the pipeline
> > > >>> should
> > > >>>>>> scale
> > > >>>>>>>>>>> automatically by 50%.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> This will help a lot in cost reduction.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> EMR cluster provides a similar feature for SPARK based
> > > >>>>>> application.
> > > >>>>>>>>>>> Thanks,
> > > >>>>>>>>>>> Prasanna.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <
> > > >>>>>>> rmetzger@apache.org>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Now that the 1.11 release is out, it is time to plan
> > > >> for
> > > >>>> the
> > > >>>>>> next
> > > >>>>>>>>> major
> > > >>>>>>>>>>>> Flink release.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Some items:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Dian Fu and me volunteer to be the release managers
> > > >>> for
> > > >>>>>> Flink
> > > >>>>>>>>> 1.12.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Timeline: We propose to stick to our approximate 4
> > > >>> month
> > > >>>>>>> release
> > > >>>>>>>>>>> cycle,
> > > >>>>>>>>>>>> thus the release should be done by late October.
> > > >> Given
> > > >>>>> that
> > > >>>>>>>>> there’s
> > > >>>>>>>>>> a
> > > >>>>>>>>>>>> holiday week in China at the beginning of October, I
> > > >>>>> propose
> > > >>>>>>> to
> > > >>>>>>>> do
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> feature freeze on master by late September.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 2.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Collecting features: It would be good to have a
> > > >> rough
> > > >>>>>> overview
> > > >>>>>>>> of
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> features that will likely be ready to be merged by
> > > >>> late
> > > >>>>>>>> September,
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>> that
> > > >>>>>>>>>>>> we want in the release.
> > > >>>>>>>>>>>> Based on the discussion, we will update the Roadmap
> > > >> on
> > > >>>> the
> > > >>>>>>> Flink
> > > >>>>>>>>>>> website
> > > >>>>>>>>>>>> again!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Test instabilities and blockers: I would like to
> > > >>> avoid a
> > > >>>>>>>> situation
> > > >>>>>>>>>>> where
> > > >>>>>>>>>>>> we have many blocking issues or build instabilities
> > > >> at
> > > >>>> the
> > > >>>>>>> time
> > > >>>>>>>> of
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> feature freeze. To achieve that, we will try to
> > > >> check
> > > >>>>> every
> > > >>>>>>>> build
> > > >>>>>>>>>>>> instability within a week, to decide if it is a
> > > >>> blocker
> > > >>>>>> (make
> > > >>>>>>>> sure
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>> use
> > > >>>>>>>>>>>> the “test-stability” label for those tickets!)
> > > >>>>>>>>>>>> Blocker issues will need to have somebody assigned
> > > >>>>>>> (responsible)
> > > >>>>>>>>>>> within
> > > >>>>>>>>>>>> a week, and we want to see progress on all blocker
> > > >>>> issues
> > > >>>>>>>>>> (downgrade,
> > > >>>>>>>>>>>> resolution, a good plan how to proceed if it is more
> > > >>>>>>>> complicated)
> > > >>>>>>>>>>>> 2.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Quality and stability of new features: In order to
> > > >>> have
> > > >>>> a
> > > >>>>>>> short
> > > >>>>>>>>>>> feature
> > > >>>>>>>>>>>> freeze phase, we encourage developers to only merge
> > > >>>>>>> well-tested
> > > >>>>>>>>> and
> > > >>>>>>>>>>>> documented features. In our experience, the feature
> > > >>>> freeze
> > > >>>>>>> works
> > > >>>>>>>>>> best
> > > >>>>>>>>>>> if
> > > >>>>>>>>>>>> new features are complete, and the community can
> > > >> focus
> > > >>>>> fully
> > > >>>>>>> on
> > > >>>>>>>>>>>> addressing
> > > >>>>>>>>>>>> newly found bugs and voting the release.
> > > >>>>>>>>>>>> By having a smooth release process, the next
> > > >>>> merge-window
> > > >>>>>> for
> > > >>>>>>>> the
> > > >>>>>>>>>> next
> > > >>>>>>>>>>>> release will come sooner.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Let me know what you think about our items, and share
> > > >>> which
> > > >>>>>>>> features
> > > >>>>>>>>>> you
> > > >>>>>>>>>>>> want in Flink 1.12.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Robert & Dian
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> --
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best Regards,
> > > >>>>>>>>>> Harold Miao
> > > >>>>>>>>>>
> > >
> > >
> >
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Yuan Mei <yu...@gmail.com>.
+1
> +1 for extending the feature freeze date to the end of October.
On Thu, Aug 6, 2020 at 12:08 PM Yu Li <ca...@gmail.com> wrote:
> +1 for extending feature freeze date to end of October.
>
> Feature development in the master branch could be unblocked through
> creating the release branch, but every coin has its two sides (smile)
>
> Best Regards,
> Yu
>
>
> On Wed, 5 Aug 2020 at 20:12, Robert Metzger <rm...@apache.org> wrote:
>
> > Thanks all for your opinion.
> >
> > @Chesnay: That is a risk, but I hope the people responsible for
> individual
> > FLIPs plan accordingly. Extending the time till the feature freeze should
> > not mean that we are extending the scope of the release.
> > Ideally, features are done before FF, and they use the time till the
> freeze
> > for additional testing and documentation polishing.
> > This FF will be virtual, there should be less disruption than a physical
> > conference with all the travelling.
> > Do you have a different proposal for the timing?
> >
> >
> > I'm currently considering splitting the feature freeze and the release
> > branch creation. Similar to the Linux kernel development, we could have a
> > "merge window" and a stabilization phase. At the end of the stabilization
> > phase, we cut the release branch and open the next merge window (I'll
> start
> > a separate thread regarding this towards the end of this release cycle,
> if
> > I still like the idea then)
> >
> >
> > On Wed, Aug 5, 2020 at 12:04 PM Chesnay Schepler <ch...@apache.org>
> > wrote:
> >
> > > I'm a bit concerned about end of October, because it means we have
> Flink
> > > forward, which usually means at least 1 week of little-to-no activity,
> > > and then 1 week until feature-freeze.
> > >
> > > On 05/08/2020 11:56, jincheng sun wrote:
> > > > +1 for end of October from me as well.
> > > >
> > > > Best,
> > > > Jincheng
> > > >
> > > >
> > > > Kostas Kloudas <kk...@gmail.com> 于2020年8月5日周三 下午4:59写道：
> > > >
> > > >> +1 for end of October from me as well.
> > > >>
> > > >> Cheers,
> > > >> Kostas
> > > >>
> > > >> On Wed, Aug 5, 2020 at 9:59 AM Till Rohrmann <tr...@apache.org>
> > > wrote:
> > > >>
> > > >>> +1 for end of October from my side as well.
> > > >>>
> > > >>> Cheers,
> > > >>> Till
> > > >>>
> > > >>> On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen <se...@apache.org>
> > wrote:
> > > >>>
> > > >>>> The end of October sounds good from my side, unless it collides
> with
> > > >> some
> > > >>>> holidays that affect many committers.
> > > >>>>
> > > >>>> Feature-wise, I believe we can definitely make good use of the
> time
> > to
> > > >>> wrap
> > > >>>> up some critical threads (like finishing the FLIP-27 source
> > efforts).
> > > >>>>
> > > >>>> So +1 to the end of October from my side.
> > > >>>>
> > > >>>> Best,
> > > >>>> Stephan
> > > >>>>
> > > >>>>
> > > >>>> On Tue, Aug 4, 2020 at 8:59 AM Robert Metzger <
> rmetzger@apache.org>
> > > >>> wrote:
> > > >>>>> Thanks a lot for commenting on the feature freeze date.
> > > >>>>>
> > > >>>>> You are raising a few good points on the timing.
> > > >>>>> If we have already (2 months before) concerns regarding the
> > deadline,
> > > >>>> then
> > > >>>>> I agree that we should move it till the end of October.
> > > >>>>>
> > > >>>>> We then just need to be careful not to run into the Christmas
> > season
> > > >> at
> > > >>>> the
> > > >>>>> end of December.
> > > >>>>>
> > > >>>>> If nobody objects within a few days, I'll update the feature
> freeze
> > > >>> date
> > > >>>> in
> > > >>>>> the Wiki.
> > > >>>>>
> > > >>>>>
> > > >>>>> On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <yk...@gmail.com>
> > wrote:
> > > >>>>>
> > > >>>>>> Regarding setting the feature freeze date to late September, I
> > have
> > > >>>> some
> > > >>>>>> concern that it might make
> > > >>>>>> the development time of 1.12 too short.
> > > >>>>>>
> > > >>>>>> One reason for this is we took too much time (about 1.5 month,
> > from
> > > >>> mid
> > > >>>>> of
> > > >>>>>> May to beginning of July)
> > > >>>>>> for testing 1.11. It's not ideal but further squeeze the
> > > >> development
> > > >>>> time
> > > >>>>>> of 1.12 won't make this better.
> > > >>>>>> Besides, AFAIK July & August is also a popular vacation season
> > for
> > > >>>>>> European. Given the fact most
> > > >>>>>> committers of Flink come from Europe, I think we should also
> > take
> > > >>> this
> > > >>>>>> into consideration.
> > > >>>>>>
> > > >>>>>> It's also true that the first week of October is the national
> > > >> holiday
> > > >>>> of
> > > >>>>>> China, so I'm wondering whether the
> > > >>>>>> end of October could be a candidate feature freeze date.
> > > >>>>>>
> > > >>>>>> Best,
> > > >>>>>> Kurt
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <
> > > >> rmetzger@apache.org>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hi all,
> > > >>>>>>>
> > > >>>>>>> Thanks a lot for the responses so far. I've put them into this
> > > >> Wiki
> > > >>>>> page:
> > > >>>>>>> https://cwiki.apache.org/confluence/display/FLINK/1.12+Release
> > > >> to
> > > >>>> keep
> > > >>>>>>> track of them. Ideally, post JIRA tickets for your feature,
> then
> > > >>> the
> > > >>>>>> status
> > > >>>>>>> will update automatically in the wiki :)
> > > >>>>>>>
> > > >>>>>>> Please keep posting features here, or add them to the Wiki
> > > >> yourself
> > > >>>> 🙏
> > > >>>>>>> @Prasanna kumar <pr...@gmail.com>: Dynamic Auto
> > > >>>> Scaling
> > > >>>>>> is a
> > > >>>>>>> feature request the community is well-aware of. Till has posted
> > > >>>>>>> "Reactive-scaling mode" as a feature he's working on for the
> 1.12
> > > >>>>>> release.
> > > >>>>>>> This work will introduce the basic building blocks and partial
> > > >>>> support
> > > >>>>>> for
> > > >>>>>>> the feature you are requesting.
> > > >>>>>>> Proper support for dynamic scaling, while maintaining Flink's
> > > >> high
> > > >>>>>>> performance (throughout, low latency) and correctness is a
> > > >>> difficult
> > > >>>>> task
> > > >>>>>>> that needs a lot of work. It will probably take a little bit of
> > > >>> time
> > > >>>>> till
> > > >>>>>>> this is fully available.
> > > >>>>>>>
> > > >>>>>>> Cheers,
> > > >>>>>>> Robert
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <
> > > >>> trohrmann@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Thanks for being our release managers for the 1.12 release
> > > >> Dian &
> > > >>>>>> Robert!
> > > >>>>>>>> Here are some features I would like to work on for this
> > > >> release:
> > > >>>>>>>> # Features
> > > >>>>>>>>
> > > >>>>>>>> ## Finishing pipelined region scheduling (
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-16430)
> > > >>>>>>>> With the pipelined region scheduler we want to implement a
> > > >>>> scheduler
> > > >>>>>>> which
> > > >>>>>>>> can serve streaming as well as batch workloads alike while
> > > >> being
> > > >>>> able
> > > >>>>>> to
> > > >>>>>>>> run jobs under constrained resources. The latter is
> > > >> particularly
> > > >>>>>>> important
> > > >>>>>>>> for bounded streaming jobs which, currently, are not well
> > > >>>> supported.
> > > >>>>>>>> ## Reactive-scaling mode
> > > >>>>>>>> Being able to react to newly available resources and rescaling
> > > >> a
> > > >>>>>> running
> > > >>>>>>>> job accordingly will make Flink's operation much easier
> because
> > > >>>>>> resources
> > > >>>>>>>> can then be controlled by an external tool (e.g. GCP
> > > >> autoscaling,
> > > >>>> K8s
> > > >>>>>>>> horizontal pod scaler, etc.). In this release we want to make
> a
> > > >>> big
> > > >>>>>> step
> > > >>>>>>>> towards this direction. As a first step we want to support the
> > > >>>>>> execution
> > > >>>>>>> of
> > > >>>>>>>> jobs with a parallelism which is lower than the specified
> > > >>>> parallelism
> > > >>>>>> in
> > > >>>>>>>> case that Flink lost a TaskManager or could not acquire enough
> > > >>>>>> resources.
> > > >>>>>>>> # Maintenance/Stability
> > > >>>>>>>>
> > > >>>>>>>> ## JM / TM finished task reconciliation (
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-17075)
> > > >>>>>>>> This prevents the system from going out of sync if a task
> state
> > > >>>>> change
> > > >>>>>>> from
> > > >>>>>>>> the TM to the JM is lost.
> > > >>>>>>>>
> > > >>>>>>>> ## Make metrics services work with Kubernetes deployments (
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-11127)
> > > >>>>>>>> Invert the direction in which the MetricFetcher connects to
> the
> > > >>>>>>>> MetricQueryFetchers. That way it will no longer be necessary
> to
> > > >>>>> expose
> > > >>>>>> on
> > > >>>>>>>> K8s for every TaskManager a port on which the
> > > >> MetricQueryFetcher
> > > >>>>> runs.
> > > >>>>>>> This
> > > >>>>>>>> will then make the deployment of Flink clusters on K8s easier.
> > > >>>>>>>>
> > > >>>>>>>> ## Handle long-blocking operations during job submission
> > > >>> (savepoint
> > > >>>>>>>> restore) (https://issues.apache.org/jira/browse/FLINK-16866)
> > > >>>>>>>> Submitting a Flink job can involve the interaction with
> > > >> external
> > > >>>>>> systems
> > > >>>>>>>> (blocking operations). Depending on the job the interactions
> > > >> can
> > > >>>> take
> > > >>>>>> so
> > > >>>>>>>> long that it exceeds the submission timeout which reports a
> > > >>> failure
> > > >>>>> on
> > > >>>>>>> the
> > > >>>>>>>> client side even though the actual submission succeeded. By
> > > >>>>> decoupling
> > > >>>>>>> the
> > > >>>>>>>> creation of the ExecutionGraph from the job submission, we can
> > > >>> make
> > > >>>>> the
> > > >>>>>>> job
> > > >>>>>>>> submission non-blocking which will solve this problem.
> > > >>>>>>>>
> > > >>>>>>>> ## Make IDs more intuitive to ease debugging (FLIP-118) (
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-15679)
> > > >>>>>>>> By making the internal Flink IDs compositional or logging how
> > > >>> they
> > > >>>>>> belong
> > > >>>>>>>> together, we can make the debugging of Flink's operations much
> > > >>>>> easier.
> > > >>>>>>>> Cheers,
> > > >>>>>>>> Till
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <
> > > >>>> felixzhengcb@gmail.com
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi All,
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks for bring-up this discussion, Robert!
> > > >>>>>>>>> Congratulations on becoming the release manager of 1.12， Dian
> > > >>> and
> > > >>>>>>> Robert
> > > >>>>>>>> !
> > > >>>>>>>>> ----------
> > > >>>>>>>>> Here are some of my thoughts of the features for native
> > > >>>> integration
> > > >>>>>>> with
> > > >>>>>>>>> Kubernetes in Flink 1.12:
> > > >>>>>>>>>
> > > >>>>>>>>> 1. Support user-specified pod templates
> > > >>>>>>>>> Description:
> > > >>>>>>>>> The current approach of introducing new configuration
> > > >>> options
> > > >>>>> for
> > > >>>>>>>> each
> > > >>>>>>>>> aspect of pod specification a user might wish is becoming
> > > >>>> unwieldy,
> > > >>>>>> we
> > > >>>>>>>> have
> > > >>>>>>>>> to maintain more and more Flink side Kubernetes configuration
> > > >>>>> options
> > > >>>>>>> and
> > > >>>>>>>>> users have to learn the gap between the declarative model
> > > >> used
> > > >>> by
> > > >>>>>>>>> Kubernetes and the configuration model used by Flink. It's a
> > > >>>> great
> > > >>>>>>>>> improvement to allow users to specify pod templates as
> > > >> central
> > > >>>>> places
> > > >>>>>>> for
> > > >>>>>>>>> all customization needs for the jobmanager and taskmanager
> > > >>> pods.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Users can leverage many of the advanced K8s features
> that
> > > >>> the
> > > >>>>>> Flink
> > > >>>>>>>>> community does not support explicitly, such as volume
> > > >> mounting,
> > > >>>> DNS
> > > >>>>>>>>> configuration, pod affinity/anti-affinity setting, etc.
> > > >>>>>>>>>
> > > >>>>>>>>> 2. Support running PyFlink on Kubernetes
> > > >>>>>>>>> Description:
> > > >>>>>>>>> Support running PyFlink on Kubernetes, including session
> > > >>>>> cluster
> > > >>>>>>> and
> > > >>>>>>>>> application cluster.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Running python application in a containerized
> > > >> environment.
> > > >>>>>>>>> 3. Support built-in init-Container
> > > >>>>>>>>> Description:
> > > >>>>>>>>> We need a built-in init-Container to help solve
> > > >> dependency
> > > >>>>>>> management
> > > >>>>>>>>> in a containerized environment, especially in the application
> > > >>>> mode.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Separate the base Flink image from dynamic dependencies.
> > > >>>>>>>>>
> > > >>>>>>>>> 4. Support accessing secured services via K8s secrets
> > > >>>>>>>>> Description:
> > > >>>>>>>>> Kubernetes Secrets
> > > >>>>>>>>> <https://kubernetes.io/docs/concepts/configuration/secret/>
> > > >>> can
> > > >>>> be
> > > >>>>>>> used
> > > >>>>>>>> to
> > > >>>>>>>>> provide credentials for a Flink application to access secured
> > > >>>>>> services.
> > > >>>>>>>> It
> > > >>>>>>>>> helps people who want to use a user-specified K8s Secret
> > > >>> through
> > > >>>> an
> > > >>>>>>>>> environment variable.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Improve user experience.
> > > >>>>>>>>>
> > > >>>>>>>>> 5. Support configuring replica of JobManager Deployment in
> > > >>>>> ZooKeeper
> > > >>>>>> HA
> > > >>>>>>>>> setups
> > > >>>>>>>>> Description:
> > > >>>>>>>>> Make the *replica* of Deployment configurable in the
> > > >>>> ZooKeeper
> > > >>>>> HA
> > > >>>>>>>>> setups.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Achieve faster failover.
> > > >>>>>>>>>
> > > >>>>>>>>> 6. Support to configure limit for CPU requirement
> > > >>>>>>>>> Description:
> > > >>>>>>>>> To leverage the Kubernetes feature of container
> > > >>> request/limit
> > > >>>>>> CPU.
> > > >>>>>>>>> Benefits:
> > > >>>>>>>>> Reduce cost.
> > > >>>>>>>>>
> > > >>>>>>>>> Regards,
> > > >>>>>>>>> Canbin Zheng
> > > >>>>>>>>>
> > > >>>>>>>>> Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
> > > >>>>>>>>>
> > > >>>>>>>>>> I'm excited to hear about this feature, very, very, very
> > > >>>> highly
> > > >>>>>>>>> encouraged
> > > >>>>>>>>>>
> > > >>>>>>>>>> Prasanna kumar <pr...@gmail.com>
> > > >> 于2020年7月23日周四
> > > >>>>>>>> 上午12:10写道：
> > > >>>>>>>>>>> Hi Flink Dev Team,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Dynamic AutoScaling Based on the incoming data load would
> > > >>> be
> > > >>>> a
> > > >>>>>>> great
> > > >>>>>>>>>>> feature.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> We should be able have some rule say If the load
> > > >> increased
> > > >>> by
> > > >>>>>> 20% ,
> > > >>>>>>>> add
> > > >>>>>>>>>>> extra resource should be added.
> > > >>>>>>>>>>> Or time based say during these peak hours the pipeline
> > > >>> should
> > > >>>>>> scale
> > > >>>>>>>>>>> automatically by 50%.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> This will help a lot in cost reduction.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> EMR cluster provides a similar feature for SPARK based
> > > >>>>>> application.
> > > >>>>>>>>>>> Thanks,
> > > >>>>>>>>>>> Prasanna.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <
> > > >>>>>>> rmetzger@apache.org>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Now that the 1.11 release is out, it is time to plan
> > > >> for
> > > >>>> the
> > > >>>>>> next
> > > >>>>>>>>> major
> > > >>>>>>>>>>>> Flink release.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Some items:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Dian Fu and me volunteer to be the release managers
> > > >>> for
> > > >>>>>> Flink
> > > >>>>>>>>> 1.12.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Timeline: We propose to stick to our approximate 4
> > > >>> month
> > > >>>>>>> release
> > > >>>>>>>>>>> cycle,
> > > >>>>>>>>>>>> thus the release should be done by late October.
> > > >> Given
> > > >>>>> that
> > > >>>>>>>>> there’s
> > > >>>>>>>>>> a
> > > >>>>>>>>>>>> holiday week in China at the beginning of October, I
> > > >>>>> propose
> > > >>>>>>> to
> > > >>>>>>>> do
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> feature freeze on master by late September.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 2.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Collecting features: It would be good to have a
> > > >> rough
> > > >>>>>> overview
> > > >>>>>>>> of
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> features that will likely be ready to be merged by
> > > >>> late
> > > >>>>>>>> September,
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>> that
> > > >>>>>>>>>>>> we want in the release.
> > > >>>>>>>>>>>> Based on the discussion, we will update the Roadmap
> > > >> on
> > > >>>> the
> > > >>>>>>> Flink
> > > >>>>>>>>>>> website
> > > >>>>>>>>>>>> again!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Test instabilities and blockers: I would like to
> > > >>> avoid a
> > > >>>>>>>> situation
> > > >>>>>>>>>>> where
> > > >>>>>>>>>>>> we have many blocking issues or build instabilities
> > > >> at
> > > >>>> the
> > > >>>>>>> time
> > > >>>>>>>> of
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> feature freeze. To achieve that, we will try to
> > > >> check
> > > >>>>> every
> > > >>>>>>>> build
> > > >>>>>>>>>>>> instability within a week, to decide if it is a
> > > >>> blocker
> > > >>>>>> (make
> > > >>>>>>>> sure
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>> use
> > > >>>>>>>>>>>> the “test-stability” label for those tickets!)
> > > >>>>>>>>>>>> Blocker issues will need to have somebody assigned
> > > >>>>>>> (responsible)
> > > >>>>>>>>>>> within
> > > >>>>>>>>>>>> a week, and we want to see progress on all blocker
> > > >>>> issues
> > > >>>>>>>>>> (downgrade,
> > > >>>>>>>>>>>> resolution, a good plan how to proceed if it is more
> > > >>>>>>>> complicated)
> > > >>>>>>>>>>>> 2.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Quality and stability of new features: In order to
> > > >>> have
> > > >>>> a
> > > >>>>>>> short
> > > >>>>>>>>>>> feature
> > > >>>>>>>>>>>> freeze phase, we encourage developers to only merge
> > > >>>>>>> well-tested
> > > >>>>>>>>> and
> > > >>>>>>>>>>>> documented features. In our experience, the feature
> > > >>>> freeze
> > > >>>>>>> works
> > > >>>>>>>>>> best
> > > >>>>>>>>>>> if
> > > >>>>>>>>>>>> new features are complete, and the community can
> > > >> focus
> > > >>>>> fully
> > > >>>>>>> on
> > > >>>>>>>>>>>> addressing
> > > >>>>>>>>>>>> newly found bugs and voting the release.
> > > >>>>>>>>>>>> By having a smooth release process, the next
> > > >>>> merge-window
> > > >>>>>> for
> > > >>>>>>>> the
> > > >>>>>>>>>> next
> > > >>>>>>>>>>>> release will come sooner.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Let me know what you think about our items, and share
> > > >>> which
> > > >>>>>>>> features
> > > >>>>>>>>>> you
> > > >>>>>>>>>>>> want in Flink 1.12.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Robert & Dian
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> --
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best Regards,
> > > >>>>>>>>>> Harold Miao
> > > >>>>>>>>>>
> > >
> > >
> >
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Yu Li <ca...@gmail.com>.
+1 for extending feature freeze date to end of October.
Feature development in the master branch could be unblocked through
creating the release branch, but every coin has its two sides (smile)
Best Regards,
Yu
On Wed, 5 Aug 2020 at 20:12, Robert Metzger <rm...@apache.org> wrote:
> Thanks all for your opinion.
>
> @Chesnay: That is a risk, but I hope the people responsible for individual
> FLIPs plan accordingly. Extending the time till the feature freeze should
> not mean that we are extending the scope of the release.
> Ideally, features are done before FF, and they use the time till the freeze
> for additional testing and documentation polishing.
> This FF will be virtual, there should be less disruption than a physical
> conference with all the travelling.
> Do you have a different proposal for the timing?
>
>
> I'm currently considering splitting the feature freeze and the release
> branch creation. Similar to the Linux kernel development, we could have a
> "merge window" and a stabilization phase. At the end of the stabilization
> phase, we cut the release branch and open the next merge window (I'll start
> a separate thread regarding this towards the end of this release cycle, if
> I still like the idea then)
>
>
> On Wed, Aug 5, 2020 at 12:04 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> > I'm a bit concerned about end of October, because it means we have Flink
> > forward, which usually means at least 1 week of little-to-no activity,
> > and then 1 week until feature-freeze.
> >
> > On 05/08/2020 11:56, jincheng sun wrote:
> > > +1 for end of October from me as well.
> > >
> > > Best,
> > > Jincheng
> > >
> > >
> > > Kostas Kloudas <kk...@gmail.com> 于2020年8月5日周三 下午4:59写道：
> > >
> > >> +1 for end of October from me as well.
> > >>
> > >> Cheers,
> > >> Kostas
> > >>
> > >> On Wed, Aug 5, 2020 at 9:59 AM Till Rohrmann <tr...@apache.org>
> > wrote:
> > >>
> > >>> +1 for end of October from my side as well.
> > >>>
> > >>> Cheers,
> > >>> Till
> > >>>
> > >>> On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen <se...@apache.org>
> wrote:
> > >>>
> > >>>> The end of October sounds good from my side, unless it collides with
> > >> some
> > >>>> holidays that affect many committers.
> > >>>>
> > >>>> Feature-wise, I believe we can definitely make good use of the time
> to
> > >>> wrap
> > >>>> up some critical threads (like finishing the FLIP-27 source
> efforts).
> > >>>>
> > >>>> So +1 to the end of October from my side.
> > >>>>
> > >>>> Best,
> > >>>> Stephan
> > >>>>
> > >>>>
> > >>>> On Tue, Aug 4, 2020 at 8:59 AM Robert Metzger <rm...@apache.org>
> > >>> wrote:
> > >>>>> Thanks a lot for commenting on the feature freeze date.
> > >>>>>
> > >>>>> You are raising a few good points on the timing.
> > >>>>> If we have already (2 months before) concerns regarding the
> deadline,
> > >>>> then
> > >>>>> I agree that we should move it till the end of October.
> > >>>>>
> > >>>>> We then just need to be careful not to run into the Christmas
> season
> > >> at
> > >>>> the
> > >>>>> end of December.
> > >>>>>
> > >>>>> If nobody objects within a few days, I'll update the feature freeze
> > >>> date
> > >>>> in
> > >>>>> the Wiki.
> > >>>>>
> > >>>>>
> > >>>>> On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <yk...@gmail.com>
> wrote:
> > >>>>>
> > >>>>>> Regarding setting the feature freeze date to late September, I
> have
> > >>>> some
> > >>>>>> concern that it might make
> > >>>>>> the development time of 1.12 too short.
> > >>>>>>
> > >>>>>> One reason for this is we took too much time (about 1.5 month,
> from
> > >>> mid
> > >>>>> of
> > >>>>>> May to beginning of July)
> > >>>>>> for testing 1.11. It's not ideal but further squeeze the
> > >> development
> > >>>> time
> > >>>>>> of 1.12 won't make this better.
> > >>>>>> Besides, AFAIK July & August is also a popular vacation season
> for
> > >>>>>> European. Given the fact most
> > >>>>>> committers of Flink come from Europe, I think we should also
> take
> > >>> this
> > >>>>>> into consideration.
> > >>>>>>
> > >>>>>> It's also true that the first week of October is the national
> > >> holiday
> > >>>> of
> > >>>>>> China, so I'm wondering whether the
> > >>>>>> end of October could be a candidate feature freeze date.
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Kurt
> > >>>>>>
> > >>>>>>
> > >>>>>> On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <
> > >> rmetzger@apache.org>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hi all,
> > >>>>>>>
> > >>>>>>> Thanks a lot for the responses so far. I've put them into this
> > >> Wiki
> > >>>>> page:
> > >>>>>>> https://cwiki.apache.org/confluence/display/FLINK/1.12+Release
> > >> to
> > >>>> keep
> > >>>>>>> track of them. Ideally, post JIRA tickets for your feature, then
> > >>> the
> > >>>>>> status
> > >>>>>>> will update automatically in the wiki :)
> > >>>>>>>
> > >>>>>>> Please keep posting features here, or add them to the Wiki
> > >> yourself
> > >>>> 🙏
> > >>>>>>> @Prasanna kumar <pr...@gmail.com>: Dynamic Auto
> > >>>> Scaling
> > >>>>>> is a
> > >>>>>>> feature request the community is well-aware of. Till has posted
> > >>>>>>> "Reactive-scaling mode" as a feature he's working on for the 1.12
> > >>>>>> release.
> > >>>>>>> This work will introduce the basic building blocks and partial
> > >>>> support
> > >>>>>> for
> > >>>>>>> the feature you are requesting.
> > >>>>>>> Proper support for dynamic scaling, while maintaining Flink's
> > >> high
> > >>>>>>> performance (throughout, low latency) and correctness is a
> > >>> difficult
> > >>>>> task
> > >>>>>>> that needs a lot of work. It will probably take a little bit of
> > >>> time
> > >>>>> till
> > >>>>>>> this is fully available.
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Robert
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <
> > >>> trohrmann@apache.org>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Thanks for being our release managers for the 1.12 release
> > >> Dian &
> > >>>>>> Robert!
> > >>>>>>>> Here are some features I would like to work on for this
> > >> release:
> > >>>>>>>> # Features
> > >>>>>>>>
> > >>>>>>>> ## Finishing pipelined region scheduling (
> > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-16430)
> > >>>>>>>> With the pipelined region scheduler we want to implement a
> > >>>> scheduler
> > >>>>>>> which
> > >>>>>>>> can serve streaming as well as batch workloads alike while
> > >> being
> > >>>> able
> > >>>>>> to
> > >>>>>>>> run jobs under constrained resources. The latter is
> > >> particularly
> > >>>>>>> important
> > >>>>>>>> for bounded streaming jobs which, currently, are not well
> > >>>> supported.
> > >>>>>>>> ## Reactive-scaling mode
> > >>>>>>>> Being able to react to newly available resources and rescaling
> > >> a
> > >>>>>> running
> > >>>>>>>> job accordingly will make Flink's operation much easier because
> > >>>>>> resources
> > >>>>>>>> can then be controlled by an external tool (e.g. GCP
> > >> autoscaling,
> > >>>> K8s
> > >>>>>>>> horizontal pod scaler, etc.). In this release we want to make a
> > >>> big
> > >>>>>> step
> > >>>>>>>> towards this direction. As a first step we want to support the
> > >>>>>> execution
> > >>>>>>> of
> > >>>>>>>> jobs with a parallelism which is lower than the specified
> > >>>> parallelism
> > >>>>>> in
> > >>>>>>>> case that Flink lost a TaskManager or could not acquire enough
> > >>>>>> resources.
> > >>>>>>>> # Maintenance/Stability
> > >>>>>>>>
> > >>>>>>>> ## JM / TM finished task reconciliation (
> > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-17075)
> > >>>>>>>> This prevents the system from going out of sync if a task state
> > >>>>> change
> > >>>>>>> from
> > >>>>>>>> the TM to the JM is lost.
> > >>>>>>>>
> > >>>>>>>> ## Make metrics services work with Kubernetes deployments (
> > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-11127)
> > >>>>>>>> Invert the direction in which the MetricFetcher connects to the
> > >>>>>>>> MetricQueryFetchers. That way it will no longer be necessary to
> > >>>>> expose
> > >>>>>> on
> > >>>>>>>> K8s for every TaskManager a port on which the
> > >> MetricQueryFetcher
> > >>>>> runs.
> > >>>>>>> This
> > >>>>>>>> will then make the deployment of Flink clusters on K8s easier.
> > >>>>>>>>
> > >>>>>>>> ## Handle long-blocking operations during job submission
> > >>> (savepoint
> > >>>>>>>> restore) (https://issues.apache.org/jira/browse/FLINK-16866)
> > >>>>>>>> Submitting a Flink job can involve the interaction with
> > >> external
> > >>>>>> systems
> > >>>>>>>> (blocking operations). Depending on the job the interactions
> > >> can
> > >>>> take
> > >>>>>> so
> > >>>>>>>> long that it exceeds the submission timeout which reports a
> > >>> failure
> > >>>>> on
> > >>>>>>> the
> > >>>>>>>> client side even though the actual submission succeeded. By
> > >>>>> decoupling
> > >>>>>>> the
> > >>>>>>>> creation of the ExecutionGraph from the job submission, we can
> > >>> make
> > >>>>> the
> > >>>>>>> job
> > >>>>>>>> submission non-blocking which will solve this problem.
> > >>>>>>>>
> > >>>>>>>> ## Make IDs more intuitive to ease debugging (FLIP-118) (
> > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-15679)
> > >>>>>>>> By making the internal Flink IDs compositional or logging how
> > >>> they
> > >>>>>> belong
> > >>>>>>>> together, we can make the debugging of Flink's operations much
> > >>>>> easier.
> > >>>>>>>> Cheers,
> > >>>>>>>> Till
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <
> > >>>> felixzhengcb@gmail.com
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi All,
> > >>>>>>>>>
> > >>>>>>>>> Thanks for bring-up this discussion, Robert!
> > >>>>>>>>> Congratulations on becoming the release manager of 1.12， Dian
> > >>> and
> > >>>>>>> Robert
> > >>>>>>>> !
> > >>>>>>>>> ----------
> > >>>>>>>>> Here are some of my thoughts of the features for native
> > >>>> integration
> > >>>>>>> with
> > >>>>>>>>> Kubernetes in Flink 1.12:
> > >>>>>>>>>
> > >>>>>>>>> 1. Support user-specified pod templates
> > >>>>>>>>> Description:
> > >>>>>>>>> The current approach of introducing new configuration
> > >>> options
> > >>>>> for
> > >>>>>>>> each
> > >>>>>>>>> aspect of pod specification a user might wish is becoming
> > >>>> unwieldy,
> > >>>>>> we
> > >>>>>>>> have
> > >>>>>>>>> to maintain more and more Flink side Kubernetes configuration
> > >>>>> options
> > >>>>>>> and
> > >>>>>>>>> users have to learn the gap between the declarative model
> > >> used
> > >>> by
> > >>>>>>>>> Kubernetes and the configuration model used by Flink. It's a
> > >>>> great
> > >>>>>>>>> improvement to allow users to specify pod templates as
> > >> central
> > >>>>> places
> > >>>>>>> for
> > >>>>>>>>> all customization needs for the jobmanager and taskmanager
> > >>> pods.
> > >>>>>>>>> Benefits:
> > >>>>>>>>> Users can leverage many of the advanced K8s features that
> > >>> the
> > >>>>>> Flink
> > >>>>>>>>> community does not support explicitly, such as volume
> > >> mounting,
> > >>>> DNS
> > >>>>>>>>> configuration, pod affinity/anti-affinity setting, etc.
> > >>>>>>>>>
> > >>>>>>>>> 2. Support running PyFlink on Kubernetes
> > >>>>>>>>> Description:
> > >>>>>>>>> Support running PyFlink on Kubernetes, including session
> > >>>>> cluster
> > >>>>>>> and
> > >>>>>>>>> application cluster.
> > >>>>>>>>> Benefits:
> > >>>>>>>>> Running python application in a containerized
> > >> environment.
> > >>>>>>>>> 3. Support built-in init-Container
> > >>>>>>>>> Description:
> > >>>>>>>>> We need a built-in init-Container to help solve
> > >> dependency
> > >>>>>>> management
> > >>>>>>>>> in a containerized environment, especially in the application
> > >>>> mode.
> > >>>>>>>>> Benefits:
> > >>>>>>>>> Separate the base Flink image from dynamic dependencies.
> > >>>>>>>>>
> > >>>>>>>>> 4. Support accessing secured services via K8s secrets
> > >>>>>>>>> Description:
> > >>>>>>>>> Kubernetes Secrets
> > >>>>>>>>> <https://kubernetes.io/docs/concepts/configuration/secret/>
> > >>> can
> > >>>> be
> > >>>>>>> used
> > >>>>>>>> to
> > >>>>>>>>> provide credentials for a Flink application to access secured
> > >>>>>> services.
> > >>>>>>>> It
> > >>>>>>>>> helps people who want to use a user-specified K8s Secret
> > >>> through
> > >>>> an
> > >>>>>>>>> environment variable.
> > >>>>>>>>> Benefits:
> > >>>>>>>>> Improve user experience.
> > >>>>>>>>>
> > >>>>>>>>> 5. Support configuring replica of JobManager Deployment in
> > >>>>> ZooKeeper
> > >>>>>> HA
> > >>>>>>>>> setups
> > >>>>>>>>> Description:
> > >>>>>>>>> Make the *replica* of Deployment configurable in the
> > >>>> ZooKeeper
> > >>>>> HA
> > >>>>>>>>> setups.
> > >>>>>>>>> Benefits:
> > >>>>>>>>> Achieve faster failover.
> > >>>>>>>>>
> > >>>>>>>>> 6. Support to configure limit for CPU requirement
> > >>>>>>>>> Description:
> > >>>>>>>>> To leverage the Kubernetes feature of container
> > >>> request/limit
> > >>>>>> CPU.
> > >>>>>>>>> Benefits:
> > >>>>>>>>> Reduce cost.
> > >>>>>>>>>
> > >>>>>>>>> Regards,
> > >>>>>>>>> Canbin Zheng
> > >>>>>>>>>
> > >>>>>>>>> Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
> > >>>>>>>>>
> > >>>>>>>>>> I'm excited to hear about this feature, very, very, very
> > >>>> highly
> > >>>>>>>>> encouraged
> > >>>>>>>>>>
> > >>>>>>>>>> Prasanna kumar <pr...@gmail.com>
> > >> 于2020年7月23日周四
> > >>>>>>>> 上午12:10写道：
> > >>>>>>>>>>> Hi Flink Dev Team,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Dynamic AutoScaling Based on the incoming data load would
> > >>> be
> > >>>> a
> > >>>>>>> great
> > >>>>>>>>>>> feature.
> > >>>>>>>>>>>
> > >>>>>>>>>>> We should be able have some rule say If the load
> > >> increased
> > >>> by
> > >>>>>> 20% ,
> > >>>>>>>> add
> > >>>>>>>>>>> extra resource should be added.
> > >>>>>>>>>>> Or time based say during these peak hours the pipeline
> > >>> should
> > >>>>>> scale
> > >>>>>>>>>>> automatically by 50%.
> > >>>>>>>>>>>
> > >>>>>>>>>>> This will help a lot in cost reduction.
> > >>>>>>>>>>>
> > >>>>>>>>>>> EMR cluster provides a similar feature for SPARK based
> > >>>>>> application.
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>> Prasanna.
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <
> > >>>>>>> rmetzger@apache.org>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Now that the 1.11 release is out, it is time to plan
> > >> for
> > >>>> the
> > >>>>>> next
> > >>>>>>>>> major
> > >>>>>>>>>>>> Flink release.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Some items:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Dian Fu and me volunteer to be the release managers
> > >>> for
> > >>>>>> Flink
> > >>>>>>>>> 1.12.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Timeline: We propose to stick to our approximate 4
> > >>> month
> > >>>>>>> release
> > >>>>>>>>>>> cycle,
> > >>>>>>>>>>>> thus the release should be done by late October.
> > >> Given
> > >>>>> that
> > >>>>>>>>> there’s
> > >>>>>>>>>> a
> > >>>>>>>>>>>> holiday week in China at the beginning of October, I
> > >>>>> propose
> > >>>>>>> to
> > >>>>>>>> do
> > >>>>>>>>>> the
> > >>>>>>>>>>>> feature freeze on master by late September.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 2.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Collecting features: It would be good to have a
> > >> rough
> > >>>>>> overview
> > >>>>>>>> of
> > >>>>>>>>>> the
> > >>>>>>>>>>>> features that will likely be ready to be merged by
> > >>> late
> > >>>>>>>> September,
> > >>>>>>>>>> and
> > >>>>>>>>>>>> that
> > >>>>>>>>>>>> we want in the release.
> > >>>>>>>>>>>> Based on the discussion, we will update the Roadmap
> > >> on
> > >>>> the
> > >>>>>>> Flink
> > >>>>>>>>>>> website
> > >>>>>>>>>>>> again!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Test instabilities and blockers: I would like to
> > >>> avoid a
> > >>>>>>>> situation
> > >>>>>>>>>>> where
> > >>>>>>>>>>>> we have many blocking issues or build instabilities
> > >> at
> > >>>> the
> > >>>>>>> time
> > >>>>>>>> of
> > >>>>>>>>>> the
> > >>>>>>>>>>>> feature freeze. To achieve that, we will try to
> > >> check
> > >>>>> every
> > >>>>>>>> build
> > >>>>>>>>>>>> instability within a week, to decide if it is a
> > >>> blocker
> > >>>>>> (make
> > >>>>>>>> sure
> > >>>>>>>>>> to
> > >>>>>>>>>>>> use
> > >>>>>>>>>>>> the “test-stability” label for those tickets!)
> > >>>>>>>>>>>> Blocker issues will need to have somebody assigned
> > >>>>>>> (responsible)
> > >>>>>>>>>>> within
> > >>>>>>>>>>>> a week, and we want to see progress on all blocker
> > >>>> issues
> > >>>>>>>>>> (downgrade,
> > >>>>>>>>>>>> resolution, a good plan how to proceed if it is more
> > >>>>>>>> complicated)
> > >>>>>>>>>>>> 2.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Quality and stability of new features: In order to
> > >>> have
> > >>>> a
> > >>>>>>> short
> > >>>>>>>>>>> feature
> > >>>>>>>>>>>> freeze phase, we encourage developers to only merge
> > >>>>>>> well-tested
> > >>>>>>>>> and
> > >>>>>>>>>>>> documented features. In our experience, the feature
> > >>>> freeze
> > >>>>>>> works
> > >>>>>>>>>> best
> > >>>>>>>>>>> if
> > >>>>>>>>>>>> new features are complete, and the community can
> > >> focus
> > >>>>> fully
> > >>>>>>> on
> > >>>>>>>>>>>> addressing
> > >>>>>>>>>>>> newly found bugs and voting the release.
> > >>>>>>>>>>>> By having a smooth release process, the next
> > >>>> merge-window
> > >>>>>> for
> > >>>>>>>> the
> > >>>>>>>>>> next
> > >>>>>>>>>>>> release will come sooner.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Let me know what you think about our items, and share
> > >>> which
> > >>>>>>>> features
> > >>>>>>>>>> you
> > >>>>>>>>>>>> want in Flink 1.12.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Robert & Dian
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> --
> > >>>>>>>>>>
> > >>>>>>>>>> Best Regards,
> > >>>>>>>>>> Harold Miao
> > >>>>>>>>>>
> >
> >
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Robert Metzger <rm...@apache.org>.
Thanks all for your opinion.
@Chesnay: That is a risk, but I hope the people responsible for individual
FLIPs plan accordingly. Extending the time till the feature freeze should
not mean that we are extending the scope of the release.
Ideally, features are done before FF, and they use the time till the freeze
for additional testing and documentation polishing.
This FF will be virtual, there should be less disruption than a physical
conference with all the travelling.
Do you have a different proposal for the timing?
I'm currently considering splitting the feature freeze and the release
branch creation. Similar to the Linux kernel development, we could have a
"merge window" and a stabilization phase. At the end of the stabilization
phase, we cut the release branch and open the next merge window (I'll start
a separate thread regarding this towards the end of this release cycle, if
I still like the idea then)
On Wed, Aug 5, 2020 at 12:04 PM Chesnay Schepler <ch...@apache.org> wrote:
> I'm a bit concerned about end of October, because it means we have Flink
> forward, which usually means at least 1 week of little-to-no activity,
> and then 1 week until feature-freeze.
>
> On 05/08/2020 11:56, jincheng sun wrote:
> > +1 for end of October from me as well.
> >
> > Best,
> > Jincheng
> >
> >
> > Kostas Kloudas <kk...@gmail.com> 于2020年8月5日周三 下午4:59写道：
> >
> >> +1 for end of October from me as well.
> >>
> >> Cheers,
> >> Kostas
> >>
> >> On Wed, Aug 5, 2020 at 9:59 AM Till Rohrmann <tr...@apache.org>
> wrote:
> >>
> >>> +1 for end of October from my side as well.
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen <se...@apache.org> wrote:
> >>>
> >>>> The end of October sounds good from my side, unless it collides with
> >> some
> >>>> holidays that affect many committers.
> >>>>
> >>>> Feature-wise, I believe we can definitely make good use of the time to
> >>> wrap
> >>>> up some critical threads (like finishing the FLIP-27 source efforts).
> >>>>
> >>>> So +1 to the end of October from my side.
> >>>>
> >>>> Best,
> >>>> Stephan
> >>>>
> >>>>
> >>>> On Tue, Aug 4, 2020 at 8:59 AM Robert Metzger <rm...@apache.org>
> >>> wrote:
> >>>>> Thanks a lot for commenting on the feature freeze date.
> >>>>>
> >>>>> You are raising a few good points on the timing.
> >>>>> If we have already (2 months before) concerns regarding the deadline,
> >>>> then
> >>>>> I agree that we should move it till the end of October.
> >>>>>
> >>>>> We then just need to be careful not to run into the Christmas season
> >> at
> >>>> the
> >>>>> end of December.
> >>>>>
> >>>>> If nobody objects within a few days, I'll update the feature freeze
> >>> date
> >>>> in
> >>>>> the Wiki.
> >>>>>
> >>>>>
> >>>>> On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <yk...@gmail.com> wrote:
> >>>>>
> >>>>>> Regarding setting the feature freeze date to late September, I have
> >>>> some
> >>>>>> concern that it might make
> >>>>>> the development time of 1.12 too short.
> >>>>>>
> >>>>>> One reason for this is we took too much time (about 1.5 month, from
> >>> mid
> >>>>> of
> >>>>>> May to beginning of July)
> >>>>>> for testing 1.11. It's not ideal but further squeeze the
> >> development
> >>>> time
> >>>>>> of 1.12 won't make this better.
> >>>>>> Besides, AFAIK July & August is also a popular vacation season for
> >>>>>> European. Given the fact most
> >>>>>> committers of Flink come from Europe, I think we should also take
> >>> this
> >>>>>> into consideration.
> >>>>>>
> >>>>>> It's also true that the first week of October is the national
> >> holiday
> >>>> of
> >>>>>> China, so I'm wondering whether the
> >>>>>> end of October could be a candidate feature freeze date.
> >>>>>>
> >>>>>> Best,
> >>>>>> Kurt
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <
> >> rmetzger@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> Thanks a lot for the responses so far. I've put them into this
> >> Wiki
> >>>>> page:
> >>>>>>> https://cwiki.apache.org/confluence/display/FLINK/1.12+Release
> >> to
> >>>> keep
> >>>>>>> track of them. Ideally, post JIRA tickets for your feature, then
> >>> the
> >>>>>> status
> >>>>>>> will update automatically in the wiki :)
> >>>>>>>
> >>>>>>> Please keep posting features here, or add them to the Wiki
> >> yourself
> >>>> 🙏
> >>>>>>> @Prasanna kumar <pr...@gmail.com>: Dynamic Auto
> >>>> Scaling
> >>>>>> is a
> >>>>>>> feature request the community is well-aware of. Till has posted
> >>>>>>> "Reactive-scaling mode" as a feature he's working on for the 1.12
> >>>>>> release.
> >>>>>>> This work will introduce the basic building blocks and partial
> >>>> support
> >>>>>> for
> >>>>>>> the feature you are requesting.
> >>>>>>> Proper support for dynamic scaling, while maintaining Flink's
> >> high
> >>>>>>> performance (throughout, low latency) and correctness is a
> >>> difficult
> >>>>> task
> >>>>>>> that needs a lot of work. It will probably take a little bit of
> >>> time
> >>>>> till
> >>>>>>> this is fully available.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Robert
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <
> >>> trohrmann@apache.org>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Thanks for being our release managers for the 1.12 release
> >> Dian &
> >>>>>> Robert!
> >>>>>>>> Here are some features I would like to work on for this
> >> release:
> >>>>>>>> # Features
> >>>>>>>>
> >>>>>>>> ## Finishing pipelined region scheduling (
> >>>>>>>> https://issues.apache.org/jira/browse/FLINK-16430)
> >>>>>>>> With the pipelined region scheduler we want to implement a
> >>>> scheduler
> >>>>>>> which
> >>>>>>>> can serve streaming as well as batch workloads alike while
> >> being
> >>>> able
> >>>>>> to
> >>>>>>>> run jobs under constrained resources. The latter is
> >> particularly
> >>>>>>> important
> >>>>>>>> for bounded streaming jobs which, currently, are not well
> >>>> supported.
> >>>>>>>> ## Reactive-scaling mode
> >>>>>>>> Being able to react to newly available resources and rescaling
> >> a
> >>>>>> running
> >>>>>>>> job accordingly will make Flink's operation much easier because
> >>>>>> resources
> >>>>>>>> can then be controlled by an external tool (e.g. GCP
> >> autoscaling,
> >>>> K8s
> >>>>>>>> horizontal pod scaler, etc.). In this release we want to make a
> >>> big
> >>>>>> step
> >>>>>>>> towards this direction. As a first step we want to support the
> >>>>>> execution
> >>>>>>> of
> >>>>>>>> jobs with a parallelism which is lower than the specified
> >>>> parallelism
> >>>>>> in
> >>>>>>>> case that Flink lost a TaskManager or could not acquire enough
> >>>>>> resources.
> >>>>>>>> # Maintenance/Stability
> >>>>>>>>
> >>>>>>>> ## JM / TM finished task reconciliation (
> >>>>>>>> https://issues.apache.org/jira/browse/FLINK-17075)
> >>>>>>>> This prevents the system from going out of sync if a task state
> >>>>> change
> >>>>>>> from
> >>>>>>>> the TM to the JM is lost.
> >>>>>>>>
> >>>>>>>> ## Make metrics services work with Kubernetes deployments (
> >>>>>>>> https://issues.apache.org/jira/browse/FLINK-11127)
> >>>>>>>> Invert the direction in which the MetricFetcher connects to the
> >>>>>>>> MetricQueryFetchers. That way it will no longer be necessary to
> >>>>> expose
> >>>>>> on
> >>>>>>>> K8s for every TaskManager a port on which the
> >> MetricQueryFetcher
> >>>>> runs.
> >>>>>>> This
> >>>>>>>> will then make the deployment of Flink clusters on K8s easier.
> >>>>>>>>
> >>>>>>>> ## Handle long-blocking operations during job submission
> >>> (savepoint
> >>>>>>>> restore) (https://issues.apache.org/jira/browse/FLINK-16866)
> >>>>>>>> Submitting a Flink job can involve the interaction with
> >> external
> >>>>>> systems
> >>>>>>>> (blocking operations). Depending on the job the interactions
> >> can
> >>>> take
> >>>>>> so
> >>>>>>>> long that it exceeds the submission timeout which reports a
> >>> failure
> >>>>> on
> >>>>>>> the
> >>>>>>>> client side even though the actual submission succeeded. By
> >>>>> decoupling
> >>>>>>> the
> >>>>>>>> creation of the ExecutionGraph from the job submission, we can
> >>> make
> >>>>> the
> >>>>>>> job
> >>>>>>>> submission non-blocking which will solve this problem.
> >>>>>>>>
> >>>>>>>> ## Make IDs more intuitive to ease debugging (FLIP-118) (
> >>>>>>>> https://issues.apache.org/jira/browse/FLINK-15679)
> >>>>>>>> By making the internal Flink IDs compositional or logging how
> >>> they
> >>>>>> belong
> >>>>>>>> together, we can make the debugging of Flink's operations much
> >>>>> easier.
> >>>>>>>> Cheers,
> >>>>>>>> Till
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <
> >>>> felixzhengcb@gmail.com
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi All,
> >>>>>>>>>
> >>>>>>>>> Thanks for bring-up this discussion, Robert!
> >>>>>>>>> Congratulations on becoming the release manager of 1.12， Dian
> >>> and
> >>>>>>> Robert
> >>>>>>>> !
> >>>>>>>>> ----------
> >>>>>>>>> Here are some of my thoughts of the features for native
> >>>> integration
> >>>>>>> with
> >>>>>>>>> Kubernetes in Flink 1.12:
> >>>>>>>>>
> >>>>>>>>> 1. Support user-specified pod templates
> >>>>>>>>> Description:
> >>>>>>>>> The current approach of introducing new configuration
> >>> options
> >>>>> for
> >>>>>>>> each
> >>>>>>>>> aspect of pod specification a user might wish is becoming
> >>>> unwieldy,
> >>>>>> we
> >>>>>>>> have
> >>>>>>>>> to maintain more and more Flink side Kubernetes configuration
> >>>>> options
> >>>>>>> and
> >>>>>>>>> users have to learn the gap between the declarative model
> >> used
> >>> by
> >>>>>>>>> Kubernetes and the configuration model used by Flink. It's a
> >>>> great
> >>>>>>>>> improvement to allow users to specify pod templates as
> >> central
> >>>>> places
> >>>>>>> for
> >>>>>>>>> all customization needs for the jobmanager and taskmanager
> >>> pods.
> >>>>>>>>> Benefits:
> >>>>>>>>> Users can leverage many of the advanced K8s features that
> >>> the
> >>>>>> Flink
> >>>>>>>>> community does not support explicitly, such as volume
> >> mounting,
> >>>> DNS
> >>>>>>>>> configuration, pod affinity/anti-affinity setting, etc.
> >>>>>>>>>
> >>>>>>>>> 2. Support running PyFlink on Kubernetes
> >>>>>>>>> Description:
> >>>>>>>>> Support running PyFlink on Kubernetes, including session
> >>>>> cluster
> >>>>>>> and
> >>>>>>>>> application cluster.
> >>>>>>>>> Benefits:
> >>>>>>>>> Running python application in a containerized
> >> environment.
> >>>>>>>>> 3. Support built-in init-Container
> >>>>>>>>> Description:
> >>>>>>>>> We need a built-in init-Container to help solve
> >> dependency
> >>>>>>> management
> >>>>>>>>> in a containerized environment, especially in the application
> >>>> mode.
> >>>>>>>>> Benefits:
> >>>>>>>>> Separate the base Flink image from dynamic dependencies.
> >>>>>>>>>
> >>>>>>>>> 4. Support accessing secured services via K8s secrets
> >>>>>>>>> Description:
> >>>>>>>>> Kubernetes Secrets
> >>>>>>>>> <https://kubernetes.io/docs/concepts/configuration/secret/>
> >>> can
> >>>> be
> >>>>>>> used
> >>>>>>>> to
> >>>>>>>>> provide credentials for a Flink application to access secured
> >>>>>> services.
> >>>>>>>> It
> >>>>>>>>> helps people who want to use a user-specified K8s Secret
> >>> through
> >>>> an
> >>>>>>>>> environment variable.
> >>>>>>>>> Benefits:
> >>>>>>>>> Improve user experience.
> >>>>>>>>>
> >>>>>>>>> 5. Support configuring replica of JobManager Deployment in
> >>>>> ZooKeeper
> >>>>>> HA
> >>>>>>>>> setups
> >>>>>>>>> Description:
> >>>>>>>>> Make the *replica* of Deployment configurable in the
> >>>> ZooKeeper
> >>>>> HA
> >>>>>>>>> setups.
> >>>>>>>>> Benefits:
> >>>>>>>>> Achieve faster failover.
> >>>>>>>>>
> >>>>>>>>> 6. Support to configure limit for CPU requirement
> >>>>>>>>> Description:
> >>>>>>>>> To leverage the Kubernetes feature of container
> >>> request/limit
> >>>>>> CPU.
> >>>>>>>>> Benefits:
> >>>>>>>>> Reduce cost.
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Canbin Zheng
> >>>>>>>>>
> >>>>>>>>> Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
> >>>>>>>>>
> >>>>>>>>>> I'm excited to hear about this feature, very, very, very
> >>>> highly
> >>>>>>>>> encouraged
> >>>>>>>>>>
> >>>>>>>>>> Prasanna kumar <pr...@gmail.com>
> >> 于2020年7月23日周四
> >>>>>>>> 上午12:10写道：
> >>>>>>>>>>> Hi Flink Dev Team,
> >>>>>>>>>>>
> >>>>>>>>>>> Dynamic AutoScaling Based on the incoming data load would
> >>> be
> >>>> a
> >>>>>>> great
> >>>>>>>>>>> feature.
> >>>>>>>>>>>
> >>>>>>>>>>> We should be able have some rule say If the load
> >> increased
> >>> by
> >>>>>> 20% ,
> >>>>>>>> add
> >>>>>>>>>>> extra resource should be added.
> >>>>>>>>>>> Or time based say during these peak hours the pipeline
> >>> should
> >>>>>> scale
> >>>>>>>>>>> automatically by 50%.
> >>>>>>>>>>>
> >>>>>>>>>>> This will help a lot in cost reduction.
> >>>>>>>>>>>
> >>>>>>>>>>> EMR cluster provides a similar feature for SPARK based
> >>>>>> application.
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Prasanna.
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <
> >>>>>>> rmetzger@apache.org>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Now that the 1.11 release is out, it is time to plan
> >> for
> >>>> the
> >>>>>> next
> >>>>>>>>> major
> >>>>>>>>>>>> Flink release.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Some items:
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Dian Fu and me volunteer to be the release managers
> >>> for
> >>>>>> Flink
> >>>>>>>>> 1.12.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Timeline: We propose to stick to our approximate 4
> >>> month
> >>>>>>> release
> >>>>>>>>>>> cycle,
> >>>>>>>>>>>> thus the release should be done by late October.
> >> Given
> >>>>> that
> >>>>>>>>> there’s
> >>>>>>>>>> a
> >>>>>>>>>>>> holiday week in China at the beginning of October, I
> >>>>> propose
> >>>>>>> to
> >>>>>>>> do
> >>>>>>>>>> the
> >>>>>>>>>>>> feature freeze on master by late September.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Collecting features: It would be good to have a
> >> rough
> >>>>>> overview
> >>>>>>>> of
> >>>>>>>>>> the
> >>>>>>>>>>>> features that will likely be ready to be merged by
> >>> late
> >>>>>>>> September,
> >>>>>>>>>> and
> >>>>>>>>>>>> that
> >>>>>>>>>>>> we want in the release.
> >>>>>>>>>>>> Based on the discussion, we will update the Roadmap
> >> on
> >>>> the
> >>>>>>> Flink
> >>>>>>>>>>> website
> >>>>>>>>>>>> again!
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Test instabilities and blockers: I would like to
> >>> avoid a
> >>>>>>>> situation
> >>>>>>>>>>> where
> >>>>>>>>>>>> we have many blocking issues or build instabilities
> >> at
> >>>> the
> >>>>>>> time
> >>>>>>>> of
> >>>>>>>>>> the
> >>>>>>>>>>>> feature freeze. To achieve that, we will try to
> >> check
> >>>>> every
> >>>>>>>> build
> >>>>>>>>>>>> instability within a week, to decide if it is a
> >>> blocker
> >>>>>> (make
> >>>>>>>> sure
> >>>>>>>>>> to
> >>>>>>>>>>>> use
> >>>>>>>>>>>> the “test-stability” label for those tickets!)
> >>>>>>>>>>>> Blocker issues will need to have somebody assigned
> >>>>>>> (responsible)
> >>>>>>>>>>> within
> >>>>>>>>>>>> a week, and we want to see progress on all blocker
> >>>> issues
> >>>>>>>>>> (downgrade,
> >>>>>>>>>>>> resolution, a good plan how to proceed if it is more
> >>>>>>>> complicated)
> >>>>>>>>>>>> 2.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Quality and stability of new features: In order to
> >>> have
> >>>> a
> >>>>>>> short
> >>>>>>>>>>> feature
> >>>>>>>>>>>> freeze phase, we encourage developers to only merge
> >>>>>>> well-tested
> >>>>>>>>> and
> >>>>>>>>>>>> documented features. In our experience, the feature
> >>>> freeze
> >>>>>>> works
> >>>>>>>>>> best
> >>>>>>>>>>> if
> >>>>>>>>>>>> new features are complete, and the community can
> >> focus
> >>>>> fully
> >>>>>>> on
> >>>>>>>>>>>> addressing
> >>>>>>>>>>>> newly found bugs and voting the release.
> >>>>>>>>>>>> By having a smooth release process, the next
> >>>> merge-window
> >>>>>> for
> >>>>>>>> the
> >>>>>>>>>> next
> >>>>>>>>>>>> release will come sooner.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Let me know what you think about our items, and share
> >>> which
> >>>>>>>> features
> >>>>>>>>>> you
> >>>>>>>>>>>> want in Flink 1.12.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Robert & Dian
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>>
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Harold Miao
> >>>>>>>>>>
>
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Chesnay Schepler <ch...@apache.org>.
I'm a bit concerned about end of October, because it means we have Flink
forward, which usually means at least 1 week of little-to-no activity,
and then 1 week until feature-freeze.
On 05/08/2020 11:56, jincheng sun wrote:
> +1 for end of October from me as well.
>
> Best,
> Jincheng
>
>
> Kostas Kloudas <kk...@gmail.com> 于2020年8月5日周三 下午4:59写道：
>
>> +1 for end of October from me as well.
>>
>> Cheers,
>> Kostas
>>
>> On Wed, Aug 5, 2020 at 9:59 AM Till Rohrmann <tr...@apache.org> wrote:
>>
>>> +1 for end of October from my side as well.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen <se...@apache.org> wrote:
>>>
>>>> The end of October sounds good from my side, unless it collides with
>> some
>>>> holidays that affect many committers.
>>>>
>>>> Feature-wise, I believe we can definitely make good use of the time to
>>> wrap
>>>> up some critical threads (like finishing the FLIP-27 source efforts).
>>>>
>>>> So +1 to the end of October from my side.
>>>>
>>>> Best,
>>>> Stephan
>>>>
>>>>
>>>> On Tue, Aug 4, 2020 at 8:59 AM Robert Metzger <rm...@apache.org>
>>> wrote:
>>>>> Thanks a lot for commenting on the feature freeze date.
>>>>>
>>>>> You are raising a few good points on the timing.
>>>>> If we have already (2 months before) concerns regarding the deadline,
>>>> then
>>>>> I agree that we should move it till the end of October.
>>>>>
>>>>> We then just need to be careful not to run into the Christmas season
>> at
>>>> the
>>>>> end of December.
>>>>>
>>>>> If nobody objects within a few days, I'll update the feature freeze
>>> date
>>>> in
>>>>> the Wiki.
>>>>>
>>>>>
>>>>> On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <yk...@gmail.com> wrote:
>>>>>
>>>>>> Regarding setting the feature freeze date to late September, I have
>>>> some
>>>>>> concern that it might make
>>>>>> the development time of 1.12 too short.
>>>>>>
>>>>>> One reason for this is we took too much time (about 1.5 month, from
>>> mid
>>>>> of
>>>>>> May to beginning of July)
>>>>>> for testing 1.11. It's not ideal but further squeeze the
>> development
>>>> time
>>>>>> of 1.12 won't make this better.
>>>>>> Besides, AFAIK July & August is also a popular vacation season for
>>>>>> European. Given the fact most
>>>>>> committers of Flink come from Europe, I think we should also take
>>> this
>>>>>> into consideration.
>>>>>>
>>>>>> It's also true that the first week of October is the national
>> holiday
>>>> of
>>>>>> China, so I'm wondering whether the
>>>>>> end of October could be a candidate feature freeze date.
>>>>>>
>>>>>> Best,
>>>>>> Kurt
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <
>> rmetzger@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Thanks a lot for the responses so far. I've put them into this
>> Wiki
>>>>> page:
>>>>>>> https://cwiki.apache.org/confluence/display/FLINK/1.12+Release
>> to
>>>> keep
>>>>>>> track of them. Ideally, post JIRA tickets for your feature, then
>>> the
>>>>>> status
>>>>>>> will update automatically in the wiki :)
>>>>>>>
>>>>>>> Please keep posting features here, or add them to the Wiki
>> yourself
>>>> 🙏
>>>>>>> @Prasanna kumar <pr...@gmail.com>: Dynamic Auto
>>>> Scaling
>>>>>> is a
>>>>>>> feature request the community is well-aware of. Till has posted
>>>>>>> "Reactive-scaling mode" as a feature he's working on for the 1.12
>>>>>> release.
>>>>>>> This work will introduce the basic building blocks and partial
>>>> support
>>>>>> for
>>>>>>> the feature you are requesting.
>>>>>>> Proper support for dynamic scaling, while maintaining Flink's
>> high
>>>>>>> performance (throughout, low latency) and correctness is a
>>> difficult
>>>>> task
>>>>>>> that needs a lot of work. It will probably take a little bit of
>>> time
>>>>> till
>>>>>>> this is fully available.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Robert
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <
>>> trohrmann@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks for being our release managers for the 1.12 release
>> Dian &
>>>>>> Robert!
>>>>>>>> Here are some features I would like to work on for this
>> release:
>>>>>>>> # Features
>>>>>>>>
>>>>>>>> ## Finishing pipelined region scheduling (
>>>>>>>> https://issues.apache.org/jira/browse/FLINK-16430)
>>>>>>>> With the pipelined region scheduler we want to implement a
>>>> scheduler
>>>>>>> which
>>>>>>>> can serve streaming as well as batch workloads alike while
>> being
>>>> able
>>>>>> to
>>>>>>>> run jobs under constrained resources. The latter is
>> particularly
>>>>>>> important
>>>>>>>> for bounded streaming jobs which, currently, are not well
>>>> supported.
>>>>>>>> ## Reactive-scaling mode
>>>>>>>> Being able to react to newly available resources and rescaling
>> a
>>>>>> running
>>>>>>>> job accordingly will make Flink's operation much easier because
>>>>>> resources
>>>>>>>> can then be controlled by an external tool (e.g. GCP
>> autoscaling,
>>>> K8s
>>>>>>>> horizontal pod scaler, etc.). In this release we want to make a
>>> big
>>>>>> step
>>>>>>>> towards this direction. As a first step we want to support the
>>>>>> execution
>>>>>>> of
>>>>>>>> jobs with a parallelism which is lower than the specified
>>>> parallelism
>>>>>> in
>>>>>>>> case that Flink lost a TaskManager or could not acquire enough
>>>>>> resources.
>>>>>>>> # Maintenance/Stability
>>>>>>>>
>>>>>>>> ## JM / TM finished task reconciliation (
>>>>>>>> https://issues.apache.org/jira/browse/FLINK-17075)
>>>>>>>> This prevents the system from going out of sync if a task state
>>>>> change
>>>>>>> from
>>>>>>>> the TM to the JM is lost.
>>>>>>>>
>>>>>>>> ## Make metrics services work with Kubernetes deployments (
>>>>>>>> https://issues.apache.org/jira/browse/FLINK-11127)
>>>>>>>> Invert the direction in which the MetricFetcher connects to the
>>>>>>>> MetricQueryFetchers. That way it will no longer be necessary to
>>>>> expose
>>>>>> on
>>>>>>>> K8s for every TaskManager a port on which the
>> MetricQueryFetcher
>>>>> runs.
>>>>>>> This
>>>>>>>> will then make the deployment of Flink clusters on K8s easier.
>>>>>>>>
>>>>>>>> ## Handle long-blocking operations during job submission
>>> (savepoint
>>>>>>>> restore) (https://issues.apache.org/jira/browse/FLINK-16866)
>>>>>>>> Submitting a Flink job can involve the interaction with
>> external
>>>>>> systems
>>>>>>>> (blocking operations). Depending on the job the interactions
>> can
>>>> take
>>>>>> so
>>>>>>>> long that it exceeds the submission timeout which reports a
>>> failure
>>>>> on
>>>>>>> the
>>>>>>>> client side even though the actual submission succeeded. By
>>>>> decoupling
>>>>>>> the
>>>>>>>> creation of the ExecutionGraph from the job submission, we can
>>> make
>>>>> the
>>>>>>> job
>>>>>>>> submission non-blocking which will solve this problem.
>>>>>>>>
>>>>>>>> ## Make IDs more intuitive to ease debugging (FLIP-118) (
>>>>>>>> https://issues.apache.org/jira/browse/FLINK-15679)
>>>>>>>> By making the internal Flink IDs compositional or logging how
>>> they
>>>>>> belong
>>>>>>>> together, we can make the debugging of Flink's operations much
>>>>> easier.
>>>>>>>> Cheers,
>>>>>>>> Till
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <
>>>> felixzhengcb@gmail.com
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> Thanks for bring-up this discussion, Robert!
>>>>>>>>> Congratulations on becoming the release manager of 1.12， Dian
>>> and
>>>>>>> Robert
>>>>>>>> !
>>>>>>>>> ----------
>>>>>>>>> Here are some of my thoughts of the features for native
>>>> integration
>>>>>>> with
>>>>>>>>> Kubernetes in Flink 1.12:
>>>>>>>>>
>>>>>>>>> 1. Support user-specified pod templates
>>>>>>>>> Description:
>>>>>>>>> The current approach of introducing new configuration
>>> options
>>>>> for
>>>>>>>> each
>>>>>>>>> aspect of pod specification a user might wish is becoming
>>>> unwieldy,
>>>>>> we
>>>>>>>> have
>>>>>>>>> to maintain more and more Flink side Kubernetes configuration
>>>>> options
>>>>>>> and
>>>>>>>>> users have to learn the gap between the declarative model
>> used
>>> by
>>>>>>>>> Kubernetes and the configuration model used by Flink. It's a
>>>> great
>>>>>>>>> improvement to allow users to specify pod templates as
>> central
>>>>> places
>>>>>>> for
>>>>>>>>> all customization needs for the jobmanager and taskmanager
>>> pods.
>>>>>>>>> Benefits:
>>>>>>>>> Users can leverage many of the advanced K8s features that
>>> the
>>>>>> Flink
>>>>>>>>> community does not support explicitly, such as volume
>> mounting,
>>>> DNS
>>>>>>>>> configuration, pod affinity/anti-affinity setting, etc.
>>>>>>>>>
>>>>>>>>> 2. Support running PyFlink on Kubernetes
>>>>>>>>> Description:
>>>>>>>>> Support running PyFlink on Kubernetes, including session
>>>>> cluster
>>>>>>> and
>>>>>>>>> application cluster.
>>>>>>>>> Benefits:
>>>>>>>>> Running python application in a containerized
>> environment.
>>>>>>>>> 3. Support built-in init-Container
>>>>>>>>> Description:
>>>>>>>>> We need a built-in init-Container to help solve
>> dependency
>>>>>>> management
>>>>>>>>> in a containerized environment, especially in the application
>>>> mode.
>>>>>>>>> Benefits:
>>>>>>>>> Separate the base Flink image from dynamic dependencies.
>>>>>>>>>
>>>>>>>>> 4. Support accessing secured services via K8s secrets
>>>>>>>>> Description:
>>>>>>>>> Kubernetes Secrets
>>>>>>>>> <https://kubernetes.io/docs/concepts/configuration/secret/>
>>> can
>>>> be
>>>>>>> used
>>>>>>>> to
>>>>>>>>> provide credentials for a Flink application to access secured
>>>>>> services.
>>>>>>>> It
>>>>>>>>> helps people who want to use a user-specified K8s Secret
>>> through
>>>> an
>>>>>>>>> environment variable.
>>>>>>>>> Benefits:
>>>>>>>>> Improve user experience.
>>>>>>>>>
>>>>>>>>> 5. Support configuring replica of JobManager Deployment in
>>>>> ZooKeeper
>>>>>> HA
>>>>>>>>> setups
>>>>>>>>> Description:
>>>>>>>>> Make the *replica* of Deployment configurable in the
>>>> ZooKeeper
>>>>> HA
>>>>>>>>> setups.
>>>>>>>>> Benefits:
>>>>>>>>> Achieve faster failover.
>>>>>>>>>
>>>>>>>>> 6. Support to configure limit for CPU requirement
>>>>>>>>> Description:
>>>>>>>>> To leverage the Kubernetes feature of container
>>> request/limit
>>>>>> CPU.
>>>>>>>>> Benefits:
>>>>>>>>> Reduce cost.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Canbin Zheng
>>>>>>>>>
>>>>>>>>> Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
>>>>>>>>>
>>>>>>>>>> I'm excited to hear about this feature, very, very, very
>>>> highly
>>>>>>>>> encouraged
>>>>>>>>>>
>>>>>>>>>> Prasanna kumar <pr...@gmail.com>
>> 于2020年7月23日周四
>>>>>>>> 上午12:10写道：
>>>>>>>>>>> Hi Flink Dev Team,
>>>>>>>>>>>
>>>>>>>>>>> Dynamic AutoScaling Based on the incoming data load would
>>> be
>>>> a
>>>>>>> great
>>>>>>>>>>> feature.
>>>>>>>>>>>
>>>>>>>>>>> We should be able have some rule say If the load
>> increased
>>> by
>>>>>> 20% ,
>>>>>>>> add
>>>>>>>>>>> extra resource should be added.
>>>>>>>>>>> Or time based say during these peak hours the pipeline
>>> should
>>>>>> scale
>>>>>>>>>>> automatically by 50%.
>>>>>>>>>>>
>>>>>>>>>>> This will help a lot in cost reduction.
>>>>>>>>>>>
>>>>>>>>>>> EMR cluster provides a similar feature for SPARK based
>>>>>> application.
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Prasanna.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <
>>>>>>> rmetzger@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> Now that the 1.11 release is out, it is time to plan
>> for
>>>> the
>>>>>> next
>>>>>>>>> major
>>>>>>>>>>>> Flink release.
>>>>>>>>>>>>
>>>>>>>>>>>> Some items:
>>>>>>>>>>>>
>>>>>>>>>>>> 1.
>>>>>>>>>>>>
>>>>>>>>>>>> Dian Fu and me volunteer to be the release managers
>>> for
>>>>>> Flink
>>>>>>>>> 1.12.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 1.
>>>>>>>>>>>>
>>>>>>>>>>>> Timeline: We propose to stick to our approximate 4
>>> month
>>>>>>> release
>>>>>>>>>>> cycle,
>>>>>>>>>>>> thus the release should be done by late October.
>> Given
>>>>> that
>>>>>>>>> there’s
>>>>>>>>>> a
>>>>>>>>>>>> holiday week in China at the beginning of October, I
>>>>> propose
>>>>>>> to
>>>>>>>> do
>>>>>>>>>> the
>>>>>>>>>>>> feature freeze on master by late September.
>>>>>>>>>>>>
>>>>>>>>>>>> 2.
>>>>>>>>>>>>
>>>>>>>>>>>> Collecting features: It would be good to have a
>> rough
>>>>>> overview
>>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>> features that will likely be ready to be merged by
>>> late
>>>>>>>> September,
>>>>>>>>>> and
>>>>>>>>>>>> that
>>>>>>>>>>>> we want in the release.
>>>>>>>>>>>> Based on the discussion, we will update the Roadmap
>> on
>>>> the
>>>>>>> Flink
>>>>>>>>>>> website
>>>>>>>>>>>> again!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 1.
>>>>>>>>>>>>
>>>>>>>>>>>> Test instabilities and blockers: I would like to
>>> avoid a
>>>>>>>> situation
>>>>>>>>>>> where
>>>>>>>>>>>> we have many blocking issues or build instabilities
>> at
>>>> the
>>>>>>> time
>>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>> feature freeze. To achieve that, we will try to
>> check
>>>>> every
>>>>>>>> build
>>>>>>>>>>>> instability within a week, to decide if it is a
>>> blocker
>>>>>> (make
>>>>>>>> sure
>>>>>>>>>> to
>>>>>>>>>>>> use
>>>>>>>>>>>> the “test-stability” label for those tickets!)
>>>>>>>>>>>> Blocker issues will need to have somebody assigned
>>>>>>> (responsible)
>>>>>>>>>>> within
>>>>>>>>>>>> a week, and we want to see progress on all blocker
>>>> issues
>>>>>>>>>> (downgrade,
>>>>>>>>>>>> resolution, a good plan how to proceed if it is more
>>>>>>>> complicated)
>>>>>>>>>>>> 2.
>>>>>>>>>>>>
>>>>>>>>>>>> Quality and stability of new features: In order to
>>> have
>>>> a
>>>>>>> short
>>>>>>>>>>> feature
>>>>>>>>>>>> freeze phase, we encourage developers to only merge
>>>>>>> well-tested
>>>>>>>>> and
>>>>>>>>>>>> documented features. In our experience, the feature
>>>> freeze
>>>>>>> works
>>>>>>>>>> best
>>>>>>>>>>> if
>>>>>>>>>>>> new features are complete, and the community can
>> focus
>>>>> fully
>>>>>>> on
>>>>>>>>>>>> addressing
>>>>>>>>>>>> newly found bugs and voting the release.
>>>>>>>>>>>> By having a smooth release process, the next
>>>> merge-window
>>>>>> for
>>>>>>>> the
>>>>>>>>>> next
>>>>>>>>>>>> release will come sooner.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Let me know what you think about our items, and share
>>> which
>>>>>>>> features
>>>>>>>>>> you
>>>>>>>>>>>> want in Flink 1.12.
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>>
>>>>>>>>>>>> Robert & Dian
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Best Regards,
>>>>>>>>>> Harold Miao
>>>>>>>>>>
Re: [DISCUSS] Planning Flink 1.12
Posted by jincheng sun <su...@gmail.com>.
+1 for end of October from me as well.
Best,
Jincheng
Kostas Kloudas <kk...@gmail.com> 于2020年8月5日周三 下午4:59写道：
> +1 for end of October from me as well.
>
> Cheers,
> Kostas
>
> On Wed, Aug 5, 2020 at 9:59 AM Till Rohrmann <tr...@apache.org> wrote:
>
> > +1 for end of October from my side as well.
> >
> > Cheers,
> > Till
> >
> > On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen <se...@apache.org> wrote:
> >
> > > The end of October sounds good from my side, unless it collides with
> some
> > > holidays that affect many committers.
> > >
> > > Feature-wise, I believe we can definitely make good use of the time to
> > wrap
> > > up some critical threads (like finishing the FLIP-27 source efforts).
> > >
> > > So +1 to the end of October from my side.
> > >
> > > Best,
> > > Stephan
> > >
> > >
> > > On Tue, Aug 4, 2020 at 8:59 AM Robert Metzger <rm...@apache.org>
> > wrote:
> > >
> > > > Thanks a lot for commenting on the feature freeze date.
> > > >
> > > > You are raising a few good points on the timing.
> > > > If we have already (2 months before) concerns regarding the deadline,
> > > then
> > > > I agree that we should move it till the end of October.
> > > >
> > > > We then just need to be careful not to run into the Christmas season
> at
> > > the
> > > > end of December.
> > > >
> > > > If nobody objects within a few days, I'll update the feature freeze
> > date
> > > in
> > > > the Wiki.
> > > >
> > > >
> > > > On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <yk...@gmail.com> wrote:
> > > >
> > > > > Regarding setting the feature freeze date to late September, I have
> > > some
> > > > > concern that it might make
> > > > > the development time of 1.12 too short.
> > > > >
> > > > > One reason for this is we took too much time (about 1.5 month, from
> > mid
> > > > of
> > > > > May to beginning of July)
> > > > > for testing 1.11. It's not ideal but further squeeze the
> development
> > > time
> > > > > of 1.12 won't make this better.
> > > > > Besides, AFAIK July & August is also a popular vacation season for
> > > > > European. Given the fact most
> > > > > committers of Flink come from Europe, I think we should also take
> > this
> > > > > into consideration.
> > > > >
> > > > > It's also true that the first week of October is the national
> holiday
> > > of
> > > > > China, so I'm wondering whether the
> > > > > end of October could be a candidate feature freeze date.
> > > > >
> > > > > Best,
> > > > > Kurt
> > > > >
> > > > >
> > > > > On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <
> rmetzger@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Thanks a lot for the responses so far. I've put them into this
> Wiki
> > > > page:
> > > > > > https://cwiki.apache.org/confluence/display/FLINK/1.12+Release
> to
> > > keep
> > > > > > track of them. Ideally, post JIRA tickets for your feature, then
> > the
> > > > > status
> > > > > > will update automatically in the wiki :)
> > > > > >
> > > > > > Please keep posting features here, or add them to the Wiki
> yourself
> > > 🙏
> > > > > >
> > > > > > @Prasanna kumar <pr...@gmail.com>: Dynamic Auto
> > > Scaling
> > > > > is a
> > > > > > feature request the community is well-aware of. Till has posted
> > > > > > "Reactive-scaling mode" as a feature he's working on for the 1.12
> > > > > release.
> > > > > > This work will introduce the basic building blocks and partial
> > > support
> > > > > for
> > > > > > the feature you are requesting.
> > > > > > Proper support for dynamic scaling, while maintaining Flink's
> high
> > > > > > performance (throughout, low latency) and correctness is a
> > difficult
> > > > task
> > > > > > that needs a lot of work. It will probably take a little bit of
> > time
> > > > till
> > > > > > this is fully available.
> > > > > >
> > > > > > Cheers,
> > > > > > Robert
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <
> > trohrmann@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for being our release managers for the 1.12 release
> Dian &
> > > > > Robert!
> > > > > > >
> > > > > > > Here are some features I would like to work on for this
> release:
> > > > > > >
> > > > > > > # Features
> > > > > > >
> > > > > > > ## Finishing pipelined region scheduling (
> > > > > > > https://issues.apache.org/jira/browse/FLINK-16430)
> > > > > > > With the pipelined region scheduler we want to implement a
> > > scheduler
> > > > > > which
> > > > > > > can serve streaming as well as batch workloads alike while
> being
> > > able
> > > > > to
> > > > > > > run jobs under constrained resources. The latter is
> particularly
> > > > > > important
> > > > > > > for bounded streaming jobs which, currently, are not well
> > > supported.
> > > > > > >
> > > > > > > ## Reactive-scaling mode
> > > > > > > Being able to react to newly available resources and rescaling
> a
> > > > > running
> > > > > > > job accordingly will make Flink's operation much easier because
> > > > > resources
> > > > > > > can then be controlled by an external tool (e.g. GCP
> autoscaling,
> > > K8s
> > > > > > > horizontal pod scaler, etc.). In this release we want to make a
> > big
> > > > > step
> > > > > > > towards this direction. As a first step we want to support the
> > > > > execution
> > > > > > of
> > > > > > > jobs with a parallelism which is lower than the specified
> > > parallelism
> > > > > in
> > > > > > > case that Flink lost a TaskManager or could not acquire enough
> > > > > resources.
> > > > > > >
> > > > > > > # Maintenance/Stability
> > > > > > >
> > > > > > > ## JM / TM finished task reconciliation (
> > > > > > > https://issues.apache.org/jira/browse/FLINK-17075)
> > > > > > > This prevents the system from going out of sync if a task state
> > > > change
> > > > > > from
> > > > > > > the TM to the JM is lost.
> > > > > > >
> > > > > > > ## Make metrics services work with Kubernetes deployments (
> > > > > > > https://issues.apache.org/jira/browse/FLINK-11127)
> > > > > > > Invert the direction in which the MetricFetcher connects to the
> > > > > > > MetricQueryFetchers. That way it will no longer be necessary to
> > > > expose
> > > > > on
> > > > > > > K8s for every TaskManager a port on which the
> MetricQueryFetcher
> > > > runs.
> > > > > > This
> > > > > > > will then make the deployment of Flink clusters on K8s easier.
> > > > > > >
> > > > > > > ## Handle long-blocking operations during job submission
> > (savepoint
> > > > > > > restore) (https://issues.apache.org/jira/browse/FLINK-16866)
> > > > > > > Submitting a Flink job can involve the interaction with
> external
> > > > > systems
> > > > > > > (blocking operations). Depending on the job the interactions
> can
> > > take
> > > > > so
> > > > > > > long that it exceeds the submission timeout which reports a
> > failure
> > > > on
> > > > > > the
> > > > > > > client side even though the actual submission succeeded. By
> > > > decoupling
> > > > > > the
> > > > > > > creation of the ExecutionGraph from the job submission, we can
> > make
> > > > the
> > > > > > job
> > > > > > > submission non-blocking which will solve this problem.
> > > > > > >
> > > > > > > ## Make IDs more intuitive to ease debugging (FLIP-118) (
> > > > > > > https://issues.apache.org/jira/browse/FLINK-15679)
> > > > > > > By making the internal Flink IDs compositional or logging how
> > they
> > > > > belong
> > > > > > > together, we can make the debugging of Flink's operations much
> > > > easier.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Till
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <
> > > felixzhengcb@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > Thanks for bring-up this discussion, Robert!
> > > > > > > > Congratulations on becoming the release manager of 1.12， Dian
> > and
> > > > > > Robert
> > > > > > > !
> > > > > > > >
> > > > > > > > ----------
> > > > > > > > Here are some of my thoughts of the features for native
> > > integration
> > > > > > with
> > > > > > > > Kubernetes in Flink 1.12:
> > > > > > > >
> > > > > > > > 1. Support user-specified pod templates
> > > > > > > > Description:
> > > > > > > > The current approach of introducing new configuration
> > options
> > > > for
> > > > > > > each
> > > > > > > > aspect of pod specification a user might wish is becoming
> > > unwieldy,
> > > > > we
> > > > > > > have
> > > > > > > > to maintain more and more Flink side Kubernetes configuration
> > > > options
> > > > > > and
> > > > > > > > users have to learn the gap between the declarative model
> used
> > by
> > > > > > > > Kubernetes and the configuration model used by Flink. It's a
> > > great
> > > > > > > > improvement to allow users to specify pod templates as
> central
> > > > places
> > > > > > for
> > > > > > > > all customization needs for the jobmanager and taskmanager
> > pods.
> > > > > > > > Benefits:
> > > > > > > > Users can leverage many of the advanced K8s features that
> > the
> > > > > Flink
> > > > > > > > community does not support explicitly, such as volume
> mounting,
> > > DNS
> > > > > > > > configuration, pod affinity/anti-affinity setting, etc.
> > > > > > > >
> > > > > > > > 2. Support running PyFlink on Kubernetes
> > > > > > > > Description:
> > > > > > > > Support running PyFlink on Kubernetes, including session
> > > > cluster
> > > > > > and
> > > > > > > > application cluster.
> > > > > > > > Benefits:
> > > > > > > > Running python application in a containerized
> environment.
> > > > > > > >
> > > > > > > > 3. Support built-in init-Container
> > > > > > > > Description:
> > > > > > > > We need a built-in init-Container to help solve
> dependency
> > > > > > management
> > > > > > > > in a containerized environment, especially in the application
> > > mode.
> > > > > > > > Benefits:
> > > > > > > > Separate the base Flink image from dynamic dependencies.
> > > > > > > >
> > > > > > > > 4. Support accessing secured services via K8s secrets
> > > > > > > > Description:
> > > > > > > > Kubernetes Secrets
> > > > > > > > <https://kubernetes.io/docs/concepts/configuration/secret/>
> > can
> > > be
> > > > > > used
> > > > > > > to
> > > > > > > > provide credentials for a Flink application to access secured
> > > > > services.
> > > > > > > It
> > > > > > > > helps people who want to use a user-specified K8s Secret
> > through
> > > an
> > > > > > > > environment variable.
> > > > > > > > Benefits:
> > > > > > > > Improve user experience.
> > > > > > > >
> > > > > > > > 5. Support configuring replica of JobManager Deployment in
> > > > ZooKeeper
> > > > > HA
> > > > > > > > setups
> > > > > > > > Description:
> > > > > > > > Make the *replica* of Deployment configurable in the
> > > ZooKeeper
> > > > HA
> > > > > > > > setups.
> > > > > > > > Benefits:
> > > > > > > > Achieve faster failover.
> > > > > > > >
> > > > > > > > 6. Support to configure limit for CPU requirement
> > > > > > > > Description:
> > > > > > > > To leverage the Kubernetes feature of container
> > request/limit
> > > > > CPU.
> > > > > > > > Benefits:
> > > > > > > > Reduce cost.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Canbin Zheng
> > > > > > > >
> > > > > > > > Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
> > > > > > > >
> > > > > > > > > I'm excited to hear about this feature, very, very, very
> > > highly
> > > > > > > > encouraged
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Prasanna kumar <pr...@gmail.com>
> 于2020年7月23日周四
> > > > > > > 上午12:10写道：
> > > > > > > > >
> > > > > > > > > > Hi Flink Dev Team,
> > > > > > > > > >
> > > > > > > > > > Dynamic AutoScaling Based on the incoming data load would
> > be
> > > a
> > > > > > great
> > > > > > > > > > feature.
> > > > > > > > > >
> > > > > > > > > > We should be able have some rule say If the load
> increased
> > by
> > > > > 20% ,
> > > > > > > add
> > > > > > > > > > extra resource should be added.
> > > > > > > > > > Or time based say during these peak hours the pipeline
> > should
> > > > > scale
> > > > > > > > > > automatically by 50%.
> > > > > > > > > >
> > > > > > > > > > This will help a lot in cost reduction.
> > > > > > > > > >
> > > > > > > > > > EMR cluster provides a similar feature for SPARK based
> > > > > application.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Prasanna.
> > > > > > > > > >
> > > > > > > > > > On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <
> > > > > > rmetzger@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > Now that the 1.11 release is out, it is time to plan
> for
> > > the
> > > > > next
> > > > > > > > major
> > > > > > > > > > > Flink release.
> > > > > > > > > > >
> > > > > > > > > > > Some items:
> > > > > > > > > > >
> > > > > > > > > > > 1.
> > > > > > > > > > >
> > > > > > > > > > > Dian Fu and me volunteer to be the release managers
> > for
> > > > > Flink
> > > > > > > > 1.12.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 1.
> > > > > > > > > > >
> > > > > > > > > > > Timeline: We propose to stick to our approximate 4
> > month
> > > > > > release
> > > > > > > > > > cycle,
> > > > > > > > > > > thus the release should be done by late October.
> Given
> > > > that
> > > > > > > > there’s
> > > > > > > > > a
> > > > > > > > > > > holiday week in China at the beginning of October, I
> > > > propose
> > > > > > to
> > > > > > > do
> > > > > > > > > the
> > > > > > > > > > > feature freeze on master by late September.
> > > > > > > > > > >
> > > > > > > > > > > 2.
> > > > > > > > > > >
> > > > > > > > > > > Collecting features: It would be good to have a
> rough
> > > > > overview
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > features that will likely be ready to be merged by
> > late
> > > > > > > September,
> > > > > > > > > and
> > > > > > > > > > > that
> > > > > > > > > > > we want in the release.
> > > > > > > > > > > Based on the discussion, we will update the Roadmap
> on
> > > the
> > > > > > Flink
> > > > > > > > > > website
> > > > > > > > > > > again!
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 1.
> > > > > > > > > > >
> > > > > > > > > > > Test instabilities and blockers: I would like to
> > avoid a
> > > > > > > situation
> > > > > > > > > > where
> > > > > > > > > > > we have many blocking issues or build instabilities
> at
> > > the
> > > > > > time
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > feature freeze. To achieve that, we will try to
> check
> > > > every
> > > > > > > build
> > > > > > > > > > > instability within a week, to decide if it is a
> > blocker
> > > > > (make
> > > > > > > sure
> > > > > > > > > to
> > > > > > > > > > > use
> > > > > > > > > > > the “test-stability” label for those tickets!)
> > > > > > > > > > > Blocker issues will need to have somebody assigned
> > > > > > (responsible)
> > > > > > > > > > within
> > > > > > > > > > > a week, and we want to see progress on all blocker
> > > issues
> > > > > > > > > (downgrade,
> > > > > > > > > > > resolution, a good plan how to proceed if it is more
> > > > > > > complicated)
> > > > > > > > > > >
> > > > > > > > > > > 2.
> > > > > > > > > > >
> > > > > > > > > > > Quality and stability of new features: In order to
> > have
> > > a
> > > > > > short
> > > > > > > > > > feature
> > > > > > > > > > > freeze phase, we encourage developers to only merge
> > > > > > well-tested
> > > > > > > > and
> > > > > > > > > > > documented features. In our experience, the feature
> > > freeze
> > > > > > works
> > > > > > > > > best
> > > > > > > > > > if
> > > > > > > > > > > new features are complete, and the community can
> focus
> > > > fully
> > > > > > on
> > > > > > > > > > > addressing
> > > > > > > > > > > newly found bugs and voting the release.
> > > > > > > > > > > By having a smooth release process, the next
> > > merge-window
> > > > > for
> > > > > > > the
> > > > > > > > > next
> > > > > > > > > > > release will come sooner.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Let me know what you think about our items, and share
> > which
> > > > > > > features
> > > > > > > > > you
> > > > > > > > > > > want in Flink 1.12.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > >
> > > > > > > > > > > Robert & Dian
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Best Regards,
> > > > > > > > > Harold Miao
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Kostas Kloudas <kk...@gmail.com>.
+1 for end of October from me as well.
Cheers,
Kostas
On Wed, Aug 5, 2020 at 9:59 AM Till Rohrmann <tr...@apache.org> wrote:
> +1 for end of October from my side as well.
>
> Cheers,
> Till
>
> On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen <se...@apache.org> wrote:
>
> > The end of October sounds good from my side, unless it collides with some
> > holidays that affect many committers.
> >
> > Feature-wise, I believe we can definitely make good use of the time to
> wrap
> > up some critical threads (like finishing the FLIP-27 source efforts).
> >
> > So +1 to the end of October from my side.
> >
> > Best,
> > Stephan
> >
> >
> > On Tue, Aug 4, 2020 at 8:59 AM Robert Metzger <rm...@apache.org>
> wrote:
> >
> > > Thanks a lot for commenting on the feature freeze date.
> > >
> > > You are raising a few good points on the timing.
> > > If we have already (2 months before) concerns regarding the deadline,
> > then
> > > I agree that we should move it till the end of October.
> > >
> > > We then just need to be careful not to run into the Christmas season at
> > the
> > > end of December.
> > >
> > > If nobody objects within a few days, I'll update the feature freeze
> date
> > in
> > > the Wiki.
> > >
> > >
> > > On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <yk...@gmail.com> wrote:
> > >
> > > > Regarding setting the feature freeze date to late September, I have
> > some
> > > > concern that it might make
> > > > the development time of 1.12 too short.
> > > >
> > > > One reason for this is we took too much time (about 1.5 month, from
> mid
> > > of
> > > > May to beginning of July)
> > > > for testing 1.11. It's not ideal but further squeeze the development
> > time
> > > > of 1.12 won't make this better.
> > > > Besides, AFAIK July & August is also a popular vacation season for
> > > > European. Given the fact most
> > > > committers of Flink come from Europe, I think we should also take
> this
> > > > into consideration.
> > > >
> > > > It's also true that the first week of October is the national holiday
> > of
> > > > China, so I'm wondering whether the
> > > > end of October could be a candidate feature freeze date.
> > > >
> > > > Best,
> > > > Kurt
> > > >
> > > >
> > > > On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <rm...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Thanks a lot for the responses so far. I've put them into this Wiki
> > > page:
> > > > > https://cwiki.apache.org/confluence/display/FLINK/1.12+Release to
> > keep
> > > > > track of them. Ideally, post JIRA tickets for your feature, then
> the
> > > > status
> > > > > will update automatically in the wiki :)
> > > > >
> > > > > Please keep posting features here, or add them to the Wiki yourself
> > 🙏
> > > > >
> > > > > @Prasanna kumar <pr...@gmail.com>: Dynamic Auto
> > Scaling
> > > > is a
> > > > > feature request the community is well-aware of. Till has posted
> > > > > "Reactive-scaling mode" as a feature he's working on for the 1.12
> > > > release.
> > > > > This work will introduce the basic building blocks and partial
> > support
> > > > for
> > > > > the feature you are requesting.
> > > > > Proper support for dynamic scaling, while maintaining Flink's high
> > > > > performance (throughout, low latency) and correctness is a
> difficult
> > > task
> > > > > that needs a lot of work. It will probably take a little bit of
> time
> > > till
> > > > > this is fully available.
> > > > >
> > > > > Cheers,
> > > > > Robert
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <
> trohrmann@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Thanks for being our release managers for the 1.12 release Dian &
> > > > Robert!
> > > > > >
> > > > > > Here are some features I would like to work on for this release:
> > > > > >
> > > > > > # Features
> > > > > >
> > > > > > ## Finishing pipelined region scheduling (
> > > > > > https://issues.apache.org/jira/browse/FLINK-16430)
> > > > > > With the pipelined region scheduler we want to implement a
> > scheduler
> > > > > which
> > > > > > can serve streaming as well as batch workloads alike while being
> > able
> > > > to
> > > > > > run jobs under constrained resources. The latter is particularly
> > > > > important
> > > > > > for bounded streaming jobs which, currently, are not well
> > supported.
> > > > > >
> > > > > > ## Reactive-scaling mode
> > > > > > Being able to react to newly available resources and rescaling a
> > > > running
> > > > > > job accordingly will make Flink's operation much easier because
> > > > resources
> > > > > > can then be controlled by an external tool (e.g. GCP autoscaling,
> > K8s
> > > > > > horizontal pod scaler, etc.). In this release we want to make a
> big
> > > > step
> > > > > > towards this direction. As a first step we want to support the
> > > > execution
> > > > > of
> > > > > > jobs with a parallelism which is lower than the specified
> > parallelism
> > > > in
> > > > > > case that Flink lost a TaskManager or could not acquire enough
> > > > resources.
> > > > > >
> > > > > > # Maintenance/Stability
> > > > > >
> > > > > > ## JM / TM finished task reconciliation (
> > > > > > https://issues.apache.org/jira/browse/FLINK-17075)
> > > > > > This prevents the system from going out of sync if a task state
> > > change
> > > > > from
> > > > > > the TM to the JM is lost.
> > > > > >
> > > > > > ## Make metrics services work with Kubernetes deployments (
> > > > > > https://issues.apache.org/jira/browse/FLINK-11127)
> > > > > > Invert the direction in which the MetricFetcher connects to the
> > > > > > MetricQueryFetchers. That way it will no longer be necessary to
> > > expose
> > > > on
> > > > > > K8s for every TaskManager a port on which the MetricQueryFetcher
> > > runs.
> > > > > This
> > > > > > will then make the deployment of Flink clusters on K8s easier.
> > > > > >
> > > > > > ## Handle long-blocking operations during job submission
> (savepoint
> > > > > > restore) (https://issues.apache.org/jira/browse/FLINK-16866)
> > > > > > Submitting a Flink job can involve the interaction with external
> > > > systems
> > > > > > (blocking operations). Depending on the job the interactions can
> > take
> > > > so
> > > > > > long that it exceeds the submission timeout which reports a
> failure
> > > on
> > > > > the
> > > > > > client side even though the actual submission succeeded. By
> > > decoupling
> > > > > the
> > > > > > creation of the ExecutionGraph from the job submission, we can
> make
> > > the
> > > > > job
> > > > > > submission non-blocking which will solve this problem.
> > > > > >
> > > > > > ## Make IDs more intuitive to ease debugging (FLIP-118) (
> > > > > > https://issues.apache.org/jira/browse/FLINK-15679)
> > > > > > By making the internal Flink IDs compositional or logging how
> they
> > > > belong
> > > > > > together, we can make the debugging of Flink's operations much
> > > easier.
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <
> > felixzhengcb@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Thanks for bring-up this discussion, Robert!
> > > > > > > Congratulations on becoming the release manager of 1.12， Dian
> and
> > > > > Robert
> > > > > > !
> > > > > > >
> > > > > > > ----------
> > > > > > > Here are some of my thoughts of the features for native
> > integration
> > > > > with
> > > > > > > Kubernetes in Flink 1.12:
> > > > > > >
> > > > > > > 1. Support user-specified pod templates
> > > > > > > Description:
> > > > > > > The current approach of introducing new configuration
> options
> > > for
> > > > > > each
> > > > > > > aspect of pod specification a user might wish is becoming
> > unwieldy,
> > > > we
> > > > > > have
> > > > > > > to maintain more and more Flink side Kubernetes configuration
> > > options
> > > > > and
> > > > > > > users have to learn the gap between the declarative model used
> by
> > > > > > > Kubernetes and the configuration model used by Flink. It's a
> > great
> > > > > > > improvement to allow users to specify pod templates as central
> > > places
> > > > > for
> > > > > > > all customization needs for the jobmanager and taskmanager
> pods.
> > > > > > > Benefits:
> > > > > > > Users can leverage many of the advanced K8s features that
> the
> > > > Flink
> > > > > > > community does not support explicitly, such as volume mounting,
> > DNS
> > > > > > > configuration, pod affinity/anti-affinity setting, etc.
> > > > > > >
> > > > > > > 2. Support running PyFlink on Kubernetes
> > > > > > > Description:
> > > > > > > Support running PyFlink on Kubernetes, including session
> > > cluster
> > > > > and
> > > > > > > application cluster.
> > > > > > > Benefits:
> > > > > > > Running python application in a containerized environment.
> > > > > > >
> > > > > > > 3. Support built-in init-Container
> > > > > > > Description:
> > > > > > > We need a built-in init-Container to help solve dependency
> > > > > management
> > > > > > > in a containerized environment, especially in the application
> > mode.
> > > > > > > Benefits:
> > > > > > > Separate the base Flink image from dynamic dependencies.
> > > > > > >
> > > > > > > 4. Support accessing secured services via K8s secrets
> > > > > > > Description:
> > > > > > > Kubernetes Secrets
> > > > > > > <https://kubernetes.io/docs/concepts/configuration/secret/>
> can
> > be
> > > > > used
> > > > > > to
> > > > > > > provide credentials for a Flink application to access secured
> > > > services.
> > > > > > It
> > > > > > > helps people who want to use a user-specified K8s Secret
> through
> > an
> > > > > > > environment variable.
> > > > > > > Benefits:
> > > > > > > Improve user experience.
> > > > > > >
> > > > > > > 5. Support configuring replica of JobManager Deployment in
> > > ZooKeeper
> > > > HA
> > > > > > > setups
> > > > > > > Description:
> > > > > > > Make the *replica* of Deployment configurable in the
> > ZooKeeper
> > > HA
> > > > > > > setups.
> > > > > > > Benefits:
> > > > > > > Achieve faster failover.
> > > > > > >
> > > > > > > 6. Support to configure limit for CPU requirement
> > > > > > > Description:
> > > > > > > To leverage the Kubernetes feature of container
> request/limit
> > > > CPU.
> > > > > > > Benefits:
> > > > > > > Reduce cost.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Canbin Zheng
> > > > > > >
> > > > > > > Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
> > > > > > >
> > > > > > > > I'm excited to hear about this feature, very, very, very
> > highly
> > > > > > > encouraged
> > > > > > > >
> > > > > > > >
> > > > > > > > Prasanna kumar <pr...@gmail.com> 于2020年7月23日周四
> > > > > > 上午12:10写道：
> > > > > > > >
> > > > > > > > > Hi Flink Dev Team,
> > > > > > > > >
> > > > > > > > > Dynamic AutoScaling Based on the incoming data load would
> be
> > a
> > > > > great
> > > > > > > > > feature.
> > > > > > > > >
> > > > > > > > > We should be able have some rule say If the load increased
> by
> > > > 20% ,
> > > > > > add
> > > > > > > > > extra resource should be added.
> > > > > > > > > Or time based say during these peak hours the pipeline
> should
> > > > scale
> > > > > > > > > automatically by 50%.
> > > > > > > > >
> > > > > > > > > This will help a lot in cost reduction.
> > > > > > > > >
> > > > > > > > > EMR cluster provides a similar feature for SPARK based
> > > > application.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Prasanna.
> > > > > > > > >
> > > > > > > > > On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <
> > > > > rmetzger@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > Now that the 1.11 release is out, it is time to plan for
> > the
> > > > next
> > > > > > > major
> > > > > > > > > > Flink release.
> > > > > > > > > >
> > > > > > > > > > Some items:
> > > > > > > > > >
> > > > > > > > > > 1.
> > > > > > > > > >
> > > > > > > > > > Dian Fu and me volunteer to be the release managers
> for
> > > > Flink
> > > > > > > 1.12.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 1.
> > > > > > > > > >
> > > > > > > > > > Timeline: We propose to stick to our approximate 4
> month
> > > > > release
> > > > > > > > > cycle,
> > > > > > > > > > thus the release should be done by late October. Given
> > > that
> > > > > > > there’s
> > > > > > > > a
> > > > > > > > > > holiday week in China at the beginning of October, I
> > > propose
> > > > > to
> > > > > > do
> > > > > > > > the
> > > > > > > > > > feature freeze on master by late September.
> > > > > > > > > >
> > > > > > > > > > 2.
> > > > > > > > > >
> > > > > > > > > > Collecting features: It would be good to have a rough
> > > > overview
> > > > > > of
> > > > > > > > the
> > > > > > > > > > features that will likely be ready to be merged by
> late
> > > > > > September,
> > > > > > > > and
> > > > > > > > > > that
> > > > > > > > > > we want in the release.
> > > > > > > > > > Based on the discussion, we will update the Roadmap on
> > the
> > > > > Flink
> > > > > > > > > website
> > > > > > > > > > again!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 1.
> > > > > > > > > >
> > > > > > > > > > Test instabilities and blockers: I would like to
> avoid a
> > > > > > situation
> > > > > > > > > where
> > > > > > > > > > we have many blocking issues or build instabilities at
> > the
> > > > > time
> > > > > > of
> > > > > > > > the
> > > > > > > > > > feature freeze. To achieve that, we will try to check
> > > every
> > > > > > build
> > > > > > > > > > instability within a week, to decide if it is a
> blocker
> > > > (make
> > > > > > sure
> > > > > > > > to
> > > > > > > > > > use
> > > > > > > > > > the “test-stability” label for those tickets!)
> > > > > > > > > > Blocker issues will need to have somebody assigned
> > > > > (responsible)
> > > > > > > > > within
> > > > > > > > > > a week, and we want to see progress on all blocker
> > issues
> > > > > > > > (downgrade,
> > > > > > > > > > resolution, a good plan how to proceed if it is more
> > > > > > complicated)
> > > > > > > > > >
> > > > > > > > > > 2.
> > > > > > > > > >
> > > > > > > > > > Quality and stability of new features: In order to
> have
> > a
> > > > > short
> > > > > > > > > feature
> > > > > > > > > > freeze phase, we encourage developers to only merge
> > > > > well-tested
> > > > > > > and
> > > > > > > > > > documented features. In our experience, the feature
> > freeze
> > > > > works
> > > > > > > > best
> > > > > > > > > if
> > > > > > > > > > new features are complete, and the community can focus
> > > fully
> > > > > on
> > > > > > > > > > addressing
> > > > > > > > > > newly found bugs and voting the release.
> > > > > > > > > > By having a smooth release process, the next
> > merge-window
> > > > for
> > > > > > the
> > > > > > > > next
> > > > > > > > > > release will come sooner.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Let me know what you think about our items, and share
> which
> > > > > > features
> > > > > > > > you
> > > > > > > > > > want in Flink 1.12.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > Robert & Dian
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Harold Miao
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Till Rohrmann <tr...@apache.org>.
+1 for end of October from my side as well.
Cheers,
Till
On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen <se...@apache.org> wrote:
> The end of October sounds good from my side, unless it collides with some
> holidays that affect many committers.
>
> Feature-wise, I believe we can definitely make good use of the time to wrap
> up some critical threads (like finishing the FLIP-27 source efforts).
>
> So +1 to the end of October from my side.
>
> Best,
> Stephan
>
>
> On Tue, Aug 4, 2020 at 8:59 AM Robert Metzger <rm...@apache.org> wrote:
>
> > Thanks a lot for commenting on the feature freeze date.
> >
> > You are raising a few good points on the timing.
> > If we have already (2 months before) concerns regarding the deadline,
> then
> > I agree that we should move it till the end of October.
> >
> > We then just need to be careful not to run into the Christmas season at
> the
> > end of December.
> >
> > If nobody objects within a few days, I'll update the feature freeze date
> in
> > the Wiki.
> >
> >
> > On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <yk...@gmail.com> wrote:
> >
> > > Regarding setting the feature freeze date to late September, I have
> some
> > > concern that it might make
> > > the development time of 1.12 too short.
> > >
> > > One reason for this is we took too much time (about 1.5 month, from mid
> > of
> > > May to beginning of July)
> > > for testing 1.11. It's not ideal but further squeeze the development
> time
> > > of 1.12 won't make this better.
> > > Besides, AFAIK July & August is also a popular vacation season for
> > > European. Given the fact most
> > > committers of Flink come from Europe, I think we should also take this
> > > into consideration.
> > >
> > > It's also true that the first week of October is the national holiday
> of
> > > China, so I'm wondering whether the
> > > end of October could be a candidate feature freeze date.
> > >
> > > Best,
> > > Kurt
> > >
> > >
> > > On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <rm...@apache.org>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Thanks a lot for the responses so far. I've put them into this Wiki
> > page:
> > > > https://cwiki.apache.org/confluence/display/FLINK/1.12+Release to
> keep
> > > > track of them. Ideally, post JIRA tickets for your feature, then the
> > > status
> > > > will update automatically in the wiki :)
> > > >
> > > > Please keep posting features here, or add them to the Wiki yourself
> 🙏
> > > >
> > > > @Prasanna kumar <pr...@gmail.com>: Dynamic Auto
> Scaling
> > > is a
> > > > feature request the community is well-aware of. Till has posted
> > > > "Reactive-scaling mode" as a feature he's working on for the 1.12
> > > release.
> > > > This work will introduce the basic building blocks and partial
> support
> > > for
> > > > the feature you are requesting.
> > > > Proper support for dynamic scaling, while maintaining Flink's high
> > > > performance (throughout, low latency) and correctness is a difficult
> > task
> > > > that needs a lot of work. It will probably take a little bit of time
> > till
> > > > this is fully available.
> > > >
> > > > Cheers,
> > > > Robert
> > > >
> > > >
> > > >
> > > > On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <tr...@apache.org>
> > > > wrote:
> > > >
> > > > > Thanks for being our release managers for the 1.12 release Dian &
> > > Robert!
> > > > >
> > > > > Here are some features I would like to work on for this release:
> > > > >
> > > > > # Features
> > > > >
> > > > > ## Finishing pipelined region scheduling (
> > > > > https://issues.apache.org/jira/browse/FLINK-16430)
> > > > > With the pipelined region scheduler we want to implement a
> scheduler
> > > > which
> > > > > can serve streaming as well as batch workloads alike while being
> able
> > > to
> > > > > run jobs under constrained resources. The latter is particularly
> > > > important
> > > > > for bounded streaming jobs which, currently, are not well
> supported.
> > > > >
> > > > > ## Reactive-scaling mode
> > > > > Being able to react to newly available resources and rescaling a
> > > running
> > > > > job accordingly will make Flink's operation much easier because
> > > resources
> > > > > can then be controlled by an external tool (e.g. GCP autoscaling,
> K8s
> > > > > horizontal pod scaler, etc.). In this release we want to make a big
> > > step
> > > > > towards this direction. As a first step we want to support the
> > > execution
> > > > of
> > > > > jobs with a parallelism which is lower than the specified
> parallelism
> > > in
> > > > > case that Flink lost a TaskManager or could not acquire enough
> > > resources.
> > > > >
> > > > > # Maintenance/Stability
> > > > >
> > > > > ## JM / TM finished task reconciliation (
> > > > > https://issues.apache.org/jira/browse/FLINK-17075)
> > > > > This prevents the system from going out of sync if a task state
> > change
> > > > from
> > > > > the TM to the JM is lost.
> > > > >
> > > > > ## Make metrics services work with Kubernetes deployments (
> > > > > https://issues.apache.org/jira/browse/FLINK-11127)
> > > > > Invert the direction in which the MetricFetcher connects to the
> > > > > MetricQueryFetchers. That way it will no longer be necessary to
> > expose
> > > on
> > > > > K8s for every TaskManager a port on which the MetricQueryFetcher
> > runs.
> > > > This
> > > > > will then make the deployment of Flink clusters on K8s easier.
> > > > >
> > > > > ## Handle long-blocking operations during job submission (savepoint
> > > > > restore) (https://issues.apache.org/jira/browse/FLINK-16866)
> > > > > Submitting a Flink job can involve the interaction with external
> > > systems
> > > > > (blocking operations). Depending on the job the interactions can
> take
> > > so
> > > > > long that it exceeds the submission timeout which reports a failure
> > on
> > > > the
> > > > > client side even though the actual submission succeeded. By
> > decoupling
> > > > the
> > > > > creation of the ExecutionGraph from the job submission, we can make
> > the
> > > > job
> > > > > submission non-blocking which will solve this problem.
> > > > >
> > > > > ## Make IDs more intuitive to ease debugging (FLIP-118) (
> > > > > https://issues.apache.org/jira/browse/FLINK-15679)
> > > > > By making the internal Flink IDs compositional or logging how they
> > > belong
> > > > > together, we can make the debugging of Flink's operations much
> > easier.
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > >
> > > > > On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <
> felixzhengcb@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Thanks for bring-up this discussion, Robert!
> > > > > > Congratulations on becoming the release manager of 1.12， Dian and
> > > > Robert
> > > > > !
> > > > > >
> > > > > > ----------
> > > > > > Here are some of my thoughts of the features for native
> integration
> > > > with
> > > > > > Kubernetes in Flink 1.12:
> > > > > >
> > > > > > 1. Support user-specified pod templates
> > > > > > Description:
> > > > > > The current approach of introducing new configuration options
> > for
> > > > > each
> > > > > > aspect of pod specification a user might wish is becoming
> unwieldy,
> > > we
> > > > > have
> > > > > > to maintain more and more Flink side Kubernetes configuration
> > options
> > > > and
> > > > > > users have to learn the gap between the declarative model used by
> > > > > > Kubernetes and the configuration model used by Flink. It's a
> great
> > > > > > improvement to allow users to specify pod templates as central
> > places
> > > > for
> > > > > > all customization needs for the jobmanager and taskmanager pods.
> > > > > > Benefits:
> > > > > > Users can leverage many of the advanced K8s features that the
> > > Flink
> > > > > > community does not support explicitly, such as volume mounting,
> DNS
> > > > > > configuration, pod affinity/anti-affinity setting, etc.
> > > > > >
> > > > > > 2. Support running PyFlink on Kubernetes
> > > > > > Description:
> > > > > > Support running PyFlink on Kubernetes, including session
> > cluster
> > > > and
> > > > > > application cluster.
> > > > > > Benefits:
> > > > > > Running python application in a containerized environment.
> > > > > >
> > > > > > 3. Support built-in init-Container
> > > > > > Description:
> > > > > > We need a built-in init-Container to help solve dependency
> > > > management
> > > > > > in a containerized environment, especially in the application
> mode.
> > > > > > Benefits:
> > > > > > Separate the base Flink image from dynamic dependencies.
> > > > > >
> > > > > > 4. Support accessing secured services via K8s secrets
> > > > > > Description:
> > > > > > Kubernetes Secrets
> > > > > > <https://kubernetes.io/docs/concepts/configuration/secret/> can
> be
> > > > used
> > > > > to
> > > > > > provide credentials for a Flink application to access secured
> > > services.
> > > > > It
> > > > > > helps people who want to use a user-specified K8s Secret through
> an
> > > > > > environment variable.
> > > > > > Benefits:
> > > > > > Improve user experience.
> > > > > >
> > > > > > 5. Support configuring replica of JobManager Deployment in
> > ZooKeeper
> > > HA
> > > > > > setups
> > > > > > Description:
> > > > > > Make the *replica* of Deployment configurable in the
> ZooKeeper
> > HA
> > > > > > setups.
> > > > > > Benefits:
> > > > > > Achieve faster failover.
> > > > > >
> > > > > > 6. Support to configure limit for CPU requirement
> > > > > > Description:
> > > > > > To leverage the Kubernetes feature of container request/limit
> > > CPU.
> > > > > > Benefits:
> > > > > > Reduce cost.
> > > > > >
> > > > > > Regards,
> > > > > > Canbin Zheng
> > > > > >
> > > > > > Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
> > > > > >
> > > > > > > I'm excited to hear about this feature, very, very, very
> highly
> > > > > > encouraged
> > > > > > >
> > > > > > >
> > > > > > > Prasanna kumar <pr...@gmail.com> 于2020年7月23日周四
> > > > > 上午12:10写道：
> > > > > > >
> > > > > > > > Hi Flink Dev Team,
> > > > > > > >
> > > > > > > > Dynamic AutoScaling Based on the incoming data load would be
> a
> > > > great
> > > > > > > > feature.
> > > > > > > >
> > > > > > > > We should be able have some rule say If the load increased by
> > > 20% ,
> > > > > add
> > > > > > > > extra resource should be added.
> > > > > > > > Or time based say during these peak hours the pipeline should
> > > scale
> > > > > > > > automatically by 50%.
> > > > > > > >
> > > > > > > > This will help a lot in cost reduction.
> > > > > > > >
> > > > > > > > EMR cluster provides a similar feature for SPARK based
> > > application.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Prasanna.
> > > > > > > >
> > > > > > > > On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <
> > > > rmetzger@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > Now that the 1.11 release is out, it is time to plan for
> the
> > > next
> > > > > > major
> > > > > > > > > Flink release.
> > > > > > > > >
> > > > > > > > > Some items:
> > > > > > > > >
> > > > > > > > > 1.
> > > > > > > > >
> > > > > > > > > Dian Fu and me volunteer to be the release managers for
> > > Flink
> > > > > > 1.12.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 1.
> > > > > > > > >
> > > > > > > > > Timeline: We propose to stick to our approximate 4 month
> > > > release
> > > > > > > > cycle,
> > > > > > > > > thus the release should be done by late October. Given
> > that
> > > > > > there’s
> > > > > > > a
> > > > > > > > > holiday week in China at the beginning of October, I
> > propose
> > > > to
> > > > > do
> > > > > > > the
> > > > > > > > > feature freeze on master by late September.
> > > > > > > > >
> > > > > > > > > 2.
> > > > > > > > >
> > > > > > > > > Collecting features: It would be good to have a rough
> > > overview
> > > > > of
> > > > > > > the
> > > > > > > > > features that will likely be ready to be merged by late
> > > > > September,
> > > > > > > and
> > > > > > > > > that
> > > > > > > > > we want in the release.
> > > > > > > > > Based on the discussion, we will update the Roadmap on
> the
> > > > Flink
> > > > > > > > website
> > > > > > > > > again!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 1.
> > > > > > > > >
> > > > > > > > > Test instabilities and blockers: I would like to avoid a
> > > > > situation
> > > > > > > > where
> > > > > > > > > we have many blocking issues or build instabilities at
> the
> > > > time
> > > > > of
> > > > > > > the
> > > > > > > > > feature freeze. To achieve that, we will try to check
> > every
> > > > > build
> > > > > > > > > instability within a week, to decide if it is a blocker
> > > (make
> > > > > sure
> > > > > > > to
> > > > > > > > > use
> > > > > > > > > the “test-stability” label for those tickets!)
> > > > > > > > > Blocker issues will need to have somebody assigned
> > > > (responsible)
> > > > > > > > within
> > > > > > > > > a week, and we want to see progress on all blocker
> issues
> > > > > > > (downgrade,
> > > > > > > > > resolution, a good plan how to proceed if it is more
> > > > > complicated)
> > > > > > > > >
> > > > > > > > > 2.
> > > > > > > > >
> > > > > > > > > Quality and stability of new features: In order to have
> a
> > > > short
> > > > > > > > feature
> > > > > > > > > freeze phase, we encourage developers to only merge
> > > > well-tested
> > > > > > and
> > > > > > > > > documented features. In our experience, the feature
> freeze
> > > > works
> > > > > > > best
> > > > > > > > if
> > > > > > > > > new features are complete, and the community can focus
> > fully
> > > > on
> > > > > > > > > addressing
> > > > > > > > > newly found bugs and voting the release.
> > > > > > > > > By having a smooth release process, the next
> merge-window
> > > for
> > > > > the
> > > > > > > next
> > > > > > > > > release will come sooner.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Let me know what you think about our items, and share which
> > > > > features
> > > > > > > you
> > > > > > > > > want in Flink 1.12.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > Robert & Dian
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Harold Miao
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Stephan Ewen <se...@apache.org>.
The end of October sounds good from my side, unless it collides with some
holidays that affect many committers.
Feature-wise, I believe we can definitely make good use of the time to wrap
up some critical threads (like finishing the FLIP-27 source efforts).
So +1 to the end of October from my side.
Best,
Stephan
On Tue, Aug 4, 2020 at 8:59 AM Robert Metzger <rm...@apache.org> wrote:
> Thanks a lot for commenting on the feature freeze date.
>
> You are raising a few good points on the timing.
> If we have already (2 months before) concerns regarding the deadline, then
> I agree that we should move it till the end of October.
>
> We then just need to be careful not to run into the Christmas season at the
> end of December.
>
> If nobody objects within a few days, I'll update the feature freeze date in
> the Wiki.
>
>
> On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <yk...@gmail.com> wrote:
>
> > Regarding setting the feature freeze date to late September, I have some
> > concern that it might make
> > the development time of 1.12 too short.
> >
> > One reason for this is we took too much time (about 1.5 month, from mid
> of
> > May to beginning of July)
> > for testing 1.11. It's not ideal but further squeeze the development time
> > of 1.12 won't make this better.
> > Besides, AFAIK July & August is also a popular vacation season for
> > European. Given the fact most
> > committers of Flink come from Europe, I think we should also take this
> > into consideration.
> >
> > It's also true that the first week of October is the national holiday of
> > China, so I'm wondering whether the
> > end of October could be a candidate feature freeze date.
> >
> > Best,
> > Kurt
> >
> >
> > On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <rm...@apache.org>
> > wrote:
> >
> > > Hi all,
> > >
> > > Thanks a lot for the responses so far. I've put them into this Wiki
> page:
> > > https://cwiki.apache.org/confluence/display/FLINK/1.12+Release to keep
> > > track of them. Ideally, post JIRA tickets for your feature, then the
> > status
> > > will update automatically in the wiki :)
> > >
> > > Please keep posting features here, or add them to the Wiki yourself 🙏
> > >
> > > @Prasanna kumar <pr...@gmail.com>: Dynamic Auto Scaling
> > is a
> > > feature request the community is well-aware of. Till has posted
> > > "Reactive-scaling mode" as a feature he's working on for the 1.12
> > release.
> > > This work will introduce the basic building blocks and partial support
> > for
> > > the feature you are requesting.
> > > Proper support for dynamic scaling, while maintaining Flink's high
> > > performance (throughout, low latency) and correctness is a difficult
> task
> > > that needs a lot of work. It will probably take a little bit of time
> till
> > > this is fully available.
> > >
> > > Cheers,
> > > Robert
> > >
> > >
> > >
> > > On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <tr...@apache.org>
> > > wrote:
> > >
> > > > Thanks for being our release managers for the 1.12 release Dian &
> > Robert!
> > > >
> > > > Here are some features I would like to work on for this release:
> > > >
> > > > # Features
> > > >
> > > > ## Finishing pipelined region scheduling (
> > > > https://issues.apache.org/jira/browse/FLINK-16430)
> > > > With the pipelined region scheduler we want to implement a scheduler
> > > which
> > > > can serve streaming as well as batch workloads alike while being able
> > to
> > > > run jobs under constrained resources. The latter is particularly
> > > important
> > > > for bounded streaming jobs which, currently, are not well supported.
> > > >
> > > > ## Reactive-scaling mode
> > > > Being able to react to newly available resources and rescaling a
> > running
> > > > job accordingly will make Flink's operation much easier because
> > resources
> > > > can then be controlled by an external tool (e.g. GCP autoscaling, K8s
> > > > horizontal pod scaler, etc.). In this release we want to make a big
> > step
> > > > towards this direction. As a first step we want to support the
> > execution
> > > of
> > > > jobs with a parallelism which is lower than the specified parallelism
> > in
> > > > case that Flink lost a TaskManager or could not acquire enough
> > resources.
> > > >
> > > > # Maintenance/Stability
> > > >
> > > > ## JM / TM finished task reconciliation (
> > > > https://issues.apache.org/jira/browse/FLINK-17075)
> > > > This prevents the system from going out of sync if a task state
> change
> > > from
> > > > the TM to the JM is lost.
> > > >
> > > > ## Make metrics services work with Kubernetes deployments (
> > > > https://issues.apache.org/jira/browse/FLINK-11127)
> > > > Invert the direction in which the MetricFetcher connects to the
> > > > MetricQueryFetchers. That way it will no longer be necessary to
> expose
> > on
> > > > K8s for every TaskManager a port on which the MetricQueryFetcher
> runs.
> > > This
> > > > will then make the deployment of Flink clusters on K8s easier.
> > > >
> > > > ## Handle long-blocking operations during job submission (savepoint
> > > > restore) (https://issues.apache.org/jira/browse/FLINK-16866)
> > > > Submitting a Flink job can involve the interaction with external
> > systems
> > > > (blocking operations). Depending on the job the interactions can take
> > so
> > > > long that it exceeds the submission timeout which reports a failure
> on
> > > the
> > > > client side even though the actual submission succeeded. By
> decoupling
> > > the
> > > > creation of the ExecutionGraph from the job submission, we can make
> the
> > > job
> > > > submission non-blocking which will solve this problem.
> > > >
> > > > ## Make IDs more intuitive to ease debugging (FLIP-118) (
> > > > https://issues.apache.org/jira/browse/FLINK-15679)
> > > > By making the internal Flink IDs compositional or logging how they
> > belong
> > > > together, we can make the debugging of Flink's operations much
> easier.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > >
> > > > On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <felixzhengcb@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Thanks for bring-up this discussion, Robert!
> > > > > Congratulations on becoming the release manager of 1.12， Dian and
> > > Robert
> > > > !
> > > > >
> > > > > ----------
> > > > > Here are some of my thoughts of the features for native integration
> > > with
> > > > > Kubernetes in Flink 1.12:
> > > > >
> > > > > 1. Support user-specified pod templates
> > > > > Description:
> > > > > The current approach of introducing new configuration options
> for
> > > > each
> > > > > aspect of pod specification a user might wish is becoming unwieldy,
> > we
> > > > have
> > > > > to maintain more and more Flink side Kubernetes configuration
> options
> > > and
> > > > > users have to learn the gap between the declarative model used by
> > > > > Kubernetes and the configuration model used by Flink. It's a great
> > > > > improvement to allow users to specify pod templates as central
> places
> > > for
> > > > > all customization needs for the jobmanager and taskmanager pods.
> > > > > Benefits:
> > > > > Users can leverage many of the advanced K8s features that the
> > Flink
> > > > > community does not support explicitly, such as volume mounting, DNS
> > > > > configuration, pod affinity/anti-affinity setting, etc.
> > > > >
> > > > > 2. Support running PyFlink on Kubernetes
> > > > > Description:
> > > > > Support running PyFlink on Kubernetes, including session
> cluster
> > > and
> > > > > application cluster.
> > > > > Benefits:
> > > > > Running python application in a containerized environment.
> > > > >
> > > > > 3. Support built-in init-Container
> > > > > Description:
> > > > > We need a built-in init-Container to help solve dependency
> > > management
> > > > > in a containerized environment, especially in the application mode.
> > > > > Benefits:
> > > > > Separate the base Flink image from dynamic dependencies.
> > > > >
> > > > > 4. Support accessing secured services via K8s secrets
> > > > > Description:
> > > > > Kubernetes Secrets
> > > > > <https://kubernetes.io/docs/concepts/configuration/secret/> can be
> > > used
> > > > to
> > > > > provide credentials for a Flink application to access secured
> > services.
> > > > It
> > > > > helps people who want to use a user-specified K8s Secret through an
> > > > > environment variable.
> > > > > Benefits:
> > > > > Improve user experience.
> > > > >
> > > > > 5. Support configuring replica of JobManager Deployment in
> ZooKeeper
> > HA
> > > > > setups
> > > > > Description:
> > > > > Make the *replica* of Deployment configurable in the ZooKeeper
> HA
> > > > > setups.
> > > > > Benefits:
> > > > > Achieve faster failover.
> > > > >
> > > > > 6. Support to configure limit for CPU requirement
> > > > > Description:
> > > > > To leverage the Kubernetes feature of container request/limit
> > CPU.
> > > > > Benefits:
> > > > > Reduce cost.
> > > > >
> > > > > Regards,
> > > > > Canbin Zheng
> > > > >
> > > > > Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
> > > > >
> > > > > > I'm excited to hear about this feature, very, very, very highly
> > > > > encouraged
> > > > > >
> > > > > >
> > > > > > Prasanna kumar <pr...@gmail.com> 于2020年7月23日周四
> > > > 上午12:10写道：
> > > > > >
> > > > > > > Hi Flink Dev Team,
> > > > > > >
> > > > > > > Dynamic AutoScaling Based on the incoming data load would be a
> > > great
> > > > > > > feature.
> > > > > > >
> > > > > > > We should be able have some rule say If the load increased by
> > 20% ,
> > > > add
> > > > > > > extra resource should be added.
> > > > > > > Or time based say during these peak hours the pipeline should
> > scale
> > > > > > > automatically by 50%.
> > > > > > >
> > > > > > > This will help a lot in cost reduction.
> > > > > > >
> > > > > > > EMR cluster provides a similar feature for SPARK based
> > application.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Prasanna.
> > > > > > >
> > > > > > > On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <
> > > rmetzger@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Now that the 1.11 release is out, it is time to plan for the
> > next
> > > > > major
> > > > > > > > Flink release.
> > > > > > > >
> > > > > > > > Some items:
> > > > > > > >
> > > > > > > > 1.
> > > > > > > >
> > > > > > > > Dian Fu and me volunteer to be the release managers for
> > Flink
> > > > > 1.12.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 1.
> > > > > > > >
> > > > > > > > Timeline: We propose to stick to our approximate 4 month
> > > release
> > > > > > > cycle,
> > > > > > > > thus the release should be done by late October. Given
> that
> > > > > there’s
> > > > > > a
> > > > > > > > holiday week in China at the beginning of October, I
> propose
> > > to
> > > > do
> > > > > > the
> > > > > > > > feature freeze on master by late September.
> > > > > > > >
> > > > > > > > 2.
> > > > > > > >
> > > > > > > > Collecting features: It would be good to have a rough
> > overview
> > > > of
> > > > > > the
> > > > > > > > features that will likely be ready to be merged by late
> > > > September,
> > > > > > and
> > > > > > > > that
> > > > > > > > we want in the release.
> > > > > > > > Based on the discussion, we will update the Roadmap on the
> > > Flink
> > > > > > > website
> > > > > > > > again!
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 1.
> > > > > > > >
> > > > > > > > Test instabilities and blockers: I would like to avoid a
> > > > situation
> > > > > > > where
> > > > > > > > we have many blocking issues or build instabilities at the
> > > time
> > > > of
> > > > > > the
> > > > > > > > feature freeze. To achieve that, we will try to check
> every
> > > > build
> > > > > > > > instability within a week, to decide if it is a blocker
> > (make
> > > > sure
> > > > > > to
> > > > > > > > use
> > > > > > > > the “test-stability” label for those tickets!)
> > > > > > > > Blocker issues will need to have somebody assigned
> > > (responsible)
> > > > > > > within
> > > > > > > > a week, and we want to see progress on all blocker issues
> > > > > > (downgrade,
> > > > > > > > resolution, a good plan how to proceed if it is more
> > > > complicated)
> > > > > > > >
> > > > > > > > 2.
> > > > > > > >
> > > > > > > > Quality and stability of new features: In order to have a
> > > short
> > > > > > > feature
> > > > > > > > freeze phase, we encourage developers to only merge
> > > well-tested
> > > > > and
> > > > > > > > documented features. In our experience, the feature freeze
> > > works
> > > > > > best
> > > > > > > if
> > > > > > > > new features are complete, and the community can focus
> fully
> > > on
> > > > > > > > addressing
> > > > > > > > newly found bugs and voting the release.
> > > > > > > > By having a smooth release process, the next merge-window
> > for
> > > > the
> > > > > > next
> > > > > > > > release will come sooner.
> > > > > > > >
> > > > > > > >
> > > > > > > > Let me know what you think about our items, and share which
> > > > features
> > > > > > you
> > > > > > > > want in Flink 1.12.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > Robert & Dian
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Best Regards,
> > > > > > Harold Miao
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Robert Metzger <rm...@apache.org>.
Thanks a lot for commenting on the feature freeze date.
You are raising a few good points on the timing.
If we have already (2 months before) concerns regarding the deadline, then
I agree that we should move it till the end of October.
We then just need to be careful not to run into the Christmas season at the
end of December.
If nobody objects within a few days, I'll update the feature freeze date in
the Wiki.
On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <yk...@gmail.com> wrote:
> Regarding setting the feature freeze date to late September, I have some
> concern that it might make
> the development time of 1.12 too short.
>
> One reason for this is we took too much time (about 1.5 month, from mid of
> May to beginning of July)
> for testing 1.11. It's not ideal but further squeeze the development time
> of 1.12 won't make this better.
> Besides, AFAIK July & August is also a popular vacation season for
> European. Given the fact most
> committers of Flink come from Europe, I think we should also take this
> into consideration.
>
> It's also true that the first week of October is the national holiday of
> China, so I'm wondering whether the
> end of October could be a candidate feature freeze date.
>
> Best,
> Kurt
>
>
> On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <rm...@apache.org>
> wrote:
>
> > Hi all,
> >
> > Thanks a lot for the responses so far. I've put them into this Wiki page:
> > https://cwiki.apache.org/confluence/display/FLINK/1.12+Release to keep
> > track of them. Ideally, post JIRA tickets for your feature, then the
> status
> > will update automatically in the wiki :)
> >
> > Please keep posting features here, or add them to the Wiki yourself 🙏
> >
> > @Prasanna kumar <pr...@gmail.com>: Dynamic Auto Scaling
> is a
> > feature request the community is well-aware of. Till has posted
> > "Reactive-scaling mode" as a feature he's working on for the 1.12
> release.
> > This work will introduce the basic building blocks and partial support
> for
> > the feature you are requesting.
> > Proper support for dynamic scaling, while maintaining Flink's high
> > performance (throughout, low latency) and correctness is a difficult task
> > that needs a lot of work. It will probably take a little bit of time till
> > this is fully available.
> >
> > Cheers,
> > Robert
> >
> >
> >
> > On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> > > Thanks for being our release managers for the 1.12 release Dian &
> Robert!
> > >
> > > Here are some features I would like to work on for this release:
> > >
> > > # Features
> > >
> > > ## Finishing pipelined region scheduling (
> > > https://issues.apache.org/jira/browse/FLINK-16430)
> > > With the pipelined region scheduler we want to implement a scheduler
> > which
> > > can serve streaming as well as batch workloads alike while being able
> to
> > > run jobs under constrained resources. The latter is particularly
> > important
> > > for bounded streaming jobs which, currently, are not well supported.
> > >
> > > ## Reactive-scaling mode
> > > Being able to react to newly available resources and rescaling a
> running
> > > job accordingly will make Flink's operation much easier because
> resources
> > > can then be controlled by an external tool (e.g. GCP autoscaling, K8s
> > > horizontal pod scaler, etc.). In this release we want to make a big
> step
> > > towards this direction. As a first step we want to support the
> execution
> > of
> > > jobs with a parallelism which is lower than the specified parallelism
> in
> > > case that Flink lost a TaskManager or could not acquire enough
> resources.
> > >
> > > # Maintenance/Stability
> > >
> > > ## JM / TM finished task reconciliation (
> > > https://issues.apache.org/jira/browse/FLINK-17075)
> > > This prevents the system from going out of sync if a task state change
> > from
> > > the TM to the JM is lost.
> > >
> > > ## Make metrics services work with Kubernetes deployments (
> > > https://issues.apache.org/jira/browse/FLINK-11127)
> > > Invert the direction in which the MetricFetcher connects to the
> > > MetricQueryFetchers. That way it will no longer be necessary to expose
> on
> > > K8s for every TaskManager a port on which the MetricQueryFetcher runs.
> > This
> > > will then make the deployment of Flink clusters on K8s easier.
> > >
> > > ## Handle long-blocking operations during job submission (savepoint
> > > restore) (https://issues.apache.org/jira/browse/FLINK-16866)
> > > Submitting a Flink job can involve the interaction with external
> systems
> > > (blocking operations). Depending on the job the interactions can take
> so
> > > long that it exceeds the submission timeout which reports a failure on
> > the
> > > client side even though the actual submission succeeded. By decoupling
> > the
> > > creation of the ExecutionGraph from the job submission, we can make the
> > job
> > > submission non-blocking which will solve this problem.
> > >
> > > ## Make IDs more intuitive to ease debugging (FLIP-118) (
> > > https://issues.apache.org/jira/browse/FLINK-15679)
> > > By making the internal Flink IDs compositional or logging how they
> belong
> > > together, we can make the debugging of Flink's operations much easier.
> > >
> > > Cheers,
> > > Till
> > >
> > >
> > > On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <fe...@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > Thanks for bring-up this discussion, Robert!
> > > > Congratulations on becoming the release manager of 1.12， Dian and
> > Robert
> > > !
> > > >
> > > > ----------
> > > > Here are some of my thoughts of the features for native integration
> > with
> > > > Kubernetes in Flink 1.12:
> > > >
> > > > 1. Support user-specified pod templates
> > > > Description:
> > > > The current approach of introducing new configuration options for
> > > each
> > > > aspect of pod specification a user might wish is becoming unwieldy,
> we
> > > have
> > > > to maintain more and more Flink side Kubernetes configuration options
> > and
> > > > users have to learn the gap between the declarative model used by
> > > > Kubernetes and the configuration model used by Flink. It's a great
> > > > improvement to allow users to specify pod templates as central places
> > for
> > > > all customization needs for the jobmanager and taskmanager pods.
> > > > Benefits:
> > > > Users can leverage many of the advanced K8s features that the
> Flink
> > > > community does not support explicitly, such as volume mounting, DNS
> > > > configuration, pod affinity/anti-affinity setting, etc.
> > > >
> > > > 2. Support running PyFlink on Kubernetes
> > > > Description:
> > > > Support running PyFlink on Kubernetes, including session cluster
> > and
> > > > application cluster.
> > > > Benefits:
> > > > Running python application in a containerized environment.
> > > >
> > > > 3. Support built-in init-Container
> > > > Description:
> > > > We need a built-in init-Container to help solve dependency
> > management
> > > > in a containerized environment, especially in the application mode.
> > > > Benefits:
> > > > Separate the base Flink image from dynamic dependencies.
> > > >
> > > > 4. Support accessing secured services via K8s secrets
> > > > Description:
> > > > Kubernetes Secrets
> > > > <https://kubernetes.io/docs/concepts/configuration/secret/> can be
> > used
> > > to
> > > > provide credentials for a Flink application to access secured
> services.
> > > It
> > > > helps people who want to use a user-specified K8s Secret through an
> > > > environment variable.
> > > > Benefits:
> > > > Improve user experience.
> > > >
> > > > 5. Support configuring replica of JobManager Deployment in ZooKeeper
> HA
> > > > setups
> > > > Description:
> > > > Make the *replica* of Deployment configurable in the ZooKeeper HA
> > > > setups.
> > > > Benefits:
> > > > Achieve faster failover.
> > > >
> > > > 6. Support to configure limit for CPU requirement
> > > > Description:
> > > > To leverage the Kubernetes feature of container request/limit
> CPU.
> > > > Benefits:
> > > > Reduce cost.
> > > >
> > > > Regards,
> > > > Canbin Zheng
> > > >
> > > > Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
> > > >
> > > > > I'm excited to hear about this feature, very, very, very highly
> > > > encouraged
> > > > >
> > > > >
> > > > > Prasanna kumar <pr...@gmail.com> 于2020年7月23日周四
> > > 上午12:10写道：
> > > > >
> > > > > > Hi Flink Dev Team,
> > > > > >
> > > > > > Dynamic AutoScaling Based on the incoming data load would be a
> > great
> > > > > > feature.
> > > > > >
> > > > > > We should be able have some rule say If the load increased by
> 20% ,
> > > add
> > > > > > extra resource should be added.
> > > > > > Or time based say during these peak hours the pipeline should
> scale
> > > > > > automatically by 50%.
> > > > > >
> > > > > > This will help a lot in cost reduction.
> > > > > >
> > > > > > EMR cluster provides a similar feature for SPARK based
> application.
> > > > > >
> > > > > > Thanks,
> > > > > > Prasanna.
> > > > > >
> > > > > > On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <
> > rmetzger@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Now that the 1.11 release is out, it is time to plan for the
> next
> > > > major
> > > > > > > Flink release.
> > > > > > >
> > > > > > > Some items:
> > > > > > >
> > > > > > > 1.
> > > > > > >
> > > > > > > Dian Fu and me volunteer to be the release managers for
> Flink
> > > > 1.12.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 1.
> > > > > > >
> > > > > > > Timeline: We propose to stick to our approximate 4 month
> > release
> > > > > > cycle,
> > > > > > > thus the release should be done by late October. Given that
> > > > there’s
> > > > > a
> > > > > > > holiday week in China at the beginning of October, I propose
> > to
> > > do
> > > > > the
> > > > > > > feature freeze on master by late September.
> > > > > > >
> > > > > > > 2.
> > > > > > >
> > > > > > > Collecting features: It would be good to have a rough
> overview
> > > of
> > > > > the
> > > > > > > features that will likely be ready to be merged by late
> > > September,
> > > > > and
> > > > > > > that
> > > > > > > we want in the release.
> > > > > > > Based on the discussion, we will update the Roadmap on the
> > Flink
> > > > > > website
> > > > > > > again!
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 1.
> > > > > > >
> > > > > > > Test instabilities and blockers: I would like to avoid a
> > > situation
> > > > > > where
> > > > > > > we have many blocking issues or build instabilities at the
> > time
> > > of
> > > > > the
> > > > > > > feature freeze. To achieve that, we will try to check every
> > > build
> > > > > > > instability within a week, to decide if it is a blocker
> (make
> > > sure
> > > > > to
> > > > > > > use
> > > > > > > the “test-stability” label for those tickets!)
> > > > > > > Blocker issues will need to have somebody assigned
> > (responsible)
> > > > > > within
> > > > > > > a week, and we want to see progress on all blocker issues
> > > > > (downgrade,
> > > > > > > resolution, a good plan how to proceed if it is more
> > > complicated)
> > > > > > >
> > > > > > > 2.
> > > > > > >
> > > > > > > Quality and stability of new features: In order to have a
> > short
> > > > > > feature
> > > > > > > freeze phase, we encourage developers to only merge
> > well-tested
> > > > and
> > > > > > > documented features. In our experience, the feature freeze
> > works
> > > > > best
> > > > > > if
> > > > > > > new features are complete, and the community can focus fully
> > on
> > > > > > > addressing
> > > > > > > newly found bugs and voting the release.
> > > > > > > By having a smooth release process, the next merge-window
> for
> > > the
> > > > > next
> > > > > > > release will come sooner.
> > > > > > >
> > > > > > >
> > > > > > > Let me know what you think about our items, and share which
> > > features
> > > > > you
> > > > > > > want in Flink 1.12.
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Robert & Dian
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best Regards,
> > > > > Harold Miao
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Kurt Young <yk...@gmail.com>.
Regarding setting the feature freeze date to late September, I have some
concern that it might make
the development time of 1.12 too short.
One reason for this is we took too much time (about 1.5 month, from mid of
May to beginning of July)
for testing 1.11. It's not ideal but further squeeze the development time
of 1.12 won't make this better.
Besides, AFAIK July & August is also a popular vacation season for
European. Given the fact most
committers of Flink come from Europe, I think we should also take this
into consideration.
It's also true that the first week of October is the national holiday of
China, so I'm wondering whether the
end of October could be a candidate feature freeze date.
Best,
Kurt
On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <rm...@apache.org> wrote:
> Hi all,
>
> Thanks a lot for the responses so far. I've put them into this Wiki page:
> https://cwiki.apache.org/confluence/display/FLINK/1.12+Release to keep
> track of them. Ideally, post JIRA tickets for your feature, then the status
> will update automatically in the wiki :)
>
> Please keep posting features here, or add them to the Wiki yourself 🙏
>
> @Prasanna kumar <pr...@gmail.com>: Dynamic Auto Scaling is a
> feature request the community is well-aware of. Till has posted
> "Reactive-scaling mode" as a feature he's working on for the 1.12 release.
> This work will introduce the basic building blocks and partial support for
> the feature you are requesting.
> Proper support for dynamic scaling, while maintaining Flink's high
> performance (throughout, low latency) and correctness is a difficult task
> that needs a lot of work. It will probably take a little bit of time till
> this is fully available.
>
> Cheers,
> Robert
>
>
>
> On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <tr...@apache.org>
> wrote:
>
> > Thanks for being our release managers for the 1.12 release Dian & Robert!
> >
> > Here are some features I would like to work on for this release:
> >
> > # Features
> >
> > ## Finishing pipelined region scheduling (
> > https://issues.apache.org/jira/browse/FLINK-16430)
> > With the pipelined region scheduler we want to implement a scheduler
> which
> > can serve streaming as well as batch workloads alike while being able to
> > run jobs under constrained resources. The latter is particularly
> important
> > for bounded streaming jobs which, currently, are not well supported.
> >
> > ## Reactive-scaling mode
> > Being able to react to newly available resources and rescaling a running
> > job accordingly will make Flink's operation much easier because resources
> > can then be controlled by an external tool (e.g. GCP autoscaling, K8s
> > horizontal pod scaler, etc.). In this release we want to make a big step
> > towards this direction. As a first step we want to support the execution
> of
> > jobs with a parallelism which is lower than the specified parallelism in
> > case that Flink lost a TaskManager or could not acquire enough resources.
> >
> > # Maintenance/Stability
> >
> > ## JM / TM finished task reconciliation (
> > https://issues.apache.org/jira/browse/FLINK-17075)
> > This prevents the system from going out of sync if a task state change
> from
> > the TM to the JM is lost.
> >
> > ## Make metrics services work with Kubernetes deployments (
> > https://issues.apache.org/jira/browse/FLINK-11127)
> > Invert the direction in which the MetricFetcher connects to the
> > MetricQueryFetchers. That way it will no longer be necessary to expose on
> > K8s for every TaskManager a port on which the MetricQueryFetcher runs.
> This
> > will then make the deployment of Flink clusters on K8s easier.
> >
> > ## Handle long-blocking operations during job submission (savepoint
> > restore) (https://issues.apache.org/jira/browse/FLINK-16866)
> > Submitting a Flink job can involve the interaction with external systems
> > (blocking operations). Depending on the job the interactions can take so
> > long that it exceeds the submission timeout which reports a failure on
> the
> > client side even though the actual submission succeeded. By decoupling
> the
> > creation of the ExecutionGraph from the job submission, we can make the
> job
> > submission non-blocking which will solve this problem.
> >
> > ## Make IDs more intuitive to ease debugging (FLIP-118) (
> > https://issues.apache.org/jira/browse/FLINK-15679)
> > By making the internal Flink IDs compositional or logging how they belong
> > together, we can make the debugging of Flink's operations much easier.
> >
> > Cheers,
> > Till
> >
> >
> > On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <fe...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > Thanks for bring-up this discussion, Robert!
> > > Congratulations on becoming the release manager of 1.12， Dian and
> Robert
> > !
> > >
> > > ----------
> > > Here are some of my thoughts of the features for native integration
> with
> > > Kubernetes in Flink 1.12:
> > >
> > > 1. Support user-specified pod templates
> > > Description:
> > > The current approach of introducing new configuration options for
> > each
> > > aspect of pod specification a user might wish is becoming unwieldy, we
> > have
> > > to maintain more and more Flink side Kubernetes configuration options
> and
> > > users have to learn the gap between the declarative model used by
> > > Kubernetes and the configuration model used by Flink. It's a great
> > > improvement to allow users to specify pod templates as central places
> for
> > > all customization needs for the jobmanager and taskmanager pods.
> > > Benefits:
> > > Users can leverage many of the advanced K8s features that the Flink
> > > community does not support explicitly, such as volume mounting, DNS
> > > configuration, pod affinity/anti-affinity setting, etc.
> > >
> > > 2. Support running PyFlink on Kubernetes
> > > Description:
> > > Support running PyFlink on Kubernetes, including session cluster
> and
> > > application cluster.
> > > Benefits:
> > > Running python application in a containerized environment.
> > >
> > > 3. Support built-in init-Container
> > > Description:
> > > We need a built-in init-Container to help solve dependency
> management
> > > in a containerized environment, especially in the application mode.
> > > Benefits:
> > > Separate the base Flink image from dynamic dependencies.
> > >
> > > 4. Support accessing secured services via K8s secrets
> > > Description:
> > > Kubernetes Secrets
> > > <https://kubernetes.io/docs/concepts/configuration/secret/> can be
> used
> > to
> > > provide credentials for a Flink application to access secured services.
> > It
> > > helps people who want to use a user-specified K8s Secret through an
> > > environment variable.
> > > Benefits:
> > > Improve user experience.
> > >
> > > 5. Support configuring replica of JobManager Deployment in ZooKeeper HA
> > > setups
> > > Description:
> > > Make the *replica* of Deployment configurable in the ZooKeeper HA
> > > setups.
> > > Benefits:
> > > Achieve faster failover.
> > >
> > > 6. Support to configure limit for CPU requirement
> > > Description:
> > > To leverage the Kubernetes feature of container request/limit CPU.
> > > Benefits:
> > > Reduce cost.
> > >
> > > Regards,
> > > Canbin Zheng
> > >
> > > Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
> > >
> > > > I'm excited to hear about this feature, very, very, very highly
> > > encouraged
> > > >
> > > >
> > > > Prasanna kumar <pr...@gmail.com> 于2020年7月23日周四
> > 上午12:10写道：
> > > >
> > > > > Hi Flink Dev Team,
> > > > >
> > > > > Dynamic AutoScaling Based on the incoming data load would be a
> great
> > > > > feature.
> > > > >
> > > > > We should be able have some rule say If the load increased by 20% ,
> > add
> > > > > extra resource should be added.
> > > > > Or time based say during these peak hours the pipeline should scale
> > > > > automatically by 50%.
> > > > >
> > > > > This will help a lot in cost reduction.
> > > > >
> > > > > EMR cluster provides a similar feature for SPARK based application.
> > > > >
> > > > > Thanks,
> > > > > Prasanna.
> > > > >
> > > > > On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <
> rmetzger@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Now that the 1.11 release is out, it is time to plan for the next
> > > major
> > > > > > Flink release.
> > > > > >
> > > > > > Some items:
> > > > > >
> > > > > > 1.
> > > > > >
> > > > > > Dian Fu and me volunteer to be the release managers for Flink
> > > 1.12.
> > > > > >
> > > > > >
> > > > > >
> > > > > > 1.
> > > > > >
> > > > > > Timeline: We propose to stick to our approximate 4 month
> release
> > > > > cycle,
> > > > > > thus the release should be done by late October. Given that
> > > there’s
> > > > a
> > > > > > holiday week in China at the beginning of October, I propose
> to
> > do
> > > > the
> > > > > > feature freeze on master by late September.
> > > > > >
> > > > > > 2.
> > > > > >
> > > > > > Collecting features: It would be good to have a rough overview
> > of
> > > > the
> > > > > > features that will likely be ready to be merged by late
> > September,
> > > > and
> > > > > > that
> > > > > > we want in the release.
> > > > > > Based on the discussion, we will update the Roadmap on the
> Flink
> > > > > website
> > > > > > again!
> > > > > >
> > > > > >
> > > > > >
> > > > > > 1.
> > > > > >
> > > > > > Test instabilities and blockers: I would like to avoid a
> > situation
> > > > > where
> > > > > > we have many blocking issues or build instabilities at the
> time
> > of
> > > > the
> > > > > > feature freeze. To achieve that, we will try to check every
> > build
> > > > > > instability within a week, to decide if it is a blocker (make
> > sure
> > > > to
> > > > > > use
> > > > > > the “test-stability” label for those tickets!)
> > > > > > Blocker issues will need to have somebody assigned
> (responsible)
> > > > > within
> > > > > > a week, and we want to see progress on all blocker issues
> > > > (downgrade,
> > > > > > resolution, a good plan how to proceed if it is more
> > complicated)
> > > > > >
> > > > > > 2.
> > > > > >
> > > > > > Quality and stability of new features: In order to have a
> short
> > > > > feature
> > > > > > freeze phase, we encourage developers to only merge
> well-tested
> > > and
> > > > > > documented features. In our experience, the feature freeze
> works
> > > > best
> > > > > if
> > > > > > new features are complete, and the community can focus fully
> on
> > > > > > addressing
> > > > > > newly found bugs and voting the release.
> > > > > > By having a smooth release process, the next merge-window for
> > the
> > > > next
> > > > > > release will come sooner.
> > > > > >
> > > > > >
> > > > > > Let me know what you think about our items, and share which
> > features
> > > > you
> > > > > > want in Flink 1.12.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Robert & Dian
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best Regards,
> > > > Harold Miao
> > > >
> > >
> >
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Robert Metzger <rm...@apache.org>.
Hi all,
Thanks a lot for the responses so far. I've put them into this Wiki page:
https://cwiki.apache.org/confluence/display/FLINK/1.12+Release to keep
track of them. Ideally, post JIRA tickets for your feature, then the status
will update automatically in the wiki :)
Please keep posting features here, or add them to the Wiki yourself 🙏
@Prasanna kumar <pr...@gmail.com>: Dynamic Auto Scaling is a
feature request the community is well-aware of. Till has posted
"Reactive-scaling mode" as a feature he's working on for the 1.12 release.
This work will introduce the basic building blocks and partial support for
the feature you are requesting.
Proper support for dynamic scaling, while maintaining Flink's high
performance (throughout, low latency) and correctness is a difficult task
that needs a lot of work. It will probably take a little bit of time till
this is fully available.
Cheers,
Robert
On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann <tr...@apache.org> wrote:
> Thanks for being our release managers for the 1.12 release Dian & Robert!
>
> Here are some features I would like to work on for this release:
>
> # Features
>
> ## Finishing pipelined region scheduling (
> https://issues.apache.org/jira/browse/FLINK-16430)
> With the pipelined region scheduler we want to implement a scheduler which
> can serve streaming as well as batch workloads alike while being able to
> run jobs under constrained resources. The latter is particularly important
> for bounded streaming jobs which, currently, are not well supported.
>
> ## Reactive-scaling mode
> Being able to react to newly available resources and rescaling a running
> job accordingly will make Flink's operation much easier because resources
> can then be controlled by an external tool (e.g. GCP autoscaling, K8s
> horizontal pod scaler, etc.). In this release we want to make a big step
> towards this direction. As a first step we want to support the execution of
> jobs with a parallelism which is lower than the specified parallelism in
> case that Flink lost a TaskManager or could not acquire enough resources.
>
> # Maintenance/Stability
>
> ## JM / TM finished task reconciliation (
> https://issues.apache.org/jira/browse/FLINK-17075)
> This prevents the system from going out of sync if a task state change from
> the TM to the JM is lost.
>
> ## Make metrics services work with Kubernetes deployments (
> https://issues.apache.org/jira/browse/FLINK-11127)
> Invert the direction in which the MetricFetcher connects to the
> MetricQueryFetchers. That way it will no longer be necessary to expose on
> K8s for every TaskManager a port on which the MetricQueryFetcher runs. This
> will then make the deployment of Flink clusters on K8s easier.
>
> ## Handle long-blocking operations during job submission (savepoint
> restore) (https://issues.apache.org/jira/browse/FLINK-16866)
> Submitting a Flink job can involve the interaction with external systems
> (blocking operations). Depending on the job the interactions can take so
> long that it exceeds the submission timeout which reports a failure on the
> client side even though the actual submission succeeded. By decoupling the
> creation of the ExecutionGraph from the job submission, we can make the job
> submission non-blocking which will solve this problem.
>
> ## Make IDs more intuitive to ease debugging (FLIP-118) (
> https://issues.apache.org/jira/browse/FLINK-15679)
> By making the internal Flink IDs compositional or logging how they belong
> together, we can make the debugging of Flink's operations much easier.
>
> Cheers,
> Till
>
>
> On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <fe...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > Thanks for bring-up this discussion, Robert!
> > Congratulations on becoming the release manager of 1.12， Dian and Robert
> !
> >
> > ----------
> > Here are some of my thoughts of the features for native integration with
> > Kubernetes in Flink 1.12:
> >
> > 1. Support user-specified pod templates
> > Description:
> > The current approach of introducing new configuration options for
> each
> > aspect of pod specification a user might wish is becoming unwieldy, we
> have
> > to maintain more and more Flink side Kubernetes configuration options and
> > users have to learn the gap between the declarative model used by
> > Kubernetes and the configuration model used by Flink. It's a great
> > improvement to allow users to specify pod templates as central places for
> > all customization needs for the jobmanager and taskmanager pods.
> > Benefits:
> > Users can leverage many of the advanced K8s features that the Flink
> > community does not support explicitly, such as volume mounting, DNS
> > configuration, pod affinity/anti-affinity setting, etc.
> >
> > 2. Support running PyFlink on Kubernetes
> > Description:
> > Support running PyFlink on Kubernetes, including session cluster and
> > application cluster.
> > Benefits:
> > Running python application in a containerized environment.
> >
> > 3. Support built-in init-Container
> > Description:
> > We need a built-in init-Container to help solve dependency management
> > in a containerized environment, especially in the application mode.
> > Benefits:
> > Separate the base Flink image from dynamic dependencies.
> >
> > 4. Support accessing secured services via K8s secrets
> > Description:
> > Kubernetes Secrets
> > <https://kubernetes.io/docs/concepts/configuration/secret/> can be used
> to
> > provide credentials for a Flink application to access secured services.
> It
> > helps people who want to use a user-specified K8s Secret through an
> > environment variable.
> > Benefits:
> > Improve user experience.
> >
> > 5. Support configuring replica of JobManager Deployment in ZooKeeper HA
> > setups
> > Description:
> > Make the *replica* of Deployment configurable in the ZooKeeper HA
> > setups.
> > Benefits:
> > Achieve faster failover.
> >
> > 6. Support to configure limit for CPU requirement
> > Description:
> > To leverage the Kubernetes feature of container request/limit CPU.
> > Benefits:
> > Reduce cost.
> >
> > Regards,
> > Canbin Zheng
> >
> > Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
> >
> > > I'm excited to hear about this feature, very, very, very highly
> > encouraged
> > >
> > >
> > > Prasanna kumar <pr...@gmail.com> 于2020年7月23日周四
> 上午12:10写道：
> > >
> > > > Hi Flink Dev Team,
> > > >
> > > > Dynamic AutoScaling Based on the incoming data load would be a great
> > > > feature.
> > > >
> > > > We should be able have some rule say If the load increased by 20% ,
> add
> > > > extra resource should be added.
> > > > Or time based say during these peak hours the pipeline should scale
> > > > automatically by 50%.
> > > >
> > > > This will help a lot in cost reduction.
> > > >
> > > > EMR cluster provides a similar feature for SPARK based application.
> > > >
> > > > Thanks,
> > > > Prasanna.
> > > >
> > > > On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <rm...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Now that the 1.11 release is out, it is time to plan for the next
> > major
> > > > > Flink release.
> > > > >
> > > > > Some items:
> > > > >
> > > > > 1.
> > > > >
> > > > > Dian Fu and me volunteer to be the release managers for Flink
> > 1.12.
> > > > >
> > > > >
> > > > >
> > > > > 1.
> > > > >
> > > > > Timeline: We propose to stick to our approximate 4 month release
> > > > cycle,
> > > > > thus the release should be done by late October. Given that
> > there’s
> > > a
> > > > > holiday week in China at the beginning of October, I propose to
> do
> > > the
> > > > > feature freeze on master by late September.
> > > > >
> > > > > 2.
> > > > >
> > > > > Collecting features: It would be good to have a rough overview
> of
> > > the
> > > > > features that will likely be ready to be merged by late
> September,
> > > and
> > > > > that
> > > > > we want in the release.
> > > > > Based on the discussion, we will update the Roadmap on the Flink
> > > > website
> > > > > again!
> > > > >
> > > > >
> > > > >
> > > > > 1.
> > > > >
> > > > > Test instabilities and blockers: I would like to avoid a
> situation
> > > > where
> > > > > we have many blocking issues or build instabilities at the time
> of
> > > the
> > > > > feature freeze. To achieve that, we will try to check every
> build
> > > > > instability within a week, to decide if it is a blocker (make
> sure
> > > to
> > > > > use
> > > > > the “test-stability” label for those tickets!)
> > > > > Blocker issues will need to have somebody assigned (responsible)
> > > > within
> > > > > a week, and we want to see progress on all blocker issues
> > > (downgrade,
> > > > > resolution, a good plan how to proceed if it is more
> complicated)
> > > > >
> > > > > 2.
> > > > >
> > > > > Quality and stability of new features: In order to have a short
> > > > feature
> > > > > freeze phase, we encourage developers to only merge well-tested
> > and
> > > > > documented features. In our experience, the feature freeze works
> > > best
> > > > if
> > > > > new features are complete, and the community can focus fully on
> > > > > addressing
> > > > > newly found bugs and voting the release.
> > > > > By having a smooth release process, the next merge-window for
> the
> > > next
> > > > > release will come sooner.
> > > > >
> > > > >
> > > > > Let me know what you think about our items, and share which
> features
> > > you
> > > > > want in Flink 1.12.
> > > > >
> > > > > Best,
> > > > >
> > > > > Robert & Dian
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Best Regards,
> > > Harold Miao
> > >
> >
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Till Rohrmann <tr...@apache.org>.
Thanks for being our release managers for the 1.12 release Dian & Robert!
Here are some features I would like to work on for this release:
# Features
## Finishing pipelined region scheduling (
https://issues.apache.org/jira/browse/FLINK-16430)
With the pipelined region scheduler we want to implement a scheduler which
can serve streaming as well as batch workloads alike while being able to
run jobs under constrained resources. The latter is particularly important
for bounded streaming jobs which, currently, are not well supported.
## Reactive-scaling mode
Being able to react to newly available resources and rescaling a running
job accordingly will make Flink's operation much easier because resources
can then be controlled by an external tool (e.g. GCP autoscaling, K8s
horizontal pod scaler, etc.). In this release we want to make a big step
towards this direction. As a first step we want to support the execution of
jobs with a parallelism which is lower than the specified parallelism in
case that Flink lost a TaskManager or could not acquire enough resources.
# Maintenance/Stability
## JM / TM finished task reconciliation (
https://issues.apache.org/jira/browse/FLINK-17075)
This prevents the system from going out of sync if a task state change from
the TM to the JM is lost.
## Make metrics services work with Kubernetes deployments (
https://issues.apache.org/jira/browse/FLINK-11127)
Invert the direction in which the MetricFetcher connects to the
MetricQueryFetchers. That way it will no longer be necessary to expose on
K8s for every TaskManager a port on which the MetricQueryFetcher runs. This
will then make the deployment of Flink clusters on K8s easier.
## Handle long-blocking operations during job submission (savepoint
restore) (https://issues.apache.org/jira/browse/FLINK-16866)
Submitting a Flink job can involve the interaction with external systems
(blocking operations). Depending on the job the interactions can take so
long that it exceeds the submission timeout which reports a failure on the
client side even though the actual submission succeeded. By decoupling the
creation of the ExecutionGraph from the job submission, we can make the job
submission non-blocking which will solve this problem.
## Make IDs more intuitive to ease debugging (FLIP-118) (
https://issues.apache.org/jira/browse/FLINK-15679)
By making the internal Flink IDs compositional or logging how they belong
together, we can make the debugging of Flink's operations much easier.
Cheers,
Till
On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng <fe...@gmail.com> wrote:
> Hi All,
>
> Thanks for bring-up this discussion, Robert!
> Congratulations on becoming the release manager of 1.12， Dian and Robert !
>
> ----------
> Here are some of my thoughts of the features for native integration with
> Kubernetes in Flink 1.12:
>
> 1. Support user-specified pod templates
> Description:
> The current approach of introducing new configuration options for each
> aspect of pod specification a user might wish is becoming unwieldy, we have
> to maintain more and more Flink side Kubernetes configuration options and
> users have to learn the gap between the declarative model used by
> Kubernetes and the configuration model used by Flink. It's a great
> improvement to allow users to specify pod templates as central places for
> all customization needs for the jobmanager and taskmanager pods.
> Benefits:
> Users can leverage many of the advanced K8s features that the Flink
> community does not support explicitly, such as volume mounting, DNS
> configuration, pod affinity/anti-affinity setting, etc.
>
> 2. Support running PyFlink on Kubernetes
> Description:
> Support running PyFlink on Kubernetes, including session cluster and
> application cluster.
> Benefits:
> Running python application in a containerized environment.
>
> 3. Support built-in init-Container
> Description:
> We need a built-in init-Container to help solve dependency management
> in a containerized environment, especially in the application mode.
> Benefits:
> Separate the base Flink image from dynamic dependencies.
>
> 4. Support accessing secured services via K8s secrets
> Description:
> Kubernetes Secrets
> <https://kubernetes.io/docs/concepts/configuration/secret/> can be used to
> provide credentials for a Flink application to access secured services. It
> helps people who want to use a user-specified K8s Secret through an
> environment variable.
> Benefits:
> Improve user experience.
>
> 5. Support configuring replica of JobManager Deployment in ZooKeeper HA
> setups
> Description:
> Make the *replica* of Deployment configurable in the ZooKeeper HA
> setups.
> Benefits:
> Achieve faster failover.
>
> 6. Support to configure limit for CPU requirement
> Description:
> To leverage the Kubernetes feature of container request/limit CPU.
> Benefits:
> Reduce cost.
>
> Regards,
> Canbin Zheng
>
> Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
>
> > I'm excited to hear about this feature, very, very, very highly
> encouraged
> >
> >
> > Prasanna kumar <pr...@gmail.com> 于2020年7月23日周四 上午12:10写道：
> >
> > > Hi Flink Dev Team,
> > >
> > > Dynamic AutoScaling Based on the incoming data load would be a great
> > > feature.
> > >
> > > We should be able have some rule say If the load increased by 20% , add
> > > extra resource should be added.
> > > Or time based say during these peak hours the pipeline should scale
> > > automatically by 50%.
> > >
> > > This will help a lot in cost reduction.
> > >
> > > EMR cluster provides a similar feature for SPARK based application.
> > >
> > > Thanks,
> > > Prasanna.
> > >
> > > On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <rm...@apache.org>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Now that the 1.11 release is out, it is time to plan for the next
> major
> > > > Flink release.
> > > >
> > > > Some items:
> > > >
> > > > 1.
> > > >
> > > > Dian Fu and me volunteer to be the release managers for Flink
> 1.12.
> > > >
> > > >
> > > >
> > > > 1.
> > > >
> > > > Timeline: We propose to stick to our approximate 4 month release
> > > cycle,
> > > > thus the release should be done by late October. Given that
> there’s
> > a
> > > > holiday week in China at the beginning of October, I propose to do
> > the
> > > > feature freeze on master by late September.
> > > >
> > > > 2.
> > > >
> > > > Collecting features: It would be good to have a rough overview of
> > the
> > > > features that will likely be ready to be merged by late September,
> > and
> > > > that
> > > > we want in the release.
> > > > Based on the discussion, we will update the Roadmap on the Flink
> > > website
> > > > again!
> > > >
> > > >
> > > >
> > > > 1.
> > > >
> > > > Test instabilities and blockers: I would like to avoid a situation
> > > where
> > > > we have many blocking issues or build instabilities at the time of
> > the
> > > > feature freeze. To achieve that, we will try to check every build
> > > > instability within a week, to decide if it is a blocker (make sure
> > to
> > > > use
> > > > the “test-stability” label for those tickets!)
> > > > Blocker issues will need to have somebody assigned (responsible)
> > > within
> > > > a week, and we want to see progress on all blocker issues
> > (downgrade,
> > > > resolution, a good plan how to proceed if it is more complicated)
> > > >
> > > > 2.
> > > >
> > > > Quality and stability of new features: In order to have a short
> > > feature
> > > > freeze phase, we encourage developers to only merge well-tested
> and
> > > > documented features. In our experience, the feature freeze works
> > best
> > > if
> > > > new features are complete, and the community can focus fully on
> > > > addressing
> > > > newly found bugs and voting the release.
> > > > By having a smooth release process, the next merge-window for the
> > next
> > > > release will come sooner.
> > > >
> > > >
> > > > Let me know what you think about our items, and share which features
> > you
> > > > want in Flink 1.12.
> > > >
> > > > Best,
> > > >
> > > > Robert & Dian
> > > >
> > >
> >
> >
> > --
> >
> > Best Regards,
> > Harold Miao
> >
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Canbin Zheng <fe...@gmail.com>.
Hi All,
Thanks for bring-up this discussion, Robert!
Congratulations on becoming the release manager of 1.12， Dian and Robert !
----------
Here are some of my thoughts of the features for native integration with
Kubernetes in Flink 1.12:
1. Support user-specified pod templates
Description:
The current approach of introducing new configuration options for each
aspect of pod specification a user might wish is becoming unwieldy, we have
to maintain more and more Flink side Kubernetes configuration options and
users have to learn the gap between the declarative model used by
Kubernetes and the configuration model used by Flink. It's a great
improvement to allow users to specify pod templates as central places for
all customization needs for the jobmanager and taskmanager pods.
Benefits:
Users can leverage many of the advanced K8s features that the Flink
community does not support explicitly, such as volume mounting, DNS
configuration, pod affinity/anti-affinity setting, etc.
2. Support running PyFlink on Kubernetes
Description:
Support running PyFlink on Kubernetes, including session cluster and
application cluster.
Benefits:
Running python application in a containerized environment.
3. Support built-in init-Container
Description:
We need a built-in init-Container to help solve dependency management
in a containerized environment, especially in the application mode.
Benefits:
Separate the base Flink image from dynamic dependencies.
4. Support accessing secured services via K8s secrets
Description:
Kubernetes Secrets
<https://kubernetes.io/docs/concepts/configuration/secret/> can be used to
provide credentials for a Flink application to access secured services. It
helps people who want to use a user-specified K8s Secret through an
environment variable.
Benefits:
Improve user experience.
5. Support configuring replica of JobManager Deployment in ZooKeeper HA
setups
Description:
Make the *replica* of Deployment configurable in the ZooKeeper HA
setups.
Benefits:
Achieve faster failover.
6. Support to configure limit for CPU requirement
Description:
To leverage the Kubernetes feature of container request/limit CPU.
Benefits:
Reduce cost.
Regards,
Canbin Zheng
Harold.Miao <mi...@gmail.com> 于2020年7月23日周四 下午12:44写道：
> I'm excited to hear about this feature, very, very, very highly encouraged
>
>
> Prasanna kumar <pr...@gmail.com> 于2020年7月23日周四 上午12:10写道：
>
> > Hi Flink Dev Team,
> >
> > Dynamic AutoScaling Based on the incoming data load would be a great
> > feature.
> >
> > We should be able have some rule say If the load increased by 20% , add
> > extra resource should be added.
> > Or time based say during these peak hours the pipeline should scale
> > automatically by 50%.
> >
> > This will help a lot in cost reduction.
> >
> > EMR cluster provides a similar feature for SPARK based application.
> >
> > Thanks,
> > Prasanna.
> >
> > On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <rm...@apache.org>
> > wrote:
> >
> > > Hi all,
> > >
> > > Now that the 1.11 release is out, it is time to plan for the next major
> > > Flink release.
> > >
> > > Some items:
> > >
> > > 1.
> > >
> > > Dian Fu and me volunteer to be the release managers for Flink 1.12.
> > >
> > >
> > >
> > > 1.
> > >
> > > Timeline: We propose to stick to our approximate 4 month release
> > cycle,
> > > thus the release should be done by late October. Given that there’s
> a
> > > holiday week in China at the beginning of October, I propose to do
> the
> > > feature freeze on master by late September.
> > >
> > > 2.
> > >
> > > Collecting features: It would be good to have a rough overview of
> the
> > > features that will likely be ready to be merged by late September,
> and
> > > that
> > > we want in the release.
> > > Based on the discussion, we will update the Roadmap on the Flink
> > website
> > > again!
> > >
> > >
> > >
> > > 1.
> > >
> > > Test instabilities and blockers: I would like to avoid a situation
> > where
> > > we have many blocking issues or build instabilities at the time of
> the
> > > feature freeze. To achieve that, we will try to check every build
> > > instability within a week, to decide if it is a blocker (make sure
> to
> > > use
> > > the “test-stability” label for those tickets!)
> > > Blocker issues will need to have somebody assigned (responsible)
> > within
> > > a week, and we want to see progress on all blocker issues
> (downgrade,
> > > resolution, a good plan how to proceed if it is more complicated)
> > >
> > > 2.
> > >
> > > Quality and stability of new features: In order to have a short
> > feature
> > > freeze phase, we encourage developers to only merge well-tested and
> > > documented features. In our experience, the feature freeze works
> best
> > if
> > > new features are complete, and the community can focus fully on
> > > addressing
> > > newly found bugs and voting the release.
> > > By having a smooth release process, the next merge-window for the
> next
> > > release will come sooner.
> > >
> > >
> > > Let me know what you think about our items, and share which features
> you
> > > want in Flink 1.12.
> > >
> > > Best,
> > >
> > > Robert & Dian
> > >
> >
>
>
> --
>
> Best Regards,
> Harold Miao
>
Re: [DISCUSS] Planning Flink 1.12
Posted by "Harold.Miao" <mi...@gmail.com>.
I'm excited to hear about this feature, very, very, very highly encouraged
Prasanna kumar <pr...@gmail.com> 于2020年7月23日周四 上午12:10写道：
> Hi Flink Dev Team,
>
> Dynamic AutoScaling Based on the incoming data load would be a great
> feature.
>
> We should be able have some rule say If the load increased by 20% , add
> extra resource should be added.
> Or time based say during these peak hours the pipeline should scale
> automatically by 50%.
>
> This will help a lot in cost reduction.
>
> EMR cluster provides a similar feature for SPARK based application.
>
> Thanks,
> Prasanna.
>
> On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <rm...@apache.org>
> wrote:
>
> > Hi all,
> >
> > Now that the 1.11 release is out, it is time to plan for the next major
> > Flink release.
> >
> > Some items:
> >
> > 1.
> >
> > Dian Fu and me volunteer to be the release managers for Flink 1.12.
> >
> >
> >
> > 1.
> >
> > Timeline: We propose to stick to our approximate 4 month release
> cycle,
> > thus the release should be done by late October. Given that there’s a
> > holiday week in China at the beginning of October, I propose to do the
> > feature freeze on master by late September.
> >
> > 2.
> >
> > Collecting features: It would be good to have a rough overview of the
> > features that will likely be ready to be merged by late September, and
> > that
> > we want in the release.
> > Based on the discussion, we will update the Roadmap on the Flink
> website
> > again!
> >
> >
> >
> > 1.
> >
> > Test instabilities and blockers: I would like to avoid a situation
> where
> > we have many blocking issues or build instabilities at the time of the
> > feature freeze. To achieve that, we will try to check every build
> > instability within a week, to decide if it is a blocker (make sure to
> > use
> > the “test-stability” label for those tickets!)
> > Blocker issues will need to have somebody assigned (responsible)
> within
> > a week, and we want to see progress on all blocker issues (downgrade,
> > resolution, a good plan how to proceed if it is more complicated)
> >
> > 2.
> >
> > Quality and stability of new features: In order to have a short
> feature
> > freeze phase, we encourage developers to only merge well-tested and
> > documented features. In our experience, the feature freeze works best
> if
> > new features are complete, and the community can focus fully on
> > addressing
> > newly found bugs and voting the release.
> > By having a smooth release process, the next merge-window for the next
> > release will come sooner.
> >
> >
> > Let me know what you think about our items, and share which features you
> > want in Flink 1.12.
> >
> > Best,
> >
> > Robert & Dian
> >
>
--
Best Regards,
Harold Miao
Re: [DISCUSS] Planning Flink 1.12
Posted by jincheng sun <su...@gmail.com>.
Hi All,
Thanks for bring-up this discussion, Robert!
Congratulations on becoming the release manager of 1.12， Dian and Robert !
----------
Here is my thoughts of the features for PyFlink in Flink 1.12:
1. Improve the usabilities for PyFlink
Description:
Improve the usabilities for PyFlink such as helping users check the
client and cluster environment, optimize error messages, improve the
current API type hint, etc.
Benefits:
Improve user experience.
2. PyFlink Table API DSL
Description:
Support Python Table API Expression DSL. Expression DSL has been
supported on the Java side(FLIP-55). This task tries to align Python Table
API with Java Table API.
Benefits:
Expression DSL is more user friendly than String expressions that users
can rely on IDE smart prompts to write expressions which can facilitate
users and increase development efficiency.
3. Python DataStream API
Description：
Support DataStream applications written in Python, including stateless
operations(keyBy, connect, union, map, flatMap, filter, etc) and stateful
operations(RichFunctions, ProcessFunctions, window, join).
Benefits:
1) By adding DataStream API in pyflink, it would provide users more
fine-grained configuration setting API for tasks(such as parallelism and
resource spec) and more complex data processing operation, which are users
strong demand while SQL and Table API are not supported at the moment.
2) For areas which have low relies on relation operations, such AI,
transformations like map, flatmap, are more prefered by users than Table
API.
4. Support Pandas UDAF in batch GroupBy aggregation
Description:
Support Pandas UDAF in batch GroupBy aggregation of Python Table API &
SQL. Both the input and output of the UDF is pandas.DataFrame.
Benefits:
1) Pandas UDAF performs better than row-at-a-time UDAF more than 10x in
certain scenarios
2) Users could use Pandas/Numpy API in the Python UDAF implementation if
the input/output data type is pandas.DataFrame
5.PyFlink Table API UDAF
Description:
Support UDAF for Python Table API.
Benefits:
Aggregations(stateful operations) can also be supported in PyFlink.
6. Support running pyflink jobs on kubernetes
Description:
Support running pyflink job on kubernetes, including dependency
management and so on just like on yarn and standalone cluster.
Benefits:
Kubernetes is a widely used container orchestration framework which has
more flexibility in application developement and deployment.
Welcome any comments and suggestions!
Best,
Jincheng
Dian Fu <di...@gmail.com> 于2020年7月23日周四 上午11:10写道：
> Thanks Robert for bringing up this discussion. This is very important to
> ensure that we have a smooth release process as there are only two months
> left before feature freeze.
>
> It would be good to have a list of the features for 1.12 as soon as
> possible. Welcome any one to post the feature list which you think
> important and want in 1.12.
>
> Regards,
> Dian
>
> > 在 2020年7月23日，上午12:10，Prasanna kumar <pr...@gmail.com> 写道：
> >
> > Hi Flink Dev Team,
> >
> > Dynamic AutoScaling Based on the incoming data load would be a great
> feature.
> >
> > We should be able have some rule say If the load increased by 20% , add
> extra resource should be added.
> > Or time based say during these peak hours the pipeline should scale
> automatically by 50%.
> >
> > This will help a lot in cost reduction.
> >
> > EMR cluster provides a similar feature for SPARK based application.
> >
> > Thanks,
> > Prasanna.
> >
> > On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <rmetzger@apache.org
> <ma...@apache.org>> wrote:
> > Hi all,
> >
> > Now that the 1.11 release is out, it is time to plan for the next major
> > Flink release.
> >
> > Some items:
> >
> > 1.
> >
> > Dian Fu and me volunteer to be the release managers for Flink 1.12.
> >
> >
> >
> > 1.
> >
> > Timeline: We propose to stick to our approximate 4 month release
> cycle,
> > thus the release should be done by late October. Given that there’s a
> > holiday week in China at the beginning of October, I propose to do the
> > feature freeze on master by late September.
> >
> > 2.
> >
> > Collecting features: It would be good to have a rough overview of the
> > features that will likely be ready to be merged by late September,
> and that
> > we want in the release.
> > Based on the discussion, we will update the Roadmap on the Flink
> website
> > again!
> >
> >
> >
> > 1.
> >
> > Test instabilities and blockers: I would like to avoid a situation
> where
> > we have many blocking issues or build instabilities at the time of the
> > feature freeze. To achieve that, we will try to check every build
> > instability within a week, to decide if it is a blocker (make sure to
> use
> > the “test-stability” label for those tickets!)
> > Blocker issues will need to have somebody assigned (responsible)
> within
> > a week, and we want to see progress on all blocker issues (downgrade,
> > resolution, a good plan how to proceed if it is more complicated)
> >
> > 2.
> >
> > Quality and stability of new features: In order to have a short
> feature
> > freeze phase, we encourage developers to only merge well-tested and
> > documented features. In our experience, the feature freeze works best
> if
> > new features are complete, and the community can focus fully on
> addressing
> > newly found bugs and voting the release.
> > By having a smooth release process, the next merge-window for the next
> > release will come sooner.
> >
> >
> > Let me know what you think about our items, and share which features you
> > want in Flink 1.12.
> >
> > Best,
> >
> > Robert & Dian
>
>
Re: [DISCUSS] Planning Flink 1.12
Posted by Dian Fu <di...@gmail.com>.
Thanks Robert for bringing up this discussion. This is very important to ensure that we have a smooth release process as there are only two months left before feature freeze.
It would be good to have a list of the features for 1.12 as soon as possible. Welcome any one to post the feature list which you think important and want in 1.12.
Regards,
Dian
> 在 2020年7月23日，上午12:10，Prasanna kumar <pr...@gmail.com> 写道：
>
> Hi Flink Dev Team,
>
> Dynamic AutoScaling Based on the incoming data load would be a great feature.
>
> We should be able have some rule say If the load increased by 20% , add extra resource should be added.
> Or time based say during these peak hours the pipeline should scale automatically by 50%.
>
> This will help a lot in cost reduction.
>
> EMR cluster provides a similar feature for SPARK based application.
>
> Thanks,
> Prasanna.
>
> On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <rmetzger@apache.org <ma...@apache.org>> wrote:
> Hi all,
>
> Now that the 1.11 release is out, it is time to plan for the next major
> Flink release.
>
> Some items:
>
> 1.
>
> Dian Fu and me volunteer to be the release managers for Flink 1.12.
>
>
>
> 1.
>
> Timeline: We propose to stick to our approximate 4 month release cycle,
> thus the release should be done by late October. Given that there’s a
> holiday week in China at the beginning of October, I propose to do the
> feature freeze on master by late September.
>
> 2.
>
> Collecting features: It would be good to have a rough overview of the
> features that will likely be ready to be merged by late September, and that
> we want in the release.
> Based on the discussion, we will update the Roadmap on the Flink website
> again!
>
>
>
> 1.
>
> Test instabilities and blockers: I would like to avoid a situation where
> we have many blocking issues or build instabilities at the time of the
> feature freeze. To achieve that, we will try to check every build
> instability within a week, to decide if it is a blocker (make sure to use
> the “test-stability” label for those tickets!)
> Blocker issues will need to have somebody assigned (responsible) within
> a week, and we want to see progress on all blocker issues (downgrade,
> resolution, a good plan how to proceed if it is more complicated)
>
> 2.
>
> Quality and stability of new features: In order to have a short feature
> freeze phase, we encourage developers to only merge well-tested and
> documented features. In our experience, the feature freeze works best if
> new features are complete, and the community can focus fully on addressing
> newly found bugs and voting the release.
> By having a smooth release process, the next merge-window for the next
> release will come sooner.
>
>
> Let me know what you think about our items, and share which features you
> want in Flink 1.12.
>
> Best,
>
> Robert & Dian
Re: [DISCUSS] Planning Flink 1.12
Posted by Prasanna kumar <pr...@gmail.com>.
Hi Flink Dev Team,
Dynamic AutoScaling Based on the incoming data load would be a great
feature.
We should be able have some rule say If the load increased by 20% , add
extra resource should be added.
Or time based say during these peak hours the pipeline should scale
automatically by 50%.
This will help a lot in cost reduction.
EMR cluster provides a similar feature for SPARK based application.
Thanks,
Prasanna.
On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger <rm...@apache.org> wrote:
> Hi all,
>
> Now that the 1.11 release is out, it is time to plan for the next major
> Flink release.
>
> Some items:
>
> 1.
>
> Dian Fu and me volunteer to be the release managers for Flink 1.12.
>
>
>
> 1.
>
> Timeline: We propose to stick to our approximate 4 month release cycle,
> thus the release should be done by late October. Given that there’s a
> holiday week in China at the beginning of October, I propose to do the
> feature freeze on master by late September.
>
> 2.
>
> Collecting features: It would be good to have a rough overview of the
> features that will likely be ready to be merged by late September, and
> that
> we want in the release.
> Based on the discussion, we will update the Roadmap on the Flink website
> again!
>
>
>
> 1.
>
> Test instabilities and blockers: I would like to avoid a situation where
> we have many blocking issues or build instabilities at the time of the
> feature freeze. To achieve that, we will try to check every build
> instability within a week, to decide if it is a blocker (make sure to
> use
> the “test-stability” label for those tickets!)
> Blocker issues will need to have somebody assigned (responsible) within
> a week, and we want to see progress on all blocker issues (downgrade,
> resolution, a good plan how to proceed if it is more complicated)
>
> 2.
>
> Quality and stability of new features: In order to have a short feature
> freeze phase, we encourage developers to only merge well-tested and
> documented features. In our experience, the feature freeze works best if
> new features are complete, and the community can focus fully on
> addressing
> newly found bugs and voting the release.
> By having a smooth release process, the next merge-window for the next
> release will come sooner.
>
>
> Let me know what you think about our items, and share which features you
> want in Flink 1.12.
>
> Best,
>
> Robert & Dian
>