You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Tzu-Li (Gordon) Tai" <tz...@apache.org> on 2019/05/01 05:15:23 UTC
[DISCUSS] Features for Apache Flink 1.9.0
Hi community,
Apache Flink 1.8.0 has been released a few weeks ago, so naturally, it’s
time to start thinking about what we want to aim for 1.9.0.
Kurt and I had collected some features that would be reasonable to consider
including for the next release, based on talking with various people as
well as observations from mailing list discussions and questions.
Note that having specific features listed here does not mean that no other
pull requests or topics will be reviewed. I am sure that there are other
ongoing efforts that we missed here and will likely make it as an
improvement or new feature in the next release. This discussion is merely
for bootstrapping a discussion for 1.9, as well as to give contributors an
idea of what the community is looking to focus on in the next couple of
weeks.
*Proposed features and focus*
In the previous major release, Apache Flink 1.8.0, the community had
prepared for some major Table & SQL additions from the Blink branch. With
this in mind, for the next release, it would be great to wind up those
efforts by merging in the Blink-based Table / SQL planner and runtime for
1.9.
Following Stephan’s previous thread [1] in the mailing list about features
in Blink, we should also start focusing on preparing for Blink’s other
several enhancements for batch execution. This includes resource
optimization, fine-grained failover, pluggable shuffle service, adapting
stream operators for batch execution, as well as better integration with
commonly used systems by batch executions such as Apache Hive.
Moreover, besides efforts related to the Blink merge, we would also like us
to work towards pushing forward some of the most discussed and anticipated
features by the community. Most of these had discussions in the mailing
lists that span multiple releases, and are also frequently brought up in
community events such as Flink Forward. This includes features such as
source event-time alignment and the source interface rework, a savepoint
connector that allows users to manipulate and query state in savepoints,
interactive programming, as well as terminating a job with a final
savepoint.
Last but not least, we have several existing contributions or discussions
for the ecosystem surrounding Flink, which we think is also very valuable
to try to merge in for 1.9. This includes a web UI rework (recently already
merged), active K8s integration, Google PubSub connector, native support
for the Protobuf format, Python support in the Table API, as well as
reworking Flink’s support for machine learning.
To wrap this up as a list of items, some of which already have JIRAs or
mailing list threads to track them:
- Merge Blink runner for Table & SQL [2]
-
Restructure flink-table to separate API from core runtime
-
Make table planners pluggable
-
Rework Table / SQL type system to integrate better with the SQL
standard [3]
-
Merge Blink planner and runtime for Table / SQL
- Further preparations for more batch execution optimization from
Blink
-
Dedicated scheduler component [4]
-
Fine grained failover for batch [5]
-
Selectable input stream operator [6]
-
Pluggable Shuffle Service [7]
-
FLIP-30: Unified Catalog API & Hive metastore integration [8]
- Heavily anticipated / discussed features in the community
-
FLIP-27: Source interface rework [9]
-
Savepoint connector [10]
-
FLIP-34: Terminate / Suspend job with savepoint [11]
-
FLIP-36: Interactive Programming [12]
- Ecosystem
-
Web UI rework [13]
-
Active K8s integration [14]
-
Google PubSub connector [15]
-
First-class Protobuf support [16]
-
FLIP-38: Python support in Table API [17]
-
FLIP-39: Flink ML pipeline and libraries on top of Table API [18]
*Suggested release timeline*
Apache Flink 1.8.0 was released earlier this month, so based on our usual
timely release schedule, we should aim for releasing 1.9.0 around mid to
end July.
Since it seems that this is going to be a fairly large release, to give the
community enough testing time, I propose that the feature freeze to be near
the end of June (8-9 weeks from now, probable June 28). This is of course a
ballpark estimation for now; we should follow-up with a separate thread
later in the release cycle to prepare contributors with an official feature
freeze date.
I’d also like to use this opportunity to propose myself and Kurt as the
release managers for 1.9.
AFAIK, we did not used to have 2 RMs for a single release in the past, but
1.9.0 is definitely quite ambitious so it would not hurt to have one more
on board :) Cheers, Gordon [1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
[2] https://issues.apache.org/jira/browse/FLINK-11439
[3] https://issues.apache.org/jira/browse/FLINK-12251
[4] https://issues.apache.org/jira/browse/FLINK-10429
[5]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
[6]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
[7] https://issues.apache.org/jira/browse/FLINK-10653
[8] https://issues.apache.org/jira/browse/FLINK-11275
[9]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
[10] https://issues.apache.org/jira/browse/FLINK-12047
[11]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
[12]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
[13] https://issues.apache.org/jira/browse/FLINK-10705
[14] https://issues.apache.org/jira/browse/FLINK-9953
[15] https://issues.apache.org/jira/browse/FLINK-9311
[16] https://issues.apache.org/jira/browse/FLINK-11333
[17]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
[18]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by "Zhang, Xuefu" <xu...@alibaba-inc.com>.
To add, Hive integration depends on a few features that are actively developed. If the completion of those features don't leave enough time for us to integrate, then our work can potentially go beyond the proposed date.
Just wanted to point out such a dependency adds uncertainty.
Thanks,
Xuefu
------------------------------------------------------------------
From:Tzu-Li (Gordon) Tai <tz...@apache.org>
Sent At:2019 Jun. 20 (Thu.) 01:01
To:dev <de...@flink.apache.org>
Cc:Xuefu <xu...@alibaba-inc.com>; Timo Walther <tw...@apache.org>; Dawid Wysakowicz <dw...@apache.org>
Subject:Re: [DISCUSS] Features for Apache Flink 1.9.0
Hi all,
Thanks for all the updates and work!
From the looks so far, overall it seems like we are still in a good spot to officially announce the feature freeze date to be on the originally proposed date, June 28.
I’ll announce this in a separate thread.
Cheers,
Gordon
On Fri, Jun 7, 2019 at 2:31 AM Bowen Li <bo...@gmail.com> wrote:
For features I'm involved in:
- FLIP-30 unified catalogs APIs [1]: close to be done. On track
- hive integration
- HiveCatalog for persisting Flink/Hive metadata [2]: close to be
done. On track
- hive data connector [3]: input/output format are close to be done.
Was blocked on source/sink interfaces. We had several discussions yesterday
and concluded that we may have a quick working solution out for 1.9. Thus
I'd say on track
- hive functions [4]: Just started. It has major dependencies on
function definitions and type system rework part II. In the last few weeks,
community is mainly focusing on Blink planner and related tasks on
SQL/Table API side as Timo mentioned above, and the work of function
definition just got started this week. I'm working closely with Timo to
push this efforts forward. It's a bit risky but I'm glad we start to make
progress now
- SQL DDL: also had discussions yesterday. Working together with Kurt, we
hope to have at least some basic DDL to offer users an end-to-end working
solution for both Flink and Hive use cases in 1.9
[1]:
- https://issues.apache.org/jira/browse/FLINK-11275
- https://issues.apache.org/jira/browse/FLINK-12625
-
https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
[2]: https://issues.apache.org/jira/browse/FLINK-12755
[3]: https://issues.apache.org/jira/browse/FLINK-10729
[4]: https://issues.apache.org/jira/browse/FLINK-12656
On Tue, Jun 4, 2019 at 12:12 AM Till Rohrmann <tr...@apache.org> wrote:
> Thanks for starting this discussion Gordon and Kurt. For the development
> threads I'm involved with here are the updates:
>
> * Pluggable scheduler: Good part of the work is completed. Gary now works
> on the glue code to use the new high level scheduler components. The
> estimate to finish this work is end of June (estimate: 4 weeks starting
> from this week). The changes to the scheduler would benefit from very
> thorough testing because they are core to Flink.
>
> * External shuffle service: As Zhijiang said, we hope to finish the work by
> the end of this week or early next week (estimate: 1 week from now).
>
> * Result partition life cycle management / fine grained recovery: The
> current estimate to complete this feature would be end of next week or
> beginning of the week afterwards (estimate: 2 weeks from now). This feature
> should enable fine grained recovery for batch.
>
> * Java 9 support: Flink builds with Java 9. Not all e2e tests are running
> with Java 9 though.
>
> * Active K8s integration: PRs are open but reviews are still pending.
>
> Cheers,
> Till
>
> On Wed, May 29, 2019 at 4:45 AM Biao Liu <mm...@gmail.com> wrote:
>
> > Thanks for being the release manager, Gordon & Kurt.
> >
> > For FLIP-27, there are still some more details need to discuss. I don't
> > think it could catch up the release of 1.9. @Aljoscha, @Stephan, do you
> > agree that?
> >
> > zhijiang <wa...@aliyun.com.invalid> 于2019年5月28日周二 下午11:28写道：
> >
> > > Hi Gordon,
> > >
> > > Thanks for the kind reminder of feature freeze date for 1.9.0. I think
> > the
> > > date makes sense on my side.
> > >
> > > For FLIP-31, I and Andrey could be done within two weeks or so.
> > > And I already finished my side work for FLIP-1.
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > ------------------------------------------------------------------
> > > From:Timo Walther <tw...@apache.org>
> > > Send Time:2019年5月28日(星期二) 19:26
> > > To:dev <de...@flink.apache.org>
> > > Subject:Re: [DISCUSS] Features for Apache Flink 1.9.0
> > >
> > > Thanks for being the release managers, Kurt and Gordon!
> > >
> > > From the Table & SQL API side, there are still a lot of open issues
> > > that need to be solved to decouple the API from a planner and enable
> the
> > > Blink planner. Also we need to make sure that the Blink planner
> supports
> > > at least everything of Flink 1.8 to not introduce a regression. We
> might
> > > need to focus more on the main features which is a runnable Blink
> > > planner and might need to postpone other discussions such as DDL, new
> > > source/sink interfaces, or proper type inference logic. However, in
> many
> > > cases there are shortcuts that we could take in order to achieve our
> > > goals. So I'm confident that we solve the big blockers until the
> feature
> > > freeze :)
> > >
> > > I will keep you updated.
> > >
> > > Thanks,
> > > Timo
> > >
> > >
> > > Am 28.05.19 um 05:07 schrieb Kurt Young:
> > > > Thanks Gordon for bringing this up.
> > > >
> > > > I'm glad to say that blink planner merge work is almost done, and i
> > will
> > > > follow up the work of
> > > > integrating blink planner with Table API to co-exist with current
> flink
> > > > planner.
> > > >
> > > > In addition to this, the following features:
> > > > 1. FLIP-32: Restructure flink-table for future contributions [1]
> > > > 2. FLIP-37: Rework of the Table API Type System [2]
> > > > 3. Hive integration work (including hive meta [3] and connectors)
> > > >
> > > > are also going well, i will spend some time to keep track of them.
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
> > > > [2]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > [3]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > >
> > > > Best,
> > > > Kurt
> > > >
> > > >
> > > > On Mon, May 27, 2019 at 7:18 PM jincheng sun <
> sunjincheng121@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Hi Gordon,
> > > >>
> > > >> Thanks for mention the feature freeze date for 1.9.0, that's very
> > > helpful
> > > >> for contributors to evaluate their dev plan！
> > > >>
> > > >> Regarding FLIP-29, we are glad to do our best to finish the dev of
> > > FLIP-29,
> > > >> then catch up with the release of 1.9.
> > > >>
> > > >> Thanks again for push the release of 1.9.0 forward!
> > > >>
> > > >> Cheers,
> > > >> Jincheng
> > > >>
> > > >>
> > > >>
> > > >> Tzu-Li (Gordon) Tai <tz...@apache.org> 于2019年5月27日周一 下午5:48写道：
> > > >>
> > > >>> Hi all,
> > > >>>
> > > >>> I want to kindly remind the community that we're now 5 weeks away
> > from
> > > >> the
> > > >>> proposed feature freeze date for 1.9.0, which is June 28.
> > > >>>
> > > >>> This is not yet a final date we have agreed on, so I would like to
> > > start
> > > >>> collecting feedback on how the mentioned features are going, and in
> > > >>> general, whether or not the date sounds reasonable given the
> current
> > > >> status
> > > >>> of the ongoing efforts.
> > > >>> Please let me know what you think!
> > > >>>
> > > >>> Cheers,
> > > >>> Gordon
> > > >>>
> > > >>>
> > > >>> On Mon, May 27, 2019 at 5:40 PM Tzu-Li (Gordon) Tai <
> > > tzulitai@apache.org
> > > >>>
> > > >>> wrote:
> > > >>>
> > > >>>> @Hequn @Jincheng
> > > >>>>
> > > >>>> Thanks for bringing up FLIP-29 to attention.
> > > >>>> As previously mentioned, the original list is not a fixed feature
> > set,
> > > >> so
> > > >>>> if FLIP-29 has ongoing efforts and can make it before the feature
> > > >> freeze,
> > > >>>> then of course it should be included!
> > > >>>>
> > > >>>> @himansh1306
> > > >>>>
> > > >>>> Concerning the ORC format for StreamingFileSink, is there already
> a
> > > >> JIRA
> > > >>>> ticket tracking that? If not, I suggest to first open one and see
> if
> > > >>> there
> > > >>>> are similar interests from committers in adding that.
> > > >>>>
> > > >>>>
> > > >>>> On Sun, May 5, 2019 at 11:19 PM Hequn Cheng <chenghequn@gmail.com
> >
> > > >>> wrote:
> > > >>>>> Hi,
> > > >>>>>
> > > >>>>> Great job, Gordon! Thanks a lot for driving this and wrapping
> > > features
> > > >>> up
> > > >>>>> to a detailed list. +1 on it!
> > > >>>>>
> > > >>>>> Would be great if we can also add flip29 to the list. @jincheng
> sun
> > > >>>>> <su...@gmail.com> and I are focusing on it these
> days. I
> > > >>> think
> > > >>>>> these features in flip29 would bring big enhancements to the
> Table
> > > >> API.
> > > >>>>> :-)
> > > >>>>>
> > > >>>>> Best, Hequn
> > > >>>>>
> > > >>>>> On Sun, May 5, 2019 at 10:41 PM Becket Qin <becket.qin@gmail.com
> >
> > > >>> wrote:
> > > >>>>>> Thanks for driving this release, Gordon. +1 on the feature list.
> > > >>>>>>
> > > >>>>>> This is a pretty exciting and ambitious release!
> > > >>>>>>
> > > >>>>>> Cheers,
> > > >>>>>>
> > > >>>>>> Jiangjie (Becket) Qin
> > > >>>>>>
> > > >>>>>> On Sun, May 5, 2019 at 4:28 PM jincheng sun <
> > > >> sunjincheng121@gmail.com
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Thanks a lot for being our release manager, Great job!
> > > >>>>>>>
> > > >>>>>>> +1 for the feature list and It's better to add FLIP-29
> > > >>>>>>> <
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739
> > > >>>>>>>> (Support
> > > >>>>>>> map/flatMap/aggregate/flatAggregate on TableAPI), as the goals
> of
> > > >>>>> release
> > > >>>>>>> 1.9.
> > > >>>>>>>
> > > >>>>>>> What do you think?
> > > >>>>>>>
> > > >>>>>>> Best,
> > > >>>>>>> Jincheng
> > > >>>>>>>
> > > >>>>>>> Bowen Li <bo...@gmail.com> 于2019年5月5日周日 上午12:47写道：
> > > >>>>>>>
> > > >>>>>>>> +1, exciting and ambitious goals, the rough timeline looks
> > > >>>>> reasonable.
> > > >>>>>>>> Let's make it happen!
> > > >>>>>>>>
> > > >>>>>>>> On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com>
> > > >> wrote:
> > > >>>>>>>>> +1 for the 1.9.0 feature list. Excited to see it is
> happening.
> > > >>>>>>>>>
> > > >>>>>>>>> Regards,
> > > >>>>>>>>> Jark
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <
> > > >>>>>>>> himansh1306@gmail.com>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> +1 for Protobuf, Hive Metastore integration & features
> > > >> related
> > > >>>>> to
> > > >>>>>>>>>> savepoint.
> > > >>>>>>>>>>
> > > >>>>>>>>>> I was hoping if we could support for ORC File format can be
> > > >>>>> added
> > > >>>>>> in
> > > >>>>>>>>>> StreamingFileSink Writer, Currently only Parquet is
> > > >> supported
> > > >>>>> with
> > > >>>>>>>>> respect
> > > >>>>>>>>>> to columnar file formats
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <
> > > >>>>> tzulitai@apache.org
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>> Hi community,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Apache Flink 1.8.0 has been released a few weeks ago, so
> > > >>>>>> naturally,
> > > >>>>>>>>> it’s
> > > >>>>>>>>>>> time to start thinking about what we want to aim for
> > > >> 1.9.0.
> > > >>>>>>>>>>> Kurt and I had collected some features that would be
> > > >>>>> reasonable
> > > >>>>>> to
> > > >>>>>>>>>> consider
> > > >>>>>>>>>>> including for the next release, based on talking with
> > > >>> various
> > > >>>>>>> people
> > > >>>>>>>> as
> > > >>>>>>>>>>> well as observations from mailing list discussions and
> > > >>>>> questions.
> > > >>>>>>>>>>> Note that having specific features listed here does not
> > > >> mean
> > > >>>>> that
> > > >>>>>>> no
> > > >>>>>>>>>> other
> > > >>>>>>>>>>> pull requests or topics will be reviewed. I am sure that
> > > >>> there
> > > >>>>>> are
> > > >>>>>>>>> other
> > > >>>>>>>>>>> ongoing efforts that we missed here and will likely make
> > > >> it
> > > >>>>> as an
> > > >>>>>>>>>>> improvement or new feature in the next release. This
> > > >>>>> discussion
> > > >>>>>> is
> > > >>>>>>>>> merely
> > > >>>>>>>>>>> for bootstrapping a discussion for 1.9, as well as to give
> > > >>>>>>>> contributors
> > > >>>>>>>>>> an
> > > >>>>>>>>>>> idea of what the community is looking to focus on in the
> > > >>> next
> > > >>>>>>> couple
> > > >>>>>>>> of
> > > >>>>>>>>>>> weeks.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> *Proposed features and focus*
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> In the previous major release, Apache Flink 1.8.0, the
> > > >>>>> community
> > > >>>>>>> had
> > > >>>>>>>>>>> prepared for some major Table & SQL additions from the
> > > >> Blink
> > > >>>>>>> branch.
> > > >>>>>>>>> With
> > > >>>>>>>>>>> this in mind, for the next release, it would be great to
> > > >>> wind
> > > >>>>> up
> > > >>>>>>>> those
> > > >>>>>>>>>>> efforts by merging in the Blink-based Table / SQL planner
> > > >>> and
> > > >>>>>>> runtime
> > > >>>>>>>>> for
> > > >>>>>>>>>>> 1.9.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Following Stephan’s previous thread [1] in the mailing
> > > >> list
> > > >>>>> about
> > > >>>>>>>>>> features
> > > >>>>>>>>>>> in Blink, we should also start focusing on preparing for
> > > >>>>> Blink’s
> > > >>>>>>>> other
> > > >>>>>>>>>>> several enhancements for batch execution. This includes
> > > >>>>> resource
> > > >>>>>>>>>>> optimization, fine-grained failover, pluggable shuffle
> > > >>>>> service,
> > > >>>>>>>>> adapting
> > > >>>>>>>>>>> stream operators for batch execution, as well as better
> > > >>>>>> integration
> > > >>>>>>>>> with
> > > >>>>>>>>>>> commonly used systems by batch executions such as Apache
> > > >>> Hive.
> > > >>>>>>>>>>> Moreover, besides efforts related to the Blink merge, we
> > > >>> would
> > > >>>>>> also
> > > >>>>>>>>> like
> > > >>>>>>>>>> us
> > > >>>>>>>>>>> to work towards pushing forward some of the most discussed
> > > >>> and
> > > >>>>>>>>>> anticipated
> > > >>>>>>>>>>> features by the community. Most of these had discussions
> > > >> in
> > > >>>>> the
> > > >>>>>>>> mailing
> > > >>>>>>>>>>> lists that span multiple releases, and are also frequently
> > > >>>>>> brought
> > > >>>>>>> up
> > > >>>>>>>>> in
> > > >>>>>>>>>>> community events such as Flink Forward. This includes
> > > >>> features
> > > >>>>>> such
> > > >>>>>>>> as
> > > >>>>>>>>>>> source event-time alignment and the source interface
> > > >>> rework, a
> > > >>>>>>>>> savepoint
> > > >>>>>>>>>>> connector that allows users to manipulate and query state
> > > >> in
> > > >>>>>>>>> savepoints,
> > > >>>>>>>>>>> interactive programming, as well as terminating a job
> > > >> with a
> > > >>>>>> final
> > > >>>>>>>>>>> savepoint.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Last but not least, we have several existing contributions
> > > >>> or
> > > >>>>>>>>> discussions
> > > >>>>>>>>>>> for the ecosystem surrounding Flink, which we think is
> > > >> also
> > > >>>>> very
> > > >>>>>>>>> valuable
> > > >>>>>>>>>>> to try to merge in for 1.9. This includes a web UI rework
> > > >>>>>> (recently
> > > >>>>>>>>>> already
> > > >>>>>>>>>>> merged), active K8s integration, Google PubSub connector,
> > > >>>>> native
> > > >>>>>>>>> support
> > > >>>>>>>>>>> for the Protobuf format, Python support in the Table API,
> > > >> as
> > > >>>>> well
> > > >>>>>>> as
> > > >>>>>>>>>>> reworking Flink’s support for machine learning.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> To wrap this up as a list of items, some of which already
> > > >>> have
> > > >>>>>>> JIRAs
> > > >>>>>>>> or
> > > >>>>>>>>>>> mailing list threads to track them:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Merge Blink runner for Table & SQL [2]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Restructure flink-table to separate API from core
> > > >>>>> runtime
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Make table planners pluggable
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Rework Table / SQL type system to integrate better
> > > >>> with
> > > >>>>> the
> > > >>>>>>> SQL
> > > >>>>>>>>>>> standard [3]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Merge Blink planner and runtime for Table / SQL
> > > >>>>>>>>>>> - Further preparations for more batch execution
> > > >>>>>> optimization
> > > >>>>>>>> from
> > > >>>>>>>>>>> Blink
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Dedicated scheduler component [4]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Fine grained failover for batch [5]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Selectable input stream operator [6]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Pluggable Shuffle Service [7]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> FLIP-30: Unified Catalog API & Hive metastore
> > > >>>>> integration
> > > >>>>>> [8]
> > > >>>>>>>>>>> - Heavily anticipated / discussed features in the
> > > >>>>> community
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> FLIP-27: Source interface rework [9]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Savepoint connector [10]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> FLIP-34: Terminate / Suspend job with savepoint [11]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> FLIP-36: Interactive Programming [12]
> > > >>>>>>>>>>> - Ecosystem
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Web UI rework [13]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Active K8s integration [14]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Google PubSub connector [15]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> First-class Protobuf support [16]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> FLIP-38: Python support in Table API [17]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> FLIP-39: Flink ML pipeline and libraries on top of
> > > >>> Table
> > > >>>>>> API
> > > >>>>>>>> [18]
> > > >>>>>>>>>>> *Suggested release timeline*
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Apache Flink 1.8.0 was released earlier this month, so
> > > >> based
> > > >>>>> on
> > > >>>>>> our
> > > >>>>>>>>> usual
> > > >>>>>>>>>>> timely release schedule, we should aim for releasing 1.9.0
> > > >>>>> around
> > > >>>>>>> mid
> > > >>>>>>>>> to
> > > >>>>>>>>>>> end July.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Since it seems that this is going to be a fairly large
> > > >>>>> release,
> > > >>>>>> to
> > > >>>>>>>> give
> > > >>>>>>>>>> the
> > > >>>>>>>>>>> community enough testing time, I propose that the feature
> > > >>>>> freeze
> > > >>>>>> to
> > > >>>>>>>> be
> > > >>>>>>>>>> near
> > > >>>>>>>>>>> the end of June (8-9 weeks from now, probable June 28).
> > > >> This
> > > >>>>> is
> > > >>>>>> of
> > > >>>>>>>>>> course a
> > > >>>>>>>>>>> ballpark estimation for now; we should follow-up with a
> > > >>>>> separate
> > > >>>>>>>> thread
> > > >>>>>>>>>>> later in the release cycle to prepare contributors with an
> > > >>>>>> official
> > > >>>>>>>>>> feature
> > > >>>>>>>>>>> freeze date.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I’d also like to use this opportunity to propose myself
> > > >> and
> > > >>>>> Kurt
> > > >>>>>> as
> > > >>>>>>>> the
> > > >>>>>>>>>>> release managers for 1.9.
> > > >>>>>>>>>>> AFAIK, we did not used to have 2 RMs for a single release
> > > >> in
> > > >>>>> the
> > > >>>>>>>> past,
> > > >>>>>>>>>> but
> > > >>>>>>>>>>> 1.9.0 is definitely quite ambitious so it would not hurt
> > > >> to
> > > >>>>> have
> > > >>>>>>> one
> > > >>>>>>>>> more
> > > >>>>>>>>>>> on board :) Cheers, Gordon [1]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
> > > >>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-11439
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/FLINK-12251
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/FLINK-10429
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [5]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
> > > >>>>>>>>>>> [6]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
> > > >>>>>>>>>>> [7] https://issues.apache.org/jira/browse/FLINK-10653
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [8] https://issues.apache.org/jira/browse/FLINK-11275
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [9]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
> > > >>>>>>>>>>> [10] https://issues.apache.org/jira/browse/FLINK-12047
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [11]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
> > > >>>>>>>>>>> [12]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> > > >>>>>>>>>>> [13] https://issues.apache.org/jira/browse/FLINK-10705
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [14] https://issues.apache.org/jira/browse/FLINK-9953
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [15] https://issues.apache.org/jira/browse/FLINK-9311
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [16] https://issues.apache.org/jira/browse/FLINK-11333
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [17]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> > > >>>>>>>>>>> [18]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
> > >
> > >
> >
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi all,
Thanks for all the updates and work!
From the looks so far, overall it seems like we are still in a good spot to
officially announce the feature freeze date to be on the originally
proposed date, June 28.
I’ll announce this in a separate thread.
Cheers,
Gordon
On Fri, Jun 7, 2019 at 2:31 AM Bowen Li <bo...@gmail.com> wrote:
> For features I'm involved in:
>
> - FLIP-30 unified catalogs APIs [1]: close to be done. On track
>
> - hive integration
> - HiveCatalog for persisting Flink/Hive metadata [2]: close to be
> done. On track
>
> - hive data connector [3]: input/output format are close to be done.
> Was blocked on source/sink interfaces. We had several discussions yesterday
> and concluded that we may have a quick working solution out for 1.9. Thus
> I'd say on track
>
> - hive functions [4]: Just started. It has major dependencies on
> function definitions and type system rework part II. In the last few weeks,
> community is mainly focusing on Blink planner and related tasks on
> SQL/Table API side as Timo mentioned above, and the work of function
> definition just got started this week. I'm working closely with Timo to
> push this efforts forward. It's a bit risky but I'm glad we start to make
> progress now
>
> - SQL DDL: also had discussions yesterday. Working together with Kurt, we
> hope to have at least some basic DDL to offer users an end-to-end working
> solution for both Flink and Hive use cases in 1.9
>
> [1]:
> - https://issues.apache.org/jira/browse/FLINK-11275
> - https://issues.apache.org/jira/browse/FLINK-12625
> -
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> [2]: https://issues.apache.org/jira/browse/FLINK-12755
> [3]: https://issues.apache.org/jira/browse/FLINK-10729
> [4]: https://issues.apache.org/jira/browse/FLINK-12656
>
> On Tue, Jun 4, 2019 at 12:12 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
> > Thanks for starting this discussion Gordon and Kurt. For the development
> > threads I'm involved with here are the updates:
> >
> > * Pluggable scheduler: Good part of the work is completed. Gary now works
> > on the glue code to use the new high level scheduler components. The
> > estimate to finish this work is end of June (estimate: 4 weeks starting
> > from this week). The changes to the scheduler would benefit from very
> > thorough testing because they are core to Flink.
> >
> > * External shuffle service: As Zhijiang said, we hope to finish the work
> by
> > the end of this week or early next week (estimate: 1 week from now).
> >
> > * Result partition life cycle management / fine grained recovery: The
> > current estimate to complete this feature would be end of next week or
> > beginning of the week afterwards (estimate: 2 weeks from now). This
> feature
> > should enable fine grained recovery for batch.
> >
> > * Java 9 support: Flink builds with Java 9. Not all e2e tests are running
> > with Java 9 though.
> >
> > * Active K8s integration: PRs are open but reviews are still pending.
> >
> > Cheers,
> > Till
> >
> > On Wed, May 29, 2019 at 4:45 AM Biao Liu <mm...@gmail.com> wrote:
> >
> > > Thanks for being the release manager, Gordon & Kurt.
> > >
> > > For FLIP-27, there are still some more details need to discuss. I don't
> > > think it could catch up the release of 1.9. @Aljoscha, @Stephan, do you
> > > agree that?
> > >
> > > zhijiang <wa...@aliyun.com.invalid> 于2019年5月28日周二 下午11:28写道：
> > >
> > > > Hi Gordon,
> > > >
> > > > Thanks for the kind reminder of feature freeze date for 1.9.0. I
> think
> > > the
> > > > date makes sense on my side.
> > > >
> > > > For FLIP-31, I and Andrey could be done within two weeks or so.
> > > > And I already finished my side work for FLIP-1.
> > > >
> > > > Best,
> > > > Zhijiang
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > From:Timo Walther <tw...@apache.org>
> > > > Send Time:2019年5月28日(星期二) 19:26
> > > > To:dev <de...@flink.apache.org>
> > > > Subject:Re: [DISCUSS] Features for Apache Flink 1.9.0
> > > >
> > > > Thanks for being the release managers, Kurt and Gordon!
> > > >
> > > > From the Table & SQL API side, there are still a lot of open issues
> > > > that need to be solved to decouple the API from a planner and enable
> > the
> > > > Blink planner. Also we need to make sure that the Blink planner
> > supports
> > > > at least everything of Flink 1.8 to not introduce a regression. We
> > might
> > > > need to focus more on the main features which is a runnable Blink
> > > > planner and might need to postpone other discussions such as DDL, new
> > > > source/sink interfaces, or proper type inference logic. However, in
> > many
> > > > cases there are shortcuts that we could take in order to achieve our
> > > > goals. So I'm confident that we solve the big blockers until the
> > feature
> > > > freeze :)
> > > >
> > > > I will keep you updated.
> > > >
> > > > Thanks,
> > > > Timo
> > > >
> > > >
> > > > Am 28.05.19 um 05:07 schrieb Kurt Young:
> > > > > Thanks Gordon for bringing this up.
> > > > >
> > > > > I'm glad to say that blink planner merge work is almost done, and i
> > > will
> > > > > follow up the work of
> > > > > integrating blink planner with Table API to co-exist with current
> > flink
> > > > > planner.
> > > > >
> > > > > In addition to this, the following features:
> > > > > 1. FLIP-32: Restructure flink-table for future contributions [1]
> > > > > 2. FLIP-37: Rework of the Table API Type System [2]
> > > > > 3. Hive integration work (including hive meta [3] and connectors)
> > > > >
> > > > > are also going well, i will spend some time to keep track of them.
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
> > > > > [2]
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > [3]
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > >
> > > > > Best,
> > > > > Kurt
> > > > >
> > > > >
> > > > > On Mon, May 27, 2019 at 7:18 PM jincheng sun <
> > sunjincheng121@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> Hi Gordon,
> > > > >>
> > > > >> Thanks for mention the feature freeze date for 1.9.0, that's very
> > > > helpful
> > > > >> for contributors to evaluate their dev plan！
> > > > >>
> > > > >> Regarding FLIP-29, we are glad to do our best to finish the dev of
> > > > FLIP-29,
> > > > >> then catch up with the release of 1.9.
> > > > >>
> > > > >> Thanks again for push the release of 1.9.0 forward!
> > > > >>
> > > > >> Cheers,
> > > > >> Jincheng
> > > > >>
> > > > >>
> > > > >>
> > > > >> Tzu-Li (Gordon) Tai <tz...@apache.org> 于2019年5月27日周一 下午5:48写道：
> > > > >>
> > > > >>> Hi all,
> > > > >>>
> > > > >>> I want to kindly remind the community that we're now 5 weeks away
> > > from
> > > > >> the
> > > > >>> proposed feature freeze date for 1.9.0, which is June 28.
> > > > >>>
> > > > >>> This is not yet a final date we have agreed on, so I would like
> to
> > > > start
> > > > >>> collecting feedback on how the mentioned features are going, and
> in
> > > > >>> general, whether or not the date sounds reasonable given the
> > current
> > > > >> status
> > > > >>> of the ongoing efforts.
> > > > >>> Please let me know what you think!
> > > > >>>
> > > > >>> Cheers,
> > > > >>> Gordon
> > > > >>>
> > > > >>>
> > > > >>> On Mon, May 27, 2019 at 5:40 PM Tzu-Li (Gordon) Tai <
> > > > tzulitai@apache.org
> > > > >>>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> @Hequn @Jincheng
> > > > >>>>
> > > > >>>> Thanks for bringing up FLIP-29 to attention.
> > > > >>>> As previously mentioned, the original list is not a fixed
> feature
> > > set,
> > > > >> so
> > > > >>>> if FLIP-29 has ongoing efforts and can make it before the
> feature
> > > > >> freeze,
> > > > >>>> then of course it should be included!
> > > > >>>>
> > > > >>>> @himansh1306
> > > > >>>>
> > > > >>>> Concerning the ORC format for StreamingFileSink, is there
> already
> > a
> > > > >> JIRA
> > > > >>>> ticket tracking that? If not, I suggest to first open one and
> see
> > if
> > > > >>> there
> > > > >>>> are similar interests from committers in adding that.
> > > > >>>>
> > > > >>>>
> > > > >>>> On Sun, May 5, 2019 at 11:19 PM Hequn Cheng <
> chenghequn@gmail.com
> > >
> > > > >>> wrote:
> > > > >>>>> Hi,
> > > > >>>>>
> > > > >>>>> Great job, Gordon! Thanks a lot for driving this and wrapping
> > > > features
> > > > >>> up
> > > > >>>>> to a detailed list. +1 on it!
> > > > >>>>>
> > > > >>>>> Would be great if we can also add flip29 to the list. @jincheng
> > sun
> > > > >>>>> <su...@gmail.com> and I are focusing on it these
> > days. I
> > > > >>> think
> > > > >>>>> these features in flip29 would bring big enhancements to the
> > Table
> > > > >> API.
> > > > >>>>> :-)
> > > > >>>>>
> > > > >>>>> Best, Hequn
> > > > >>>>>
> > > > >>>>> On Sun, May 5, 2019 at 10:41 PM Becket Qin <
> becket.qin@gmail.com
> > >
> > > > >>> wrote:
> > > > >>>>>> Thanks for driving this release, Gordon. +1 on the feature
> list.
> > > > >>>>>>
> > > > >>>>>> This is a pretty exciting and ambitious release!
> > > > >>>>>>
> > > > >>>>>> Cheers,
> > > > >>>>>>
> > > > >>>>>> Jiangjie (Becket) Qin
> > > > >>>>>>
> > > > >>>>>> On Sun, May 5, 2019 at 4:28 PM jincheng sun <
> > > > >> sunjincheng121@gmail.com
> > > > >>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Thanks a lot for being our release manager, Great job!
> > > > >>>>>>>
> > > > >>>>>>> +1 for the feature list and It's better to add FLIP-29
> > > > >>>>>>> <
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739
> > > > >>>>>>>> (Support
> > > > >>>>>>> map/flatMap/aggregate/flatAggregate on TableAPI), as the
> goals
> > of
> > > > >>>>> release
> > > > >>>>>>> 1.9.
> > > > >>>>>>>
> > > > >>>>>>> What do you think?
> > > > >>>>>>>
> > > > >>>>>>> Best,
> > > > >>>>>>> Jincheng
> > > > >>>>>>>
> > > > >>>>>>> Bowen Li <bo...@gmail.com> 于2019年5月5日周日 上午12:47写道：
> > > > >>>>>>>
> > > > >>>>>>>> +1, exciting and ambitious goals, the rough timeline looks
> > > > >>>>> reasonable.
> > > > >>>>>>>> Let's make it happen!
> > > > >>>>>>>>
> > > > >>>>>>>> On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com>
> > > > >> wrote:
> > > > >>>>>>>>> +1 for the 1.9.0 feature list. Excited to see it is
> > happening.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Regards,
> > > > >>>>>>>>> Jark
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <
> > > > >>>>>>>> himansh1306@gmail.com>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> +1 for Protobuf, Hive Metastore integration & features
> > > > >> related
> > > > >>>>> to
> > > > >>>>>>>>>> savepoint.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I was hoping if we could support for ORC File format can
> be
> > > > >>>>> added
> > > > >>>>>> in
> > > > >>>>>>>>>> StreamingFileSink Writer, Currently only Parquet is
> > > > >> supported
> > > > >>>>> with
> > > > >>>>>>>>> respect
> > > > >>>>>>>>>> to columnar file formats
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <
> > > > >>>>> tzulitai@apache.org
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>> Hi community,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Apache Flink 1.8.0 has been released a few weeks ago, so
> > > > >>>>>> naturally,
> > > > >>>>>>>>> it’s
> > > > >>>>>>>>>>> time to start thinking about what we want to aim for
> > > > >> 1.9.0.
> > > > >>>>>>>>>>> Kurt and I had collected some features that would be
> > > > >>>>> reasonable
> > > > >>>>>> to
> > > > >>>>>>>>>> consider
> > > > >>>>>>>>>>> including for the next release, based on talking with
> > > > >>> various
> > > > >>>>>>> people
> > > > >>>>>>>> as
> > > > >>>>>>>>>>> well as observations from mailing list discussions and
> > > > >>>>> questions.
> > > > >>>>>>>>>>> Note that having specific features listed here does not
> > > > >> mean
> > > > >>>>> that
> > > > >>>>>>> no
> > > > >>>>>>>>>> other
> > > > >>>>>>>>>>> pull requests or topics will be reviewed. I am sure that
> > > > >>> there
> > > > >>>>>> are
> > > > >>>>>>>>> other
> > > > >>>>>>>>>>> ongoing efforts that we missed here and will likely make
> > > > >> it
> > > > >>>>> as an
> > > > >>>>>>>>>>> improvement or new feature in the next release. This
> > > > >>>>> discussion
> > > > >>>>>> is
> > > > >>>>>>>>> merely
> > > > >>>>>>>>>>> for bootstrapping a discussion for 1.9, as well as to
> give
> > > > >>>>>>>> contributors
> > > > >>>>>>>>>> an
> > > > >>>>>>>>>>> idea of what the community is looking to focus on in the
> > > > >>> next
> > > > >>>>>>> couple
> > > > >>>>>>>> of
> > > > >>>>>>>>>>> weeks.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> *Proposed features and focus*
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> In the previous major release, Apache Flink 1.8.0, the
> > > > >>>>> community
> > > > >>>>>>> had
> > > > >>>>>>>>>>> prepared for some major Table & SQL additions from the
> > > > >> Blink
> > > > >>>>>>> branch.
> > > > >>>>>>>>> With
> > > > >>>>>>>>>>> this in mind, for the next release, it would be great to
> > > > >>> wind
> > > > >>>>> up
> > > > >>>>>>>> those
> > > > >>>>>>>>>>> efforts by merging in the Blink-based Table / SQL planner
> > > > >>> and
> > > > >>>>>>> runtime
> > > > >>>>>>>>> for
> > > > >>>>>>>>>>> 1.9.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Following Stephan’s previous thread [1] in the mailing
> > > > >> list
> > > > >>>>> about
> > > > >>>>>>>>>> features
> > > > >>>>>>>>>>> in Blink, we should also start focusing on preparing for
> > > > >>>>> Blink’s
> > > > >>>>>>>> other
> > > > >>>>>>>>>>> several enhancements for batch execution. This includes
> > > > >>>>> resource
> > > > >>>>>>>>>>> optimization, fine-grained failover, pluggable shuffle
> > > > >>>>> service,
> > > > >>>>>>>>> adapting
> > > > >>>>>>>>>>> stream operators for batch execution, as well as better
> > > > >>>>>> integration
> > > > >>>>>>>>> with
> > > > >>>>>>>>>>> commonly used systems by batch executions such as Apache
> > > > >>> Hive.
> > > > >>>>>>>>>>> Moreover, besides efforts related to the Blink merge, we
> > > > >>> would
> > > > >>>>>> also
> > > > >>>>>>>>> like
> > > > >>>>>>>>>> us
> > > > >>>>>>>>>>> to work towards pushing forward some of the most
> discussed
> > > > >>> and
> > > > >>>>>>>>>> anticipated
> > > > >>>>>>>>>>> features by the community. Most of these had discussions
> > > > >> in
> > > > >>>>> the
> > > > >>>>>>>> mailing
> > > > >>>>>>>>>>> lists that span multiple releases, and are also
> frequently
> > > > >>>>>> brought
> > > > >>>>>>> up
> > > > >>>>>>>>> in
> > > > >>>>>>>>>>> community events such as Flink Forward. This includes
> > > > >>> features
> > > > >>>>>> such
> > > > >>>>>>>> as
> > > > >>>>>>>>>>> source event-time alignment and the source interface
> > > > >>> rework, a
> > > > >>>>>>>>> savepoint
> > > > >>>>>>>>>>> connector that allows users to manipulate and query state
> > > > >> in
> > > > >>>>>>>>> savepoints,
> > > > >>>>>>>>>>> interactive programming, as well as terminating a job
> > > > >> with a
> > > > >>>>>> final
> > > > >>>>>>>>>>> savepoint.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Last but not least, we have several existing
> contributions
> > > > >>> or
> > > > >>>>>>>>> discussions
> > > > >>>>>>>>>>> for the ecosystem surrounding Flink, which we think is
> > > > >> also
> > > > >>>>> very
> > > > >>>>>>>>> valuable
> > > > >>>>>>>>>>> to try to merge in for 1.9. This includes a web UI rework
> > > > >>>>>> (recently
> > > > >>>>>>>>>> already
> > > > >>>>>>>>>>> merged), active K8s integration, Google PubSub connector,
> > > > >>>>> native
> > > > >>>>>>>>> support
> > > > >>>>>>>>>>> for the Protobuf format, Python support in the Table API,
> > > > >> as
> > > > >>>>> well
> > > > >>>>>>> as
> > > > >>>>>>>>>>> reworking Flink’s support for machine learning.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> To wrap this up as a list of items, some of which already
> > > > >>> have
> > > > >>>>>>> JIRAs
> > > > >>>>>>>> or
> > > > >>>>>>>>>>> mailing list threads to track them:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> - Merge Blink runner for Table & SQL [2]
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Restructure flink-table to separate API from core
> > > > >>>>> runtime
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Make table planners pluggable
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Rework Table / SQL type system to integrate better
> > > > >>> with
> > > > >>>>> the
> > > > >>>>>>> SQL
> > > > >>>>>>>>>>> standard [3]
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Merge Blink planner and runtime for Table / SQL
> > > > >>>>>>>>>>> - Further preparations for more batch execution
> > > > >>>>>> optimization
> > > > >>>>>>>> from
> > > > >>>>>>>>>>> Blink
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Dedicated scheduler component [4]
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Fine grained failover for batch [5]
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Selectable input stream operator [6]
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Pluggable Shuffle Service [7]
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> FLIP-30: Unified Catalog API & Hive metastore
> > > > >>>>> integration
> > > > >>>>>> [8]
> > > > >>>>>>>>>>> - Heavily anticipated / discussed features in the
> > > > >>>>> community
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> FLIP-27: Source interface rework [9]
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Savepoint connector [10]
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> FLIP-34: Terminate / Suspend job with savepoint
> [11]
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> FLIP-36: Interactive Programming [12]
> > > > >>>>>>>>>>> - Ecosystem
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Web UI rework [13]
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Active K8s integration [14]
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Google PubSub connector [15]
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> First-class Protobuf support [16]
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> FLIP-38: Python support in Table API [17]
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> FLIP-39: Flink ML pipeline and libraries on top of
> > > > >>> Table
> > > > >>>>>> API
> > > > >>>>>>>> [18]
> > > > >>>>>>>>>>> *Suggested release timeline*
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Apache Flink 1.8.0 was released earlier this month, so
> > > > >> based
> > > > >>>>> on
> > > > >>>>>> our
> > > > >>>>>>>>> usual
> > > > >>>>>>>>>>> timely release schedule, we should aim for releasing
> 1.9.0
> > > > >>>>> around
> > > > >>>>>>> mid
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>> end July.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Since it seems that this is going to be a fairly large
> > > > >>>>> release,
> > > > >>>>>> to
> > > > >>>>>>>> give
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>> community enough testing time, I propose that the feature
> > > > >>>>> freeze
> > > > >>>>>> to
> > > > >>>>>>>> be
> > > > >>>>>>>>>> near
> > > > >>>>>>>>>>> the end of June (8-9 weeks from now, probable June 28).
> > > > >> This
> > > > >>>>> is
> > > > >>>>>> of
> > > > >>>>>>>>>> course a
> > > > >>>>>>>>>>> ballpark estimation for now; we should follow-up with a
> > > > >>>>> separate
> > > > >>>>>>>> thread
> > > > >>>>>>>>>>> later in the release cycle to prepare contributors with
> an
> > > > >>>>>> official
> > > > >>>>>>>>>> feature
> > > > >>>>>>>>>>> freeze date.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I’d also like to use this opportunity to propose myself
> > > > >> and
> > > > >>>>> Kurt
> > > > >>>>>> as
> > > > >>>>>>>> the
> > > > >>>>>>>>>>> release managers for 1.9.
> > > > >>>>>>>>>>> AFAIK, we did not used to have 2 RMs for a single release
> > > > >> in
> > > > >>>>> the
> > > > >>>>>>>> past,
> > > > >>>>>>>>>> but
> > > > >>>>>>>>>>> 1.9.0 is definitely quite ambitious so it would not hurt
> > > > >> to
> > > > >>>>> have
> > > > >>>>>>> one
> > > > >>>>>>>>> more
> > > > >>>>>>>>>>> on board :) Cheers, Gordon [1]
> > > > >>>>>>>>>>>
> > > > >>
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
> > > > >>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-11439
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/FLINK-12251
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/FLINK-10429
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> [5]
> > > > >>>>>>>>>>>
> > > > >>
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
> > > > >>>>>>>>>>> [6]
> > > > >>>>>>>>>>>
> > > > >>
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
> > > > >>>>>>>>>>> [7] https://issues.apache.org/jira/browse/FLINK-10653
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> [8] https://issues.apache.org/jira/browse/FLINK-11275
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> [9]
> > > > >>>>>>>>>>>
> > > > >>
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
> > > > >>>>>>>>>>> [10] https://issues.apache.org/jira/browse/FLINK-12047
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> [11]
> > > > >>>>>>>>>>>
> > > > >>
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
> > > > >>>>>>>>>>> [12]
> > > > >>>>>>>>>>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> > > > >>>>>>>>>>> [13] https://issues.apache.org/jira/browse/FLINK-10705
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> [14] https://issues.apache.org/jira/browse/FLINK-9953
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> [15] https://issues.apache.org/jira/browse/FLINK-9311
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> [16] https://issues.apache.org/jira/browse/FLINK-11333
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> [17]
> > > > >>>>>>>>>>>
> > > > >>
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> > > > >>>>>>>>>>> [18]
> > > > >>>>>>>>>>>
> > > > >>
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by Bowen Li <bo...@gmail.com>.
For features I'm involved in:
- FLIP-30 unified catalogs APIs [1]: close to be done. On track
- hive integration
- HiveCatalog for persisting Flink/Hive metadata [2]: close to be
done. On track
- hive data connector [3]: input/output format are close to be done.
Was blocked on source/sink interfaces. We had several discussions yesterday
and concluded that we may have a quick working solution out for 1.9. Thus
I'd say on track
- hive functions [4]: Just started. It has major dependencies on
function definitions and type system rework part II. In the last few weeks,
community is mainly focusing on Blink planner and related tasks on
SQL/Table API side as Timo mentioned above, and the work of function
definition just got started this week. I'm working closely with Timo to
push this efforts forward. It's a bit risky but I'm glad we start to make
progress now
- SQL DDL: also had discussions yesterday. Working together with Kurt, we
hope to have at least some basic DDL to offer users an end-to-end working
solution for both Flink and Hive use cases in 1.9
[1]:
- https://issues.apache.org/jira/browse/FLINK-11275
- https://issues.apache.org/jira/browse/FLINK-12625
-
https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
[2]: https://issues.apache.org/jira/browse/FLINK-12755
[3]: https://issues.apache.org/jira/browse/FLINK-10729
[4]: https://issues.apache.org/jira/browse/FLINK-12656
On Tue, Jun 4, 2019 at 12:12 AM Till Rohrmann <tr...@apache.org> wrote:
> Thanks for starting this discussion Gordon and Kurt. For the development
> threads I'm involved with here are the updates:
>
> * Pluggable scheduler: Good part of the work is completed. Gary now works
> on the glue code to use the new high level scheduler components. The
> estimate to finish this work is end of June (estimate: 4 weeks starting
> from this week). The changes to the scheduler would benefit from very
> thorough testing because they are core to Flink.
>
> * External shuffle service: As Zhijiang said, we hope to finish the work by
> the end of this week or early next week (estimate: 1 week from now).
>
> * Result partition life cycle management / fine grained recovery: The
> current estimate to complete this feature would be end of next week or
> beginning of the week afterwards (estimate: 2 weeks from now). This feature
> should enable fine grained recovery for batch.
>
> * Java 9 support: Flink builds with Java 9. Not all e2e tests are running
> with Java 9 though.
>
> * Active K8s integration: PRs are open but reviews are still pending.
>
> Cheers,
> Till
>
> On Wed, May 29, 2019 at 4:45 AM Biao Liu <mm...@gmail.com> wrote:
>
> > Thanks for being the release manager, Gordon & Kurt.
> >
> > For FLIP-27, there are still some more details need to discuss. I don't
> > think it could catch up the release of 1.9. @Aljoscha, @Stephan, do you
> > agree that?
> >
> > zhijiang <wa...@aliyun.com.invalid> 于2019年5月28日周二 下午11:28写道：
> >
> > > Hi Gordon,
> > >
> > > Thanks for the kind reminder of feature freeze date for 1.9.0. I think
> > the
> > > date makes sense on my side.
> > >
> > > For FLIP-31, I and Andrey could be done within two weeks or so.
> > > And I already finished my side work for FLIP-1.
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > ------------------------------------------------------------------
> > > From:Timo Walther <tw...@apache.org>
> > > Send Time:2019年5月28日(星期二) 19:26
> > > To:dev <de...@flink.apache.org>
> > > Subject:Re: [DISCUSS] Features for Apache Flink 1.9.0
> > >
> > > Thanks for being the release managers, Kurt and Gordon!
> > >
> > > From the Table & SQL API side, there are still a lot of open issues
> > > that need to be solved to decouple the API from a planner and enable
> the
> > > Blink planner. Also we need to make sure that the Blink planner
> supports
> > > at least everything of Flink 1.8 to not introduce a regression. We
> might
> > > need to focus more on the main features which is a runnable Blink
> > > planner and might need to postpone other discussions such as DDL, new
> > > source/sink interfaces, or proper type inference logic. However, in
> many
> > > cases there are shortcuts that we could take in order to achieve our
> > > goals. So I'm confident that we solve the big blockers until the
> feature
> > > freeze :)
> > >
> > > I will keep you updated.
> > >
> > > Thanks,
> > > Timo
> > >
> > >
> > > Am 28.05.19 um 05:07 schrieb Kurt Young:
> > > > Thanks Gordon for bringing this up.
> > > >
> > > > I'm glad to say that blink planner merge work is almost done, and i
> > will
> > > > follow up the work of
> > > > integrating blink planner with Table API to co-exist with current
> flink
> > > > planner.
> > > >
> > > > In addition to this, the following features:
> > > > 1. FLIP-32: Restructure flink-table for future contributions [1]
> > > > 2. FLIP-37: Rework of the Table API Type System [2]
> > > > 3. Hive integration work (including hive meta [3] and connectors)
> > > >
> > > > are also going well, i will spend some time to keep track of them.
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
> > > > [2]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > [3]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > >
> > > > Best,
> > > > Kurt
> > > >
> > > >
> > > > On Mon, May 27, 2019 at 7:18 PM jincheng sun <
> sunjincheng121@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Hi Gordon,
> > > >>
> > > >> Thanks for mention the feature freeze date for 1.9.0, that's very
> > > helpful
> > > >> for contributors to evaluate their dev plan！
> > > >>
> > > >> Regarding FLIP-29, we are glad to do our best to finish the dev of
> > > FLIP-29,
> > > >> then catch up with the release of 1.9.
> > > >>
> > > >> Thanks again for push the release of 1.9.0 forward!
> > > >>
> > > >> Cheers,
> > > >> Jincheng
> > > >>
> > > >>
> > > >>
> > > >> Tzu-Li (Gordon) Tai <tz...@apache.org> 于2019年5月27日周一 下午5:48写道：
> > > >>
> > > >>> Hi all,
> > > >>>
> > > >>> I want to kindly remind the community that we're now 5 weeks away
> > from
> > > >> the
> > > >>> proposed feature freeze date for 1.9.0, which is June 28.
> > > >>>
> > > >>> This is not yet a final date we have agreed on, so I would like to
> > > start
> > > >>> collecting feedback on how the mentioned features are going, and in
> > > >>> general, whether or not the date sounds reasonable given the
> current
> > > >> status
> > > >>> of the ongoing efforts.
> > > >>> Please let me know what you think!
> > > >>>
> > > >>> Cheers,
> > > >>> Gordon
> > > >>>
> > > >>>
> > > >>> On Mon, May 27, 2019 at 5:40 PM Tzu-Li (Gordon) Tai <
> > > tzulitai@apache.org
> > > >>>
> > > >>> wrote:
> > > >>>
> > > >>>> @Hequn @Jincheng
> > > >>>>
> > > >>>> Thanks for bringing up FLIP-29 to attention.
> > > >>>> As previously mentioned, the original list is not a fixed feature
> > set,
> > > >> so
> > > >>>> if FLIP-29 has ongoing efforts and can make it before the feature
> > > >> freeze,
> > > >>>> then of course it should be included!
> > > >>>>
> > > >>>> @himansh1306
> > > >>>>
> > > >>>> Concerning the ORC format for StreamingFileSink, is there already
> a
> > > >> JIRA
> > > >>>> ticket tracking that? If not, I suggest to first open one and see
> if
> > > >>> there
> > > >>>> are similar interests from committers in adding that.
> > > >>>>
> > > >>>>
> > > >>>> On Sun, May 5, 2019 at 11:19 PM Hequn Cheng <chenghequn@gmail.com
> >
> > > >>> wrote:
> > > >>>>> Hi,
> > > >>>>>
> > > >>>>> Great job, Gordon! Thanks a lot for driving this and wrapping
> > > features
> > > >>> up
> > > >>>>> to a detailed list. +1 on it!
> > > >>>>>
> > > >>>>> Would be great if we can also add flip29 to the list. @jincheng
> sun
> > > >>>>> <su...@gmail.com> and I are focusing on it these
> days. I
> > > >>> think
> > > >>>>> these features in flip29 would bring big enhancements to the
> Table
> > > >> API.
> > > >>>>> :-)
> > > >>>>>
> > > >>>>> Best, Hequn
> > > >>>>>
> > > >>>>> On Sun, May 5, 2019 at 10:41 PM Becket Qin <becket.qin@gmail.com
> >
> > > >>> wrote:
> > > >>>>>> Thanks for driving this release, Gordon. +1 on the feature list.
> > > >>>>>>
> > > >>>>>> This is a pretty exciting and ambitious release!
> > > >>>>>>
> > > >>>>>> Cheers,
> > > >>>>>>
> > > >>>>>> Jiangjie (Becket) Qin
> > > >>>>>>
> > > >>>>>> On Sun, May 5, 2019 at 4:28 PM jincheng sun <
> > > >> sunjincheng121@gmail.com
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Thanks a lot for being our release manager, Great job!
> > > >>>>>>>
> > > >>>>>>> +1 for the feature list and It's better to add FLIP-29
> > > >>>>>>> <
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739
> > > >>>>>>>> (Support
> > > >>>>>>> map/flatMap/aggregate/flatAggregate on TableAPI), as the goals
> of
> > > >>>>> release
> > > >>>>>>> 1.9.
> > > >>>>>>>
> > > >>>>>>> What do you think?
> > > >>>>>>>
> > > >>>>>>> Best,
> > > >>>>>>> Jincheng
> > > >>>>>>>
> > > >>>>>>> Bowen Li <bo...@gmail.com> 于2019年5月5日周日 上午12:47写道：
> > > >>>>>>>
> > > >>>>>>>> +1, exciting and ambitious goals, the rough timeline looks
> > > >>>>> reasonable.
> > > >>>>>>>> Let's make it happen!
> > > >>>>>>>>
> > > >>>>>>>> On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com>
> > > >> wrote:
> > > >>>>>>>>> +1 for the 1.9.0 feature list. Excited to see it is
> happening.
> > > >>>>>>>>>
> > > >>>>>>>>> Regards,
> > > >>>>>>>>> Jark
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <
> > > >>>>>>>> himansh1306@gmail.com>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> +1 for Protobuf, Hive Metastore integration & features
> > > >> related
> > > >>>>> to
> > > >>>>>>>>>> savepoint.
> > > >>>>>>>>>>
> > > >>>>>>>>>> I was hoping if we could support for ORC File format can be
> > > >>>>> added
> > > >>>>>> in
> > > >>>>>>>>>> StreamingFileSink Writer, Currently only Parquet is
> > > >> supported
> > > >>>>> with
> > > >>>>>>>>> respect
> > > >>>>>>>>>> to columnar file formats
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <
> > > >>>>> tzulitai@apache.org
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>> Hi community,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Apache Flink 1.8.0 has been released a few weeks ago, so
> > > >>>>>> naturally,
> > > >>>>>>>>> it’s
> > > >>>>>>>>>>> time to start thinking about what we want to aim for
> > > >> 1.9.0.
> > > >>>>>>>>>>> Kurt and I had collected some features that would be
> > > >>>>> reasonable
> > > >>>>>> to
> > > >>>>>>>>>> consider
> > > >>>>>>>>>>> including for the next release, based on talking with
> > > >>> various
> > > >>>>>>> people
> > > >>>>>>>> as
> > > >>>>>>>>>>> well as observations from mailing list discussions and
> > > >>>>> questions.
> > > >>>>>>>>>>> Note that having specific features listed here does not
> > > >> mean
> > > >>>>> that
> > > >>>>>>> no
> > > >>>>>>>>>> other
> > > >>>>>>>>>>> pull requests or topics will be reviewed. I am sure that
> > > >>> there
> > > >>>>>> are
> > > >>>>>>>>> other
> > > >>>>>>>>>>> ongoing efforts that we missed here and will likely make
> > > >> it
> > > >>>>> as an
> > > >>>>>>>>>>> improvement or new feature in the next release. This
> > > >>>>> discussion
> > > >>>>>> is
> > > >>>>>>>>> merely
> > > >>>>>>>>>>> for bootstrapping a discussion for 1.9, as well as to give
> > > >>>>>>>> contributors
> > > >>>>>>>>>> an
> > > >>>>>>>>>>> idea of what the community is looking to focus on in the
> > > >>> next
> > > >>>>>>> couple
> > > >>>>>>>> of
> > > >>>>>>>>>>> weeks.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> *Proposed features and focus*
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> In the previous major release, Apache Flink 1.8.0, the
> > > >>>>> community
> > > >>>>>>> had
> > > >>>>>>>>>>> prepared for some major Table & SQL additions from the
> > > >> Blink
> > > >>>>>>> branch.
> > > >>>>>>>>> With
> > > >>>>>>>>>>> this in mind, for the next release, it would be great to
> > > >>> wind
> > > >>>>> up
> > > >>>>>>>> those
> > > >>>>>>>>>>> efforts by merging in the Blink-based Table / SQL planner
> > > >>> and
> > > >>>>>>> runtime
> > > >>>>>>>>> for
> > > >>>>>>>>>>> 1.9.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Following Stephan’s previous thread [1] in the mailing
> > > >> list
> > > >>>>> about
> > > >>>>>>>>>> features
> > > >>>>>>>>>>> in Blink, we should also start focusing on preparing for
> > > >>>>> Blink’s
> > > >>>>>>>> other
> > > >>>>>>>>>>> several enhancements for batch execution. This includes
> > > >>>>> resource
> > > >>>>>>>>>>> optimization, fine-grained failover, pluggable shuffle
> > > >>>>> service,
> > > >>>>>>>>> adapting
> > > >>>>>>>>>>> stream operators for batch execution, as well as better
> > > >>>>>> integration
> > > >>>>>>>>> with
> > > >>>>>>>>>>> commonly used systems by batch executions such as Apache
> > > >>> Hive.
> > > >>>>>>>>>>> Moreover, besides efforts related to the Blink merge, we
> > > >>> would
> > > >>>>>> also
> > > >>>>>>>>> like
> > > >>>>>>>>>> us
> > > >>>>>>>>>>> to work towards pushing forward some of the most discussed
> > > >>> and
> > > >>>>>>>>>> anticipated
> > > >>>>>>>>>>> features by the community. Most of these had discussions
> > > >> in
> > > >>>>> the
> > > >>>>>>>> mailing
> > > >>>>>>>>>>> lists that span multiple releases, and are also frequently
> > > >>>>>> brought
> > > >>>>>>> up
> > > >>>>>>>>> in
> > > >>>>>>>>>>> community events such as Flink Forward. This includes
> > > >>> features
> > > >>>>>> such
> > > >>>>>>>> as
> > > >>>>>>>>>>> source event-time alignment and the source interface
> > > >>> rework, a
> > > >>>>>>>>> savepoint
> > > >>>>>>>>>>> connector that allows users to manipulate and query state
> > > >> in
> > > >>>>>>>>> savepoints,
> > > >>>>>>>>>>> interactive programming, as well as terminating a job
> > > >> with a
> > > >>>>>> final
> > > >>>>>>>>>>> savepoint.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Last but not least, we have several existing contributions
> > > >>> or
> > > >>>>>>>>> discussions
> > > >>>>>>>>>>> for the ecosystem surrounding Flink, which we think is
> > > >> also
> > > >>>>> very
> > > >>>>>>>>> valuable
> > > >>>>>>>>>>> to try to merge in for 1.9. This includes a web UI rework
> > > >>>>>> (recently
> > > >>>>>>>>>> already
> > > >>>>>>>>>>> merged), active K8s integration, Google PubSub connector,
> > > >>>>> native
> > > >>>>>>>>> support
> > > >>>>>>>>>>> for the Protobuf format, Python support in the Table API,
> > > >> as
> > > >>>>> well
> > > >>>>>>> as
> > > >>>>>>>>>>> reworking Flink’s support for machine learning.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> To wrap this up as a list of items, some of which already
> > > >>> have
> > > >>>>>>> JIRAs
> > > >>>>>>>> or
> > > >>>>>>>>>>> mailing list threads to track them:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Merge Blink runner for Table & SQL [2]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Restructure flink-table to separate API from core
> > > >>>>> runtime
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Make table planners pluggable
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Rework Table / SQL type system to integrate better
> > > >>> with
> > > >>>>> the
> > > >>>>>>> SQL
> > > >>>>>>>>>>> standard [3]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Merge Blink planner and runtime for Table / SQL
> > > >>>>>>>>>>> - Further preparations for more batch execution
> > > >>>>>> optimization
> > > >>>>>>>> from
> > > >>>>>>>>>>> Blink
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Dedicated scheduler component [4]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Fine grained failover for batch [5]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Selectable input stream operator [6]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Pluggable Shuffle Service [7]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> FLIP-30: Unified Catalog API & Hive metastore
> > > >>>>> integration
> > > >>>>>> [8]
> > > >>>>>>>>>>> - Heavily anticipated / discussed features in the
> > > >>>>> community
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> FLIP-27: Source interface rework [9]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Savepoint connector [10]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> FLIP-34: Terminate / Suspend job with savepoint [11]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> FLIP-36: Interactive Programming [12]
> > > >>>>>>>>>>> - Ecosystem
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Web UI rework [13]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Active K8s integration [14]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Google PubSub connector [15]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> First-class Protobuf support [16]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> FLIP-38: Python support in Table API [17]
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> FLIP-39: Flink ML pipeline and libraries on top of
> > > >>> Table
> > > >>>>>> API
> > > >>>>>>>> [18]
> > > >>>>>>>>>>> *Suggested release timeline*
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Apache Flink 1.8.0 was released earlier this month, so
> > > >> based
> > > >>>>> on
> > > >>>>>> our
> > > >>>>>>>>> usual
> > > >>>>>>>>>>> timely release schedule, we should aim for releasing 1.9.0
> > > >>>>> around
> > > >>>>>>> mid
> > > >>>>>>>>> to
> > > >>>>>>>>>>> end July.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Since it seems that this is going to be a fairly large
> > > >>>>> release,
> > > >>>>>> to
> > > >>>>>>>> give
> > > >>>>>>>>>> the
> > > >>>>>>>>>>> community enough testing time, I propose that the feature
> > > >>>>> freeze
> > > >>>>>> to
> > > >>>>>>>> be
> > > >>>>>>>>>> near
> > > >>>>>>>>>>> the end of June (8-9 weeks from now, probable June 28).
> > > >> This
> > > >>>>> is
> > > >>>>>> of
> > > >>>>>>>>>> course a
> > > >>>>>>>>>>> ballpark estimation for now; we should follow-up with a
> > > >>>>> separate
> > > >>>>>>>> thread
> > > >>>>>>>>>>> later in the release cycle to prepare contributors with an
> > > >>>>>> official
> > > >>>>>>>>>> feature
> > > >>>>>>>>>>> freeze date.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I’d also like to use this opportunity to propose myself
> > > >> and
> > > >>>>> Kurt
> > > >>>>>> as
> > > >>>>>>>> the
> > > >>>>>>>>>>> release managers for 1.9.
> > > >>>>>>>>>>> AFAIK, we did not used to have 2 RMs for a single release
> > > >> in
> > > >>>>> the
> > > >>>>>>>> past,
> > > >>>>>>>>>> but
> > > >>>>>>>>>>> 1.9.0 is definitely quite ambitious so it would not hurt
> > > >> to
> > > >>>>> have
> > > >>>>>>> one
> > > >>>>>>>>> more
> > > >>>>>>>>>>> on board :) Cheers, Gordon [1]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
> > > >>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-11439
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/FLINK-12251
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/FLINK-10429
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [5]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
> > > >>>>>>>>>>> [6]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
> > > >>>>>>>>>>> [7] https://issues.apache.org/jira/browse/FLINK-10653
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [8] https://issues.apache.org/jira/browse/FLINK-11275
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [9]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
> > > >>>>>>>>>>> [10] https://issues.apache.org/jira/browse/FLINK-12047
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [11]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
> > > >>>>>>>>>>> [12]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> > > >>>>>>>>>>> [13] https://issues.apache.org/jira/browse/FLINK-10705
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [14] https://issues.apache.org/jira/browse/FLINK-9953
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [15] https://issues.apache.org/jira/browse/FLINK-9311
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [16] https://issues.apache.org/jira/browse/FLINK-11333
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [17]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> > > >>>>>>>>>>> [18]
> > > >>>>>>>>>>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
> > >
> > >
> >
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by Till Rohrmann <tr...@apache.org>.
Thanks for starting this discussion Gordon and Kurt. For the development
threads I'm involved with here are the updates:
* Pluggable scheduler: Good part of the work is completed. Gary now works
on the glue code to use the new high level scheduler components. The
estimate to finish this work is end of June (estimate: 4 weeks starting
from this week). The changes to the scheduler would benefit from very
thorough testing because they are core to Flink.
* External shuffle service: As Zhijiang said, we hope to finish the work by
the end of this week or early next week (estimate: 1 week from now).
* Result partition life cycle management / fine grained recovery: The
current estimate to complete this feature would be end of next week or
beginning of the week afterwards (estimate: 2 weeks from now). This feature
should enable fine grained recovery for batch.
* Java 9 support: Flink builds with Java 9. Not all e2e tests are running
with Java 9 though.
* Active K8s integration: PRs are open but reviews are still pending.
Cheers,
Till
On Wed, May 29, 2019 at 4:45 AM Biao Liu <mm...@gmail.com> wrote:
> Thanks for being the release manager, Gordon & Kurt.
>
> For FLIP-27, there are still some more details need to discuss. I don't
> think it could catch up the release of 1.9. @Aljoscha, @Stephan, do you
> agree that?
>
> zhijiang <wa...@aliyun.com.invalid> 于2019年5月28日周二 下午11:28写道：
>
> > Hi Gordon,
> >
> > Thanks for the kind reminder of feature freeze date for 1.9.0. I think
> the
> > date makes sense on my side.
> >
> > For FLIP-31, I and Andrey could be done within two weeks or so.
> > And I already finished my side work for FLIP-1.
> >
> > Best,
> > Zhijiang
> >
> >
> > ------------------------------------------------------------------
> > From:Timo Walther <tw...@apache.org>
> > Send Time:2019年5月28日(星期二) 19:26
> > To:dev <de...@flink.apache.org>
> > Subject:Re: [DISCUSS] Features for Apache Flink 1.9.0
> >
> > Thanks for being the release managers, Kurt and Gordon!
> >
> > From the Table & SQL API side, there are still a lot of open issues
> > that need to be solved to decouple the API from a planner and enable the
> > Blink planner. Also we need to make sure that the Blink planner supports
> > at least everything of Flink 1.8 to not introduce a regression. We might
> > need to focus more on the main features which is a runnable Blink
> > planner and might need to postpone other discussions such as DDL, new
> > source/sink interfaces, or proper type inference logic. However, in many
> > cases there are shortcuts that we could take in order to achieve our
> > goals. So I'm confident that we solve the big blockers until the feature
> > freeze :)
> >
> > I will keep you updated.
> >
> > Thanks,
> > Timo
> >
> >
> > Am 28.05.19 um 05:07 schrieb Kurt Young:
> > > Thanks Gordon for bringing this up.
> > >
> > > I'm glad to say that blink planner merge work is almost done, and i
> will
> > > follow up the work of
> > > integrating blink planner with Table API to co-exist with current flink
> > > planner.
> > >
> > > In addition to this, the following features:
> > > 1. FLIP-32: Restructure flink-table for future contributions [1]
> > > 2. FLIP-37: Rework of the Table API Type System [2]
> > > 3. Hive integration work (including hive meta [3] and connectors)
> > >
> > > are also going well, i will spend some time to keep track of them.
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
> > > [2]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > [3]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > >
> > > Best,
> > > Kurt
> > >
> > >
> > > On Mon, May 27, 2019 at 7:18 PM jincheng sun <sunjincheng121@gmail.com
> >
> > > wrote:
> > >
> > >> Hi Gordon,
> > >>
> > >> Thanks for mention the feature freeze date for 1.9.0, that's very
> > helpful
> > >> for contributors to evaluate their dev plan！
> > >>
> > >> Regarding FLIP-29, we are glad to do our best to finish the dev of
> > FLIP-29,
> > >> then catch up with the release of 1.9.
> > >>
> > >> Thanks again for push the release of 1.9.0 forward!
> > >>
> > >> Cheers,
> > >> Jincheng
> > >>
> > >>
> > >>
> > >> Tzu-Li (Gordon) Tai <tz...@apache.org> 于2019年5月27日周一 下午5:48写道：
> > >>
> > >>> Hi all,
> > >>>
> > >>> I want to kindly remind the community that we're now 5 weeks away
> from
> > >> the
> > >>> proposed feature freeze date for 1.9.0, which is June 28.
> > >>>
> > >>> This is not yet a final date we have agreed on, so I would like to
> > start
> > >>> collecting feedback on how the mentioned features are going, and in
> > >>> general, whether or not the date sounds reasonable given the current
> > >> status
> > >>> of the ongoing efforts.
> > >>> Please let me know what you think!
> > >>>
> > >>> Cheers,
> > >>> Gordon
> > >>>
> > >>>
> > >>> On Mon, May 27, 2019 at 5:40 PM Tzu-Li (Gordon) Tai <
> > tzulitai@apache.org
> > >>>
> > >>> wrote:
> > >>>
> > >>>> @Hequn @Jincheng
> > >>>>
> > >>>> Thanks for bringing up FLIP-29 to attention.
> > >>>> As previously mentioned, the original list is not a fixed feature
> set,
> > >> so
> > >>>> if FLIP-29 has ongoing efforts and can make it before the feature
> > >> freeze,
> > >>>> then of course it should be included!
> > >>>>
> > >>>> @himansh1306
> > >>>>
> > >>>> Concerning the ORC format for StreamingFileSink, is there already a
> > >> JIRA
> > >>>> ticket tracking that? If not, I suggest to first open one and see if
> > >>> there
> > >>>> are similar interests from committers in adding that.
> > >>>>
> > >>>>
> > >>>> On Sun, May 5, 2019 at 11:19 PM Hequn Cheng <ch...@gmail.com>
> > >>> wrote:
> > >>>>> Hi,
> > >>>>>
> > >>>>> Great job, Gordon! Thanks a lot for driving this and wrapping
> > features
> > >>> up
> > >>>>> to a detailed list. +1 on it!
> > >>>>>
> > >>>>> Would be great if we can also add flip29 to the list. @jincheng sun
> > >>>>> <su...@gmail.com> and I are focusing on it these days. I
> > >>> think
> > >>>>> these features in flip29 would bring big enhancements to the Table
> > >> API.
> > >>>>> :-)
> > >>>>>
> > >>>>> Best, Hequn
> > >>>>>
> > >>>>> On Sun, May 5, 2019 at 10:41 PM Becket Qin <be...@gmail.com>
> > >>> wrote:
> > >>>>>> Thanks for driving this release, Gordon. +1 on the feature list.
> > >>>>>>
> > >>>>>> This is a pretty exciting and ambitious release!
> > >>>>>>
> > >>>>>> Cheers,
> > >>>>>>
> > >>>>>> Jiangjie (Becket) Qin
> > >>>>>>
> > >>>>>> On Sun, May 5, 2019 at 4:28 PM jincheng sun <
> > >> sunjincheng121@gmail.com
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Thanks a lot for being our release manager, Great job!
> > >>>>>>>
> > >>>>>>> +1 for the feature list and It's better to add FLIP-29
> > >>>>>>> <
> > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739
> > >>>>>>>> (Support
> > >>>>>>> map/flatMap/aggregate/flatAggregate on TableAPI), as the goals of
> > >>>>> release
> > >>>>>>> 1.9.
> > >>>>>>>
> > >>>>>>> What do you think?
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Jincheng
> > >>>>>>>
> > >>>>>>> Bowen Li <bo...@gmail.com> 于2019年5月5日周日 上午12:47写道：
> > >>>>>>>
> > >>>>>>>> +1, exciting and ambitious goals, the rough timeline looks
> > >>>>> reasonable.
> > >>>>>>>> Let's make it happen!
> > >>>>>>>>
> > >>>>>>>> On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com>
> > >> wrote:
> > >>>>>>>>> +1 for the 1.9.0 feature list. Excited to see it is happening.
> > >>>>>>>>>
> > >>>>>>>>> Regards,
> > >>>>>>>>> Jark
> > >>>>>>>>>
> > >>>>>>>>> On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <
> > >>>>>>>> himansh1306@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> +1 for Protobuf, Hive Metastore integration & features
> > >> related
> > >>>>> to
> > >>>>>>>>>> savepoint.
> > >>>>>>>>>>
> > >>>>>>>>>> I was hoping if we could support for ORC File format can be
> > >>>>> added
> > >>>>>> in
> > >>>>>>>>>> StreamingFileSink Writer, Currently only Parquet is
> > >> supported
> > >>>>> with
> > >>>>>>>>> respect
> > >>>>>>>>>> to columnar file formats
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <
> > >>>>> tzulitai@apache.org
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>> Hi community,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Apache Flink 1.8.0 has been released a few weeks ago, so
> > >>>>>> naturally,
> > >>>>>>>>> it’s
> > >>>>>>>>>>> time to start thinking about what we want to aim for
> > >> 1.9.0.
> > >>>>>>>>>>> Kurt and I had collected some features that would be
> > >>>>> reasonable
> > >>>>>> to
> > >>>>>>>>>> consider
> > >>>>>>>>>>> including for the next release, based on talking with
> > >>> various
> > >>>>>>> people
> > >>>>>>>> as
> > >>>>>>>>>>> well as observations from mailing list discussions and
> > >>>>> questions.
> > >>>>>>>>>>> Note that having specific features listed here does not
> > >> mean
> > >>>>> that
> > >>>>>>> no
> > >>>>>>>>>> other
> > >>>>>>>>>>> pull requests or topics will be reviewed. I am sure that
> > >>> there
> > >>>>>> are
> > >>>>>>>>> other
> > >>>>>>>>>>> ongoing efforts that we missed here and will likely make
> > >> it
> > >>>>> as an
> > >>>>>>>>>>> improvement or new feature in the next release. This
> > >>>>> discussion
> > >>>>>> is
> > >>>>>>>>> merely
> > >>>>>>>>>>> for bootstrapping a discussion for 1.9, as well as to give
> > >>>>>>>> contributors
> > >>>>>>>>>> an
> > >>>>>>>>>>> idea of what the community is looking to focus on in the
> > >>> next
> > >>>>>>> couple
> > >>>>>>>> of
> > >>>>>>>>>>> weeks.
> > >>>>>>>>>>>
> > >>>>>>>>>>> *Proposed features and focus*
> > >>>>>>>>>>>
> > >>>>>>>>>>> In the previous major release, Apache Flink 1.8.0, the
> > >>>>> community
> > >>>>>>> had
> > >>>>>>>>>>> prepared for some major Table & SQL additions from the
> > >> Blink
> > >>>>>>> branch.
> > >>>>>>>>> With
> > >>>>>>>>>>> this in mind, for the next release, it would be great to
> > >>> wind
> > >>>>> up
> > >>>>>>>> those
> > >>>>>>>>>>> efforts by merging in the Blink-based Table / SQL planner
> > >>> and
> > >>>>>>> runtime
> > >>>>>>>>> for
> > >>>>>>>>>>> 1.9.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Following Stephan’s previous thread [1] in the mailing
> > >> list
> > >>>>> about
> > >>>>>>>>>> features
> > >>>>>>>>>>> in Blink, we should also start focusing on preparing for
> > >>>>> Blink’s
> > >>>>>>>> other
> > >>>>>>>>>>> several enhancements for batch execution. This includes
> > >>>>> resource
> > >>>>>>>>>>> optimization, fine-grained failover, pluggable shuffle
> > >>>>> service,
> > >>>>>>>>> adapting
> > >>>>>>>>>>> stream operators for batch execution, as well as better
> > >>>>>> integration
> > >>>>>>>>> with
> > >>>>>>>>>>> commonly used systems by batch executions such as Apache
> > >>> Hive.
> > >>>>>>>>>>> Moreover, besides efforts related to the Blink merge, we
> > >>> would
> > >>>>>> also
> > >>>>>>>>> like
> > >>>>>>>>>> us
> > >>>>>>>>>>> to work towards pushing forward some of the most discussed
> > >>> and
> > >>>>>>>>>> anticipated
> > >>>>>>>>>>> features by the community. Most of these had discussions
> > >> in
> > >>>>> the
> > >>>>>>>> mailing
> > >>>>>>>>>>> lists that span multiple releases, and are also frequently
> > >>>>>> brought
> > >>>>>>> up
> > >>>>>>>>> in
> > >>>>>>>>>>> community events such as Flink Forward. This includes
> > >>> features
> > >>>>>> such
> > >>>>>>>> as
> > >>>>>>>>>>> source event-time alignment and the source interface
> > >>> rework, a
> > >>>>>>>>> savepoint
> > >>>>>>>>>>> connector that allows users to manipulate and query state
> > >> in
> > >>>>>>>>> savepoints,
> > >>>>>>>>>>> interactive programming, as well as terminating a job
> > >> with a
> > >>>>>> final
> > >>>>>>>>>>> savepoint.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Last but not least, we have several existing contributions
> > >>> or
> > >>>>>>>>> discussions
> > >>>>>>>>>>> for the ecosystem surrounding Flink, which we think is
> > >> also
> > >>>>> very
> > >>>>>>>>> valuable
> > >>>>>>>>>>> to try to merge in for 1.9. This includes a web UI rework
> > >>>>>> (recently
> > >>>>>>>>>> already
> > >>>>>>>>>>> merged), active K8s integration, Google PubSub connector,
> > >>>>> native
> > >>>>>>>>> support
> > >>>>>>>>>>> for the Protobuf format, Python support in the Table API,
> > >> as
> > >>>>> well
> > >>>>>>> as
> > >>>>>>>>>>> reworking Flink’s support for machine learning.
> > >>>>>>>>>>>
> > >>>>>>>>>>> To wrap this up as a list of items, some of which already
> > >>> have
> > >>>>>>> JIRAs
> > >>>>>>>> or
> > >>>>>>>>>>> mailing list threads to track them:
> > >>>>>>>>>>>
> > >>>>>>>>>>> - Merge Blink runner for Table & SQL [2]
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> Restructure flink-table to separate API from core
> > >>>>> runtime
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> Make table planners pluggable
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> Rework Table / SQL type system to integrate better
> > >>> with
> > >>>>> the
> > >>>>>>> SQL
> > >>>>>>>>>>> standard [3]
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> Merge Blink planner and runtime for Table / SQL
> > >>>>>>>>>>> - Further preparations for more batch execution
> > >>>>>> optimization
> > >>>>>>>> from
> > >>>>>>>>>>> Blink
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> Dedicated scheduler component [4]
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> Fine grained failover for batch [5]
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> Selectable input stream operator [6]
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> Pluggable Shuffle Service [7]
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> FLIP-30: Unified Catalog API & Hive metastore
> > >>>>> integration
> > >>>>>> [8]
> > >>>>>>>>>>> - Heavily anticipated / discussed features in the
> > >>>>> community
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> FLIP-27: Source interface rework [9]
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> Savepoint connector [10]
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> FLIP-34: Terminate / Suspend job with savepoint [11]
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> FLIP-36: Interactive Programming [12]
> > >>>>>>>>>>> - Ecosystem
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> Web UI rework [13]
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> Active K8s integration [14]
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> Google PubSub connector [15]
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> First-class Protobuf support [16]
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> FLIP-38: Python support in Table API [17]
> > >>>>>>>>>>> -
> > >>>>>>>>>>>
> > >>>>>>>>>>> FLIP-39: Flink ML pipeline and libraries on top of
> > >>> Table
> > >>>>>> API
> > >>>>>>>> [18]
> > >>>>>>>>>>> *Suggested release timeline*
> > >>>>>>>>>>>
> > >>>>>>>>>>> Apache Flink 1.8.0 was released earlier this month, so
> > >> based
> > >>>>> on
> > >>>>>> our
> > >>>>>>>>> usual
> > >>>>>>>>>>> timely release schedule, we should aim for releasing 1.9.0
> > >>>>> around
> > >>>>>>> mid
> > >>>>>>>>> to
> > >>>>>>>>>>> end July.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Since it seems that this is going to be a fairly large
> > >>>>> release,
> > >>>>>> to
> > >>>>>>>> give
> > >>>>>>>>>> the
> > >>>>>>>>>>> community enough testing time, I propose that the feature
> > >>>>> freeze
> > >>>>>> to
> > >>>>>>>> be
> > >>>>>>>>>> near
> > >>>>>>>>>>> the end of June (8-9 weeks from now, probable June 28).
> > >> This
> > >>>>> is
> > >>>>>> of
> > >>>>>>>>>> course a
> > >>>>>>>>>>> ballpark estimation for now; we should follow-up with a
> > >>>>> separate
> > >>>>>>>> thread
> > >>>>>>>>>>> later in the release cycle to prepare contributors with an
> > >>>>>> official
> > >>>>>>>>>> feature
> > >>>>>>>>>>> freeze date.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I’d also like to use this opportunity to propose myself
> > >> and
> > >>>>> Kurt
> > >>>>>> as
> > >>>>>>>> the
> > >>>>>>>>>>> release managers for 1.9.
> > >>>>>>>>>>> AFAIK, we did not used to have 2 RMs for a single release
> > >> in
> > >>>>> the
> > >>>>>>>> past,
> > >>>>>>>>>> but
> > >>>>>>>>>>> 1.9.0 is definitely quite ambitious so it would not hurt
> > >> to
> > >>>>> have
> > >>>>>>> one
> > >>>>>>>>> more
> > >>>>>>>>>>> on board :) Cheers, Gordon [1]
> > >>>>>>>>>>>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
> > >>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-11439
> > >>>>>>>>>>>
> > >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/FLINK-12251
> > >>>>>>>>>>>
> > >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/FLINK-10429
> > >>>>>>>>>>>
> > >>>>>>>>>>> [5]
> > >>>>>>>>>>>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
> > >>>>>>>>>>> [6]
> > >>>>>>>>>>>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
> > >>>>>>>>>>> [7] https://issues.apache.org/jira/browse/FLINK-10653
> > >>>>>>>>>>>
> > >>>>>>>>>>> [8] https://issues.apache.org/jira/browse/FLINK-11275
> > >>>>>>>>>>>
> > >>>>>>>>>>> [9]
> > >>>>>>>>>>>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
> > >>>>>>>>>>> [10] https://issues.apache.org/jira/browse/FLINK-12047
> > >>>>>>>>>>>
> > >>>>>>>>>>> [11]
> > >>>>>>>>>>>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
> > >>>>>>>>>>> [12]
> > >>>>>>>>>>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> > >>>>>>>>>>> [13] https://issues.apache.org/jira/browse/FLINK-10705
> > >>>>>>>>>>>
> > >>>>>>>>>>> [14] https://issues.apache.org/jira/browse/FLINK-9953
> > >>>>>>>>>>>
> > >>>>>>>>>>> [15] https://issues.apache.org/jira/browse/FLINK-9311
> > >>>>>>>>>>>
> > >>>>>>>>>>> [16] https://issues.apache.org/jira/browse/FLINK-11333
> > >>>>>>>>>>>
> > >>>>>>>>>>> [17]
> > >>>>>>>>>>>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> > >>>>>>>>>>> [18]
> > >>>>>>>>>>>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
> >
> >
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by Biao Liu <mm...@gmail.com>.
Thanks for being the release manager, Gordon & Kurt.
For FLIP-27, there are still some more details need to discuss. I don't
think it could catch up the release of 1.9. @Aljoscha, @Stephan, do you
agree that?
zhijiang <wa...@aliyun.com.invalid> 于2019年5月28日周二 下午11:28写道：
> Hi Gordon,
>
> Thanks for the kind reminder of feature freeze date for 1.9.0. I think the
> date makes sense on my side.
>
> For FLIP-31, I and Andrey could be done within two weeks or so.
> And I already finished my side work for FLIP-1.
>
> Best,
> Zhijiang
>
>
> ------------------------------------------------------------------
> From:Timo Walther <tw...@apache.org>
> Send Time:2019年5月28日(星期二) 19:26
> To:dev <de...@flink.apache.org>
> Subject:Re: [DISCUSS] Features for Apache Flink 1.9.0
>
> Thanks for being the release managers, Kurt and Gordon!
>
> From the Table & SQL API side, there are still a lot of open issues
> that need to be solved to decouple the API from a planner and enable the
> Blink planner. Also we need to make sure that the Blink planner supports
> at least everything of Flink 1.8 to not introduce a regression. We might
> need to focus more on the main features which is a runnable Blink
> planner and might need to postpone other discussions such as DDL, new
> source/sink interfaces, or proper type inference logic. However, in many
> cases there are shortcuts that we could take in order to achieve our
> goals. So I'm confident that we solve the big blockers until the feature
> freeze :)
>
> I will keep you updated.
>
> Thanks,
> Timo
>
>
> Am 28.05.19 um 05:07 schrieb Kurt Young:
> > Thanks Gordon for bringing this up.
> >
> > I'm glad to say that blink planner merge work is almost done, and i will
> > follow up the work of
> > integrating blink planner with Table API to co-exist with current flink
> > planner.
> >
> > In addition to this, the following features:
> > 1. FLIP-32: Restructure flink-table for future contributions [1]
> > 2. FLIP-37: Rework of the Table API Type System [2]
> > 3. Hive integration work (including hive meta [3] and connectors)
> >
> > are also going well, i will spend some time to keep track of them.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
> > [2]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > [3]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> >
> > Best,
> > Kurt
> >
> >
> > On Mon, May 27, 2019 at 7:18 PM jincheng sun <su...@gmail.com>
> > wrote:
> >
> >> Hi Gordon,
> >>
> >> Thanks for mention the feature freeze date for 1.9.0, that's very
> helpful
> >> for contributors to evaluate their dev plan！
> >>
> >> Regarding FLIP-29, we are glad to do our best to finish the dev of
> FLIP-29,
> >> then catch up with the release of 1.9.
> >>
> >> Thanks again for push the release of 1.9.0 forward!
> >>
> >> Cheers,
> >> Jincheng
> >>
> >>
> >>
> >> Tzu-Li (Gordon) Tai <tz...@apache.org> 于2019年5月27日周一 下午5:48写道：
> >>
> >>> Hi all,
> >>>
> >>> I want to kindly remind the community that we're now 5 weeks away from
> >> the
> >>> proposed feature freeze date for 1.9.0, which is June 28.
> >>>
> >>> This is not yet a final date we have agreed on, so I would like to
> start
> >>> collecting feedback on how the mentioned features are going, and in
> >>> general, whether or not the date sounds reasonable given the current
> >> status
> >>> of the ongoing efforts.
> >>> Please let me know what you think!
> >>>
> >>> Cheers,
> >>> Gordon
> >>>
> >>>
> >>> On Mon, May 27, 2019 at 5:40 PM Tzu-Li (Gordon) Tai <
> tzulitai@apache.org
> >>>
> >>> wrote:
> >>>
> >>>> @Hequn @Jincheng
> >>>>
> >>>> Thanks for bringing up FLIP-29 to attention.
> >>>> As previously mentioned, the original list is not a fixed feature set,
> >> so
> >>>> if FLIP-29 has ongoing efforts and can make it before the feature
> >> freeze,
> >>>> then of course it should be included!
> >>>>
> >>>> @himansh1306
> >>>>
> >>>> Concerning the ORC format for StreamingFileSink, is there already a
> >> JIRA
> >>>> ticket tracking that? If not, I suggest to first open one and see if
> >>> there
> >>>> are similar interests from committers in adding that.
> >>>>
> >>>>
> >>>> On Sun, May 5, 2019 at 11:19 PM Hequn Cheng <ch...@gmail.com>
> >>> wrote:
> >>>>> Hi,
> >>>>>
> >>>>> Great job, Gordon! Thanks a lot for driving this and wrapping
> features
> >>> up
> >>>>> to a detailed list. +1 on it!
> >>>>>
> >>>>> Would be great if we can also add flip29 to the list. @jincheng sun
> >>>>> <su...@gmail.com> and I are focusing on it these days. I
> >>> think
> >>>>> these features in flip29 would bring big enhancements to the Table
> >> API.
> >>>>> :-)
> >>>>>
> >>>>> Best, Hequn
> >>>>>
> >>>>> On Sun, May 5, 2019 at 10:41 PM Becket Qin <be...@gmail.com>
> >>> wrote:
> >>>>>> Thanks for driving this release, Gordon. +1 on the feature list.
> >>>>>>
> >>>>>> This is a pretty exciting and ambitious release!
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Jiangjie (Becket) Qin
> >>>>>>
> >>>>>> On Sun, May 5, 2019 at 4:28 PM jincheng sun <
> >> sunjincheng121@gmail.com
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks a lot for being our release manager, Great job!
> >>>>>>>
> >>>>>>> +1 for the feature list and It's better to add FLIP-29
> >>>>>>> <
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739
> >>>>>>>> (Support
> >>>>>>> map/flatMap/aggregate/flatAggregate on TableAPI), as the goals of
> >>>>> release
> >>>>>>> 1.9.
> >>>>>>>
> >>>>>>> What do you think?
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Jincheng
> >>>>>>>
> >>>>>>> Bowen Li <bo...@gmail.com> 于2019年5月5日周日 上午12:47写道：
> >>>>>>>
> >>>>>>>> +1, exciting and ambitious goals, the rough timeline looks
> >>>>> reasonable.
> >>>>>>>> Let's make it happen!
> >>>>>>>>
> >>>>>>>> On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com>
> >> wrote:
> >>>>>>>>> +1 for the 1.9.0 feature list. Excited to see it is happening.
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Jark
> >>>>>>>>>
> >>>>>>>>> On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <
> >>>>>>>> himansh1306@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> +1 for Protobuf, Hive Metastore integration & features
> >> related
> >>>>> to
> >>>>>>>>>> savepoint.
> >>>>>>>>>>
> >>>>>>>>>> I was hoping if we could support for ORC File format can be
> >>>>> added
> >>>>>> in
> >>>>>>>>>> StreamingFileSink Writer, Currently only Parquet is
> >> supported
> >>>>> with
> >>>>>>>>> respect
> >>>>>>>>>> to columnar file formats
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <
> >>>>> tzulitai@apache.org
> >>>>>>>>>> wrote:
> >>>>>>>>>>> Hi community,
> >>>>>>>>>>>
> >>>>>>>>>>> Apache Flink 1.8.0 has been released a few weeks ago, so
> >>>>>> naturally,
> >>>>>>>>> it’s
> >>>>>>>>>>> time to start thinking about what we want to aim for
> >> 1.9.0.
> >>>>>>>>>>> Kurt and I had collected some features that would be
> >>>>> reasonable
> >>>>>> to
> >>>>>>>>>> consider
> >>>>>>>>>>> including for the next release, based on talking with
> >>> various
> >>>>>>> people
> >>>>>>>> as
> >>>>>>>>>>> well as observations from mailing list discussions and
> >>>>> questions.
> >>>>>>>>>>> Note that having specific features listed here does not
> >> mean
> >>>>> that
> >>>>>>> no
> >>>>>>>>>> other
> >>>>>>>>>>> pull requests or topics will be reviewed. I am sure that
> >>> there
> >>>>>> are
> >>>>>>>>> other
> >>>>>>>>>>> ongoing efforts that we missed here and will likely make
> >> it
> >>>>> as an
> >>>>>>>>>>> improvement or new feature in the next release. This
> >>>>> discussion
> >>>>>> is
> >>>>>>>>> merely
> >>>>>>>>>>> for bootstrapping a discussion for 1.9, as well as to give
> >>>>>>>> contributors
> >>>>>>>>>> an
> >>>>>>>>>>> idea of what the community is looking to focus on in the
> >>> next
> >>>>>>> couple
> >>>>>>>> of
> >>>>>>>>>>> weeks.
> >>>>>>>>>>>
> >>>>>>>>>>> *Proposed features and focus*
> >>>>>>>>>>>
> >>>>>>>>>>> In the previous major release, Apache Flink 1.8.0, the
> >>>>> community
> >>>>>>> had
> >>>>>>>>>>> prepared for some major Table & SQL additions from the
> >> Blink
> >>>>>>> branch.
> >>>>>>>>> With
> >>>>>>>>>>> this in mind, for the next release, it would be great to
> >>> wind
> >>>>> up
> >>>>>>>> those
> >>>>>>>>>>> efforts by merging in the Blink-based Table / SQL planner
> >>> and
> >>>>>>> runtime
> >>>>>>>>> for
> >>>>>>>>>>> 1.9.
> >>>>>>>>>>>
> >>>>>>>>>>> Following Stephan’s previous thread [1] in the mailing
> >> list
> >>>>> about
> >>>>>>>>>> features
> >>>>>>>>>>> in Blink, we should also start focusing on preparing for
> >>>>> Blink’s
> >>>>>>>> other
> >>>>>>>>>>> several enhancements for batch execution. This includes
> >>>>> resource
> >>>>>>>>>>> optimization, fine-grained failover, pluggable shuffle
> >>>>> service,
> >>>>>>>>> adapting
> >>>>>>>>>>> stream operators for batch execution, as well as better
> >>>>>> integration
> >>>>>>>>> with
> >>>>>>>>>>> commonly used systems by batch executions such as Apache
> >>> Hive.
> >>>>>>>>>>> Moreover, besides efforts related to the Blink merge, we
> >>> would
> >>>>>> also
> >>>>>>>>> like
> >>>>>>>>>> us
> >>>>>>>>>>> to work towards pushing forward some of the most discussed
> >>> and
> >>>>>>>>>> anticipated
> >>>>>>>>>>> features by the community. Most of these had discussions
> >> in
> >>>>> the
> >>>>>>>> mailing
> >>>>>>>>>>> lists that span multiple releases, and are also frequently
> >>>>>> brought
> >>>>>>> up
> >>>>>>>>> in
> >>>>>>>>>>> community events such as Flink Forward. This includes
> >>> features
> >>>>>> such
> >>>>>>>> as
> >>>>>>>>>>> source event-time alignment and the source interface
> >>> rework, a
> >>>>>>>>> savepoint
> >>>>>>>>>>> connector that allows users to manipulate and query state
> >> in
> >>>>>>>>> savepoints,
> >>>>>>>>>>> interactive programming, as well as terminating a job
> >> with a
> >>>>>> final
> >>>>>>>>>>> savepoint.
> >>>>>>>>>>>
> >>>>>>>>>>> Last but not least, we have several existing contributions
> >>> or
> >>>>>>>>> discussions
> >>>>>>>>>>> for the ecosystem surrounding Flink, which we think is
> >> also
> >>>>> very
> >>>>>>>>> valuable
> >>>>>>>>>>> to try to merge in for 1.9. This includes a web UI rework
> >>>>>> (recently
> >>>>>>>>>> already
> >>>>>>>>>>> merged), active K8s integration, Google PubSub connector,
> >>>>> native
> >>>>>>>>> support
> >>>>>>>>>>> for the Protobuf format, Python support in the Table API,
> >> as
> >>>>> well
> >>>>>>> as
> >>>>>>>>>>> reworking Flink’s support for machine learning.
> >>>>>>>>>>>
> >>>>>>>>>>> To wrap this up as a list of items, some of which already
> >>> have
> >>>>>>> JIRAs
> >>>>>>>> or
> >>>>>>>>>>> mailing list threads to track them:
> >>>>>>>>>>>
> >>>>>>>>>>> - Merge Blink runner for Table & SQL [2]
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> Restructure flink-table to separate API from core
> >>>>> runtime
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> Make table planners pluggable
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> Rework Table / SQL type system to integrate better
> >>> with
> >>>>> the
> >>>>>>> SQL
> >>>>>>>>>>> standard [3]
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> Merge Blink planner and runtime for Table / SQL
> >>>>>>>>>>> - Further preparations for more batch execution
> >>>>>> optimization
> >>>>>>>> from
> >>>>>>>>>>> Blink
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> Dedicated scheduler component [4]
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> Fine grained failover for batch [5]
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> Selectable input stream operator [6]
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> Pluggable Shuffle Service [7]
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> FLIP-30: Unified Catalog API & Hive metastore
> >>>>> integration
> >>>>>> [8]
> >>>>>>>>>>> - Heavily anticipated / discussed features in the
> >>>>> community
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> FLIP-27: Source interface rework [9]
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> Savepoint connector [10]
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> FLIP-34: Terminate / Suspend job with savepoint [11]
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> FLIP-36: Interactive Programming [12]
> >>>>>>>>>>> - Ecosystem
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> Web UI rework [13]
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> Active K8s integration [14]
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> Google PubSub connector [15]
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> First-class Protobuf support [16]
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> FLIP-38: Python support in Table API [17]
> >>>>>>>>>>> -
> >>>>>>>>>>>
> >>>>>>>>>>> FLIP-39: Flink ML pipeline and libraries on top of
> >>> Table
> >>>>>> API
> >>>>>>>> [18]
> >>>>>>>>>>> *Suggested release timeline*
> >>>>>>>>>>>
> >>>>>>>>>>> Apache Flink 1.8.0 was released earlier this month, so
> >> based
> >>>>> on
> >>>>>> our
> >>>>>>>>> usual
> >>>>>>>>>>> timely release schedule, we should aim for releasing 1.9.0
> >>>>> around
> >>>>>>> mid
> >>>>>>>>> to
> >>>>>>>>>>> end July.
> >>>>>>>>>>>
> >>>>>>>>>>> Since it seems that this is going to be a fairly large
> >>>>> release,
> >>>>>> to
> >>>>>>>> give
> >>>>>>>>>> the
> >>>>>>>>>>> community enough testing time, I propose that the feature
> >>>>> freeze
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>> near
> >>>>>>>>>>> the end of June (8-9 weeks from now, probable June 28).
> >> This
> >>>>> is
> >>>>>> of
> >>>>>>>>>> course a
> >>>>>>>>>>> ballpark estimation for now; we should follow-up with a
> >>>>> separate
> >>>>>>>> thread
> >>>>>>>>>>> later in the release cycle to prepare contributors with an
> >>>>>> official
> >>>>>>>>>> feature
> >>>>>>>>>>> freeze date.
> >>>>>>>>>>>
> >>>>>>>>>>> I’d also like to use this opportunity to propose myself
> >> and
> >>>>> Kurt
> >>>>>> as
> >>>>>>>> the
> >>>>>>>>>>> release managers for 1.9.
> >>>>>>>>>>> AFAIK, we did not used to have 2 RMs for a single release
> >> in
> >>>>> the
> >>>>>>>> past,
> >>>>>>>>>> but
> >>>>>>>>>>> 1.9.0 is definitely quite ambitious so it would not hurt
> >> to
> >>>>> have
> >>>>>>> one
> >>>>>>>>> more
> >>>>>>>>>>> on board :) Cheers, Gordon [1]
> >>>>>>>>>>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
> >>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-11439
> >>>>>>>>>>>
> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/FLINK-12251
> >>>>>>>>>>>
> >>>>>>>>>>> [4] https://issues.apache.org/jira/browse/FLINK-10429
> >>>>>>>>>>>
> >>>>>>>>>>> [5]
> >>>>>>>>>>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
> >>>>>>>>>>> [6]
> >>>>>>>>>>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
> >>>>>>>>>>> [7] https://issues.apache.org/jira/browse/FLINK-10653
> >>>>>>>>>>>
> >>>>>>>>>>> [8] https://issues.apache.org/jira/browse/FLINK-11275
> >>>>>>>>>>>
> >>>>>>>>>>> [9]
> >>>>>>>>>>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
> >>>>>>>>>>> [10] https://issues.apache.org/jira/browse/FLINK-12047
> >>>>>>>>>>>
> >>>>>>>>>>> [11]
> >>>>>>>>>>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
> >>>>>>>>>>> [12]
> >>>>>>>>>>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> >>>>>>>>>>> [13] https://issues.apache.org/jira/browse/FLINK-10705
> >>>>>>>>>>>
> >>>>>>>>>>> [14] https://issues.apache.org/jira/browse/FLINK-9953
> >>>>>>>>>>>
> >>>>>>>>>>> [15] https://issues.apache.org/jira/browse/FLINK-9311
> >>>>>>>>>>>
> >>>>>>>>>>> [16] https://issues.apache.org/jira/browse/FLINK-11333
> >>>>>>>>>>>
> >>>>>>>>>>> [17]
> >>>>>>>>>>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> >>>>>>>>>>> [18]
> >>>>>>>>>>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
>
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by zhijiang <wa...@aliyun.com.INVALID>.
Hi Gordon,
Thanks for the kind reminder of feature freeze date for 1.9.0. I think the date makes sense on my side.
For FLIP-31, I and Andrey could be done within two weeks or so.
And I already finished my side work for FLIP-1.
Best,
Zhijiang
------------------------------------------------------------------
From:Timo Walther <tw...@apache.org>
Send Time:2019年5月28日(星期二) 19:26
To:dev <de...@flink.apache.org>
Subject:Re: [DISCUSS] Features for Apache Flink 1.9.0
Thanks for being the release managers, Kurt and Gordon!
From the Table & SQL API side, there are still a lot of open issues
that need to be solved to decouple the API from a planner and enable the
Blink planner. Also we need to make sure that the Blink planner supports
at least everything of Flink 1.8 to not introduce a regression. We might
need to focus more on the main features which is a runnable Blink
planner and might need to postpone other discussions such as DDL, new
source/sink interfaces, or proper type inference logic. However, in many
cases there are shortcuts that we could take in order to achieve our
goals. So I'm confident that we solve the big blockers until the feature
freeze :)
I will keep you updated.
Thanks,
Timo
Am 28.05.19 um 05:07 schrieb Kurt Young:
> Thanks Gordon for bringing this up.
>
> I'm glad to say that blink planner merge work is almost done, and i will
> follow up the work of
> integrating blink planner with Table API to co-exist with current flink
> planner.
>
> In addition to this, the following features:
> 1. FLIP-32: Restructure flink-table for future contributions [1]
> 2. FLIP-37: Rework of the Table API Type System [2]
> 3. Hive integration work (including hive meta [3] and connectors)
>
> are also going well, i will spend some time to keep track of them.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> [3]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
>
> Best,
> Kurt
>
>
> On Mon, May 27, 2019 at 7:18 PM jincheng sun <su...@gmail.com>
> wrote:
>
>> Hi Gordon,
>>
>> Thanks for mention the feature freeze date for 1.9.0, that's very helpful
>> for contributors to evaluate their dev plan！
>>
>> Regarding FLIP-29, we are glad to do our best to finish the dev of FLIP-29,
>> then catch up with the release of 1.9.
>>
>> Thanks again for push the release of 1.9.0 forward!
>>
>> Cheers,
>> Jincheng
>>
>>
>>
>> Tzu-Li (Gordon) Tai <tz...@apache.org> 于2019年5月27日周一 下午5:48写道：
>>
>>> Hi all,
>>>
>>> I want to kindly remind the community that we're now 5 weeks away from
>> the
>>> proposed feature freeze date for 1.9.0, which is June 28.
>>>
>>> This is not yet a final date we have agreed on, so I would like to start
>>> collecting feedback on how the mentioned features are going, and in
>>> general, whether or not the date sounds reasonable given the current
>> status
>>> of the ongoing efforts.
>>> Please let me know what you think!
>>>
>>> Cheers,
>>> Gordon
>>>
>>>
>>> On Mon, May 27, 2019 at 5:40 PM Tzu-Li (Gordon) Tai <tzulitai@apache.org
>>>
>>> wrote:
>>>
>>>> @Hequn @Jincheng
>>>>
>>>> Thanks for bringing up FLIP-29 to attention.
>>>> As previously mentioned, the original list is not a fixed feature set,
>> so
>>>> if FLIP-29 has ongoing efforts and can make it before the feature
>> freeze,
>>>> then of course it should be included!
>>>>
>>>> @himansh1306
>>>>
>>>> Concerning the ORC format for StreamingFileSink, is there already a
>> JIRA
>>>> ticket tracking that? If not, I suggest to first open one and see if
>>> there
>>>> are similar interests from committers in adding that.
>>>>
>>>>
>>>> On Sun, May 5, 2019 at 11:19 PM Hequn Cheng <ch...@gmail.com>
>>> wrote:
>>>>> Hi,
>>>>>
>>>>> Great job, Gordon! Thanks a lot for driving this and wrapping features
>>> up
>>>>> to a detailed list. +1 on it!
>>>>>
>>>>> Would be great if we can also add flip29 to the list. @jincheng sun
>>>>> <su...@gmail.com> and I are focusing on it these days. I
>>> think
>>>>> these features in flip29 would bring big enhancements to the Table
>> API.
>>>>> :-)
>>>>>
>>>>> Best, Hequn
>>>>>
>>>>> On Sun, May 5, 2019 at 10:41 PM Becket Qin <be...@gmail.com>
>>> wrote:
>>>>>> Thanks for driving this release, Gordon. +1 on the feature list.
>>>>>>
>>>>>> This is a pretty exciting and ambitious release!
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Jiangjie (Becket) Qin
>>>>>>
>>>>>> On Sun, May 5, 2019 at 4:28 PM jincheng sun <
>> sunjincheng121@gmail.com
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks a lot for being our release manager, Great job!
>>>>>>>
>>>>>>> +1 for the feature list and It's better to add FLIP-29
>>>>>>> <
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739
>>>>>>>> (Support
>>>>>>> map/flatMap/aggregate/flatAggregate on TableAPI), as the goals of
>>>>> release
>>>>>>> 1.9.
>>>>>>>
>>>>>>> What do you think?
>>>>>>>
>>>>>>> Best,
>>>>>>> Jincheng
>>>>>>>
>>>>>>> Bowen Li <bo...@gmail.com> 于2019年5月5日周日 上午12:47写道：
>>>>>>>
>>>>>>>> +1, exciting and ambitious goals, the rough timeline looks
>>>>> reasonable.
>>>>>>>> Let's make it happen!
>>>>>>>>
>>>>>>>> On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com>
>> wrote:
>>>>>>>>> +1 for the 1.9.0 feature list. Excited to see it is happening.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Jark
>>>>>>>>>
>>>>>>>>> On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <
>>>>>>>> himansh1306@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> +1 for Protobuf, Hive Metastore integration & features
>> related
>>>>> to
>>>>>>>>>> savepoint.
>>>>>>>>>>
>>>>>>>>>> I was hoping if we could support for ORC File format can be
>>>>> added
>>>>>> in
>>>>>>>>>> StreamingFileSink Writer, Currently only Parquet is
>> supported
>>>>> with
>>>>>>>>> respect
>>>>>>>>>> to columnar file formats
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <
>>>>> tzulitai@apache.org
>>>>>>>>>> wrote:
>>>>>>>>>>> Hi community,
>>>>>>>>>>>
>>>>>>>>>>> Apache Flink 1.8.0 has been released a few weeks ago, so
>>>>>> naturally,
>>>>>>>>> it’s
>>>>>>>>>>> time to start thinking about what we want to aim for
>> 1.9.0.
>>>>>>>>>>> Kurt and I had collected some features that would be
>>>>> reasonable
>>>>>> to
>>>>>>>>>> consider
>>>>>>>>>>> including for the next release, based on talking with
>>> various
>>>>>>> people
>>>>>>>> as
>>>>>>>>>>> well as observations from mailing list discussions and
>>>>> questions.
>>>>>>>>>>> Note that having specific features listed here does not
>> mean
>>>>> that
>>>>>>> no
>>>>>>>>>> other
>>>>>>>>>>> pull requests or topics will be reviewed. I am sure that
>>> there
>>>>>> are
>>>>>>>>> other
>>>>>>>>>>> ongoing efforts that we missed here and will likely make
>> it
>>>>> as an
>>>>>>>>>>> improvement or new feature in the next release. This
>>>>> discussion
>>>>>> is
>>>>>>>>> merely
>>>>>>>>>>> for bootstrapping a discussion for 1.9, as well as to give
>>>>>>>> contributors
>>>>>>>>>> an
>>>>>>>>>>> idea of what the community is looking to focus on in the
>>> next
>>>>>>> couple
>>>>>>>> of
>>>>>>>>>>> weeks.
>>>>>>>>>>>
>>>>>>>>>>> *Proposed features and focus*
>>>>>>>>>>>
>>>>>>>>>>> In the previous major release, Apache Flink 1.8.0, the
>>>>> community
>>>>>>> had
>>>>>>>>>>> prepared for some major Table & SQL additions from the
>> Blink
>>>>>>> branch.
>>>>>>>>> With
>>>>>>>>>>> this in mind, for the next release, it would be great to
>>> wind
>>>>> up
>>>>>>>> those
>>>>>>>>>>> efforts by merging in the Blink-based Table / SQL planner
>>> and
>>>>>>> runtime
>>>>>>>>> for
>>>>>>>>>>> 1.9.
>>>>>>>>>>>
>>>>>>>>>>> Following Stephan’s previous thread [1] in the mailing
>> list
>>>>> about
>>>>>>>>>> features
>>>>>>>>>>> in Blink, we should also start focusing on preparing for
>>>>> Blink’s
>>>>>>>> other
>>>>>>>>>>> several enhancements for batch execution. This includes
>>>>> resource
>>>>>>>>>>> optimization, fine-grained failover, pluggable shuffle
>>>>> service,
>>>>>>>>> adapting
>>>>>>>>>>> stream operators for batch execution, as well as better
>>>>>> integration
>>>>>>>>> with
>>>>>>>>>>> commonly used systems by batch executions such as Apache
>>> Hive.
>>>>>>>>>>> Moreover, besides efforts related to the Blink merge, we
>>> would
>>>>>> also
>>>>>>>>> like
>>>>>>>>>> us
>>>>>>>>>>> to work towards pushing forward some of the most discussed
>>> and
>>>>>>>>>> anticipated
>>>>>>>>>>> features by the community. Most of these had discussions
>> in
>>>>> the
>>>>>>>> mailing
>>>>>>>>>>> lists that span multiple releases, and are also frequently
>>>>>> brought
>>>>>>> up
>>>>>>>>> in
>>>>>>>>>>> community events such as Flink Forward. This includes
>>> features
>>>>>> such
>>>>>>>> as
>>>>>>>>>>> source event-time alignment and the source interface
>>> rework, a
>>>>>>>>> savepoint
>>>>>>>>>>> connector that allows users to manipulate and query state
>> in
>>>>>>>>> savepoints,
>>>>>>>>>>> interactive programming, as well as terminating a job
>> with a
>>>>>> final
>>>>>>>>>>> savepoint.
>>>>>>>>>>>
>>>>>>>>>>> Last but not least, we have several existing contributions
>>> or
>>>>>>>>> discussions
>>>>>>>>>>> for the ecosystem surrounding Flink, which we think is
>> also
>>>>> very
>>>>>>>>> valuable
>>>>>>>>>>> to try to merge in for 1.9. This includes a web UI rework
>>>>>> (recently
>>>>>>>>>> already
>>>>>>>>>>> merged), active K8s integration, Google PubSub connector,
>>>>> native
>>>>>>>>> support
>>>>>>>>>>> for the Protobuf format, Python support in the Table API,
>> as
>>>>> well
>>>>>>> as
>>>>>>>>>>> reworking Flink’s support for machine learning.
>>>>>>>>>>>
>>>>>>>>>>> To wrap this up as a list of items, some of which already
>>> have
>>>>>>> JIRAs
>>>>>>>> or
>>>>>>>>>>> mailing list threads to track them:
>>>>>>>>>>>
>>>>>>>>>>> - Merge Blink runner for Table & SQL [2]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Restructure flink-table to separate API from core
>>>>> runtime
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Make table planners pluggable
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Rework Table / SQL type system to integrate better
>>> with
>>>>> the
>>>>>>> SQL
>>>>>>>>>>> standard [3]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Merge Blink planner and runtime for Table / SQL
>>>>>>>>>>> - Further preparations for more batch execution
>>>>>> optimization
>>>>>>>> from
>>>>>>>>>>> Blink
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Dedicated scheduler component [4]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Fine grained failover for batch [5]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Selectable input stream operator [6]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Pluggable Shuffle Service [7]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> FLIP-30: Unified Catalog API & Hive metastore
>>>>> integration
>>>>>> [8]
>>>>>>>>>>> - Heavily anticipated / discussed features in the
>>>>> community
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> FLIP-27: Source interface rework [9]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Savepoint connector [10]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> FLIP-34: Terminate / Suspend job with savepoint [11]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> FLIP-36: Interactive Programming [12]
>>>>>>>>>>> - Ecosystem
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Web UI rework [13]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Active K8s integration [14]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Google PubSub connector [15]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> First-class Protobuf support [16]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> FLIP-38: Python support in Table API [17]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> FLIP-39: Flink ML pipeline and libraries on top of
>>> Table
>>>>>> API
>>>>>>>> [18]
>>>>>>>>>>> *Suggested release timeline*
>>>>>>>>>>>
>>>>>>>>>>> Apache Flink 1.8.0 was released earlier this month, so
>> based
>>>>> on
>>>>>> our
>>>>>>>>> usual
>>>>>>>>>>> timely release schedule, we should aim for releasing 1.9.0
>>>>> around
>>>>>>> mid
>>>>>>>>> to
>>>>>>>>>>> end July.
>>>>>>>>>>>
>>>>>>>>>>> Since it seems that this is going to be a fairly large
>>>>> release,
>>>>>> to
>>>>>>>> give
>>>>>>>>>> the
>>>>>>>>>>> community enough testing time, I propose that the feature
>>>>> freeze
>>>>>> to
>>>>>>>> be
>>>>>>>>>> near
>>>>>>>>>>> the end of June (8-9 weeks from now, probable June 28).
>> This
>>>>> is
>>>>>> of
>>>>>>>>>> course a
>>>>>>>>>>> ballpark estimation for now; we should follow-up with a
>>>>> separate
>>>>>>>> thread
>>>>>>>>>>> later in the release cycle to prepare contributors with an
>>>>>> official
>>>>>>>>>> feature
>>>>>>>>>>> freeze date.
>>>>>>>>>>>
>>>>>>>>>>> I’d also like to use this opportunity to propose myself
>> and
>>>>> Kurt
>>>>>> as
>>>>>>>> the
>>>>>>>>>>> release managers for 1.9.
>>>>>>>>>>> AFAIK, we did not used to have 2 RMs for a single release
>> in
>>>>> the
>>>>>>>> past,
>>>>>>>>>> but
>>>>>>>>>>> 1.9.0 is definitely quite ambitious so it would not hurt
>> to
>>>>> have
>>>>>>> one
>>>>>>>>> more
>>>>>>>>>>> on board :) Cheers, Gordon [1]
>>>>>>>>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-11439
>>>>>>>>>>>
>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/FLINK-12251
>>>>>>>>>>>
>>>>>>>>>>> [4] https://issues.apache.org/jira/browse/FLINK-10429
>>>>>>>>>>>
>>>>>>>>>>> [5]
>>>>>>>>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
>>>>>>>>>>> [6]
>>>>>>>>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
>>>>>>>>>>> [7] https://issues.apache.org/jira/browse/FLINK-10653
>>>>>>>>>>>
>>>>>>>>>>> [8] https://issues.apache.org/jira/browse/FLINK-11275
>>>>>>>>>>>
>>>>>>>>>>> [9]
>>>>>>>>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
>>>>>>>>>>> [10] https://issues.apache.org/jira/browse/FLINK-12047
>>>>>>>>>>>
>>>>>>>>>>> [11]
>>>>>>>>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
>>>>>>>>>>> [12]
>>>>>>>>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
>>>>>>>>>>> [13] https://issues.apache.org/jira/browse/FLINK-10705
>>>>>>>>>>>
>>>>>>>>>>> [14] https://issues.apache.org/jira/browse/FLINK-9953
>>>>>>>>>>>
>>>>>>>>>>> [15] https://issues.apache.org/jira/browse/FLINK-9311
>>>>>>>>>>>
>>>>>>>>>>> [16] https://issues.apache.org/jira/browse/FLINK-11333
>>>>>>>>>>>
>>>>>>>>>>> [17]
>>>>>>>>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
>>>>>>>>>>> [18]
>>>>>>>>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by Timo Walther <tw...@apache.org>.
Thanks for being the release managers, Kurt and Gordon!
From the Table & SQL API side, there are still a lot of open issues
that need to be solved to decouple the API from a planner and enable the
Blink planner. Also we need to make sure that the Blink planner supports
at least everything of Flink 1.8 to not introduce a regression. We might
need to focus more on the main features which is a runnable Blink
planner and might need to postpone other discussions such as DDL, new
source/sink interfaces, or proper type inference logic. However, in many
cases there are shortcuts that we could take in order to achieve our
goals. So I'm confident that we solve the big blockers until the feature
freeze :)
I will keep you updated.
Thanks,
Timo
Am 28.05.19 um 05:07 schrieb Kurt Young:
> Thanks Gordon for bringing this up.
>
> I'm glad to say that blink planner merge work is almost done, and i will
> follow up the work of
> integrating blink planner with Table API to co-exist with current flink
> planner.
>
> In addition to this, the following features:
> 1. FLIP-32: Restructure flink-table for future contributions [1]
> 2. FLIP-37: Rework of the Table API Type System [2]
> 3. Hive integration work (including hive meta [3] and connectors)
>
> are also going well, i will spend some time to keep track of them.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> [3]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
>
> Best,
> Kurt
>
>
> On Mon, May 27, 2019 at 7:18 PM jincheng sun <su...@gmail.com>
> wrote:
>
>> Hi Gordon,
>>
>> Thanks for mention the feature freeze date for 1.9.0, that's very helpful
>> for contributors to evaluate their dev plan！
>>
>> Regarding FLIP-29, we are glad to do our best to finish the dev of FLIP-29,
>> then catch up with the release of 1.9.
>>
>> Thanks again for push the release of 1.9.0 forward!
>>
>> Cheers,
>> Jincheng
>>
>>
>>
>> Tzu-Li (Gordon) Tai <tz...@apache.org> 于2019年5月27日周一 下午5:48写道：
>>
>>> Hi all,
>>>
>>> I want to kindly remind the community that we're now 5 weeks away from
>> the
>>> proposed feature freeze date for 1.9.0, which is June 28.
>>>
>>> This is not yet a final date we have agreed on, so I would like to start
>>> collecting feedback on how the mentioned features are going, and in
>>> general, whether or not the date sounds reasonable given the current
>> status
>>> of the ongoing efforts.
>>> Please let me know what you think!
>>>
>>> Cheers,
>>> Gordon
>>>
>>>
>>> On Mon, May 27, 2019 at 5:40 PM Tzu-Li (Gordon) Tai <tzulitai@apache.org
>>>
>>> wrote:
>>>
>>>> @Hequn @Jincheng
>>>>
>>>> Thanks for bringing up FLIP-29 to attention.
>>>> As previously mentioned, the original list is not a fixed feature set,
>> so
>>>> if FLIP-29 has ongoing efforts and can make it before the feature
>> freeze,
>>>> then of course it should be included!
>>>>
>>>> @himansh1306
>>>>
>>>> Concerning the ORC format for StreamingFileSink, is there already a
>> JIRA
>>>> ticket tracking that? If not, I suggest to first open one and see if
>>> there
>>>> are similar interests from committers in adding that.
>>>>
>>>>
>>>> On Sun, May 5, 2019 at 11:19 PM Hequn Cheng <ch...@gmail.com>
>>> wrote:
>>>>> Hi,
>>>>>
>>>>> Great job, Gordon! Thanks a lot for driving this and wrapping features
>>> up
>>>>> to a detailed list. +1 on it!
>>>>>
>>>>> Would be great if we can also add flip29 to the list. @jincheng sun
>>>>> <su...@gmail.com> and I are focusing on it these days. I
>>> think
>>>>> these features in flip29 would bring big enhancements to the Table
>> API.
>>>>> :-)
>>>>>
>>>>> Best, Hequn
>>>>>
>>>>> On Sun, May 5, 2019 at 10:41 PM Becket Qin <be...@gmail.com>
>>> wrote:
>>>>>> Thanks for driving this release, Gordon. +1 on the feature list.
>>>>>>
>>>>>> This is a pretty exciting and ambitious release!
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Jiangjie (Becket) Qin
>>>>>>
>>>>>> On Sun, May 5, 2019 at 4:28 PM jincheng sun <
>> sunjincheng121@gmail.com
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks a lot for being our release manager, Great job!
>>>>>>>
>>>>>>> +1 for the feature list and It's better to add FLIP-29
>>>>>>> <
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739
>>>>>>>> (Support
>>>>>>> map/flatMap/aggregate/flatAggregate on TableAPI), as the goals of
>>>>> release
>>>>>>> 1.9.
>>>>>>>
>>>>>>> What do you think?
>>>>>>>
>>>>>>> Best,
>>>>>>> Jincheng
>>>>>>>
>>>>>>> Bowen Li <bo...@gmail.com> 于2019年5月5日周日 上午12:47写道：
>>>>>>>
>>>>>>>> +1, exciting and ambitious goals, the rough timeline looks
>>>>> reasonable.
>>>>>>>> Let's make it happen!
>>>>>>>>
>>>>>>>> On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com>
>> wrote:
>>>>>>>>> +1 for the 1.9.0 feature list. Excited to see it is happening.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Jark
>>>>>>>>>
>>>>>>>>> On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <
>>>>>>>> himansh1306@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> +1 for Protobuf, Hive Metastore integration & features
>> related
>>>>> to
>>>>>>>>>> savepoint.
>>>>>>>>>>
>>>>>>>>>> I was hoping if we could support for ORC File format can be
>>>>> added
>>>>>> in
>>>>>>>>>> StreamingFileSink Writer, Currently only Parquet is
>> supported
>>>>> with
>>>>>>>>> respect
>>>>>>>>>> to columnar file formats
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <
>>>>> tzulitai@apache.org
>>>>>>>>>> wrote:
>>>>>>>>>>> Hi community,
>>>>>>>>>>>
>>>>>>>>>>> Apache Flink 1.8.0 has been released a few weeks ago, so
>>>>>> naturally,
>>>>>>>>> it’s
>>>>>>>>>>> time to start thinking about what we want to aim for
>> 1.9.0.
>>>>>>>>>>> Kurt and I had collected some features that would be
>>>>> reasonable
>>>>>> to
>>>>>>>>>> consider
>>>>>>>>>>> including for the next release, based on talking with
>>> various
>>>>>>> people
>>>>>>>> as
>>>>>>>>>>> well as observations from mailing list discussions and
>>>>> questions.
>>>>>>>>>>> Note that having specific features listed here does not
>> mean
>>>>> that
>>>>>>> no
>>>>>>>>>> other
>>>>>>>>>>> pull requests or topics will be reviewed. I am sure that
>>> there
>>>>>> are
>>>>>>>>> other
>>>>>>>>>>> ongoing efforts that we missed here and will likely make
>> it
>>>>> as an
>>>>>>>>>>> improvement or new feature in the next release. This
>>>>> discussion
>>>>>> is
>>>>>>>>> merely
>>>>>>>>>>> for bootstrapping a discussion for 1.9, as well as to give
>>>>>>>> contributors
>>>>>>>>>> an
>>>>>>>>>>> idea of what the community is looking to focus on in the
>>> next
>>>>>>> couple
>>>>>>>> of
>>>>>>>>>>> weeks.
>>>>>>>>>>>
>>>>>>>>>>> *Proposed features and focus*
>>>>>>>>>>>
>>>>>>>>>>> In the previous major release, Apache Flink 1.8.0, the
>>>>> community
>>>>>>> had
>>>>>>>>>>> prepared for some major Table & SQL additions from the
>> Blink
>>>>>>> branch.
>>>>>>>>> With
>>>>>>>>>>> this in mind, for the next release, it would be great to
>>> wind
>>>>> up
>>>>>>>> those
>>>>>>>>>>> efforts by merging in the Blink-based Table / SQL planner
>>> and
>>>>>>> runtime
>>>>>>>>> for
>>>>>>>>>>> 1.9.
>>>>>>>>>>>
>>>>>>>>>>> Following Stephan’s previous thread [1] in the mailing
>> list
>>>>> about
>>>>>>>>>> features
>>>>>>>>>>> in Blink, we should also start focusing on preparing for
>>>>> Blink’s
>>>>>>>> other
>>>>>>>>>>> several enhancements for batch execution. This includes
>>>>> resource
>>>>>>>>>>> optimization, fine-grained failover, pluggable shuffle
>>>>> service,
>>>>>>>>> adapting
>>>>>>>>>>> stream operators for batch execution, as well as better
>>>>>> integration
>>>>>>>>> with
>>>>>>>>>>> commonly used systems by batch executions such as Apache
>>> Hive.
>>>>>>>>>>> Moreover, besides efforts related to the Blink merge, we
>>> would
>>>>>> also
>>>>>>>>> like
>>>>>>>>>> us
>>>>>>>>>>> to work towards pushing forward some of the most discussed
>>> and
>>>>>>>>>> anticipated
>>>>>>>>>>> features by the community. Most of these had discussions
>> in
>>>>> the
>>>>>>>> mailing
>>>>>>>>>>> lists that span multiple releases, and are also frequently
>>>>>> brought
>>>>>>> up
>>>>>>>>> in
>>>>>>>>>>> community events such as Flink Forward. This includes
>>> features
>>>>>> such
>>>>>>>> as
>>>>>>>>>>> source event-time alignment and the source interface
>>> rework, a
>>>>>>>>> savepoint
>>>>>>>>>>> connector that allows users to manipulate and query state
>> in
>>>>>>>>> savepoints,
>>>>>>>>>>> interactive programming, as well as terminating a job
>> with a
>>>>>> final
>>>>>>>>>>> savepoint.
>>>>>>>>>>>
>>>>>>>>>>> Last but not least, we have several existing contributions
>>> or
>>>>>>>>> discussions
>>>>>>>>>>> for the ecosystem surrounding Flink, which we think is
>> also
>>>>> very
>>>>>>>>> valuable
>>>>>>>>>>> to try to merge in for 1.9. This includes a web UI rework
>>>>>> (recently
>>>>>>>>>> already
>>>>>>>>>>> merged), active K8s integration, Google PubSub connector,
>>>>> native
>>>>>>>>> support
>>>>>>>>>>> for the Protobuf format, Python support in the Table API,
>> as
>>>>> well
>>>>>>> as
>>>>>>>>>>> reworking Flink’s support for machine learning.
>>>>>>>>>>>
>>>>>>>>>>> To wrap this up as a list of items, some of which already
>>> have
>>>>>>> JIRAs
>>>>>>>> or
>>>>>>>>>>> mailing list threads to track them:
>>>>>>>>>>>
>>>>>>>>>>> - Merge Blink runner for Table & SQL [2]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Restructure flink-table to separate API from core
>>>>> runtime
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Make table planners pluggable
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Rework Table / SQL type system to integrate better
>>> with
>>>>> the
>>>>>>> SQL
>>>>>>>>>>> standard [3]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Merge Blink planner and runtime for Table / SQL
>>>>>>>>>>> - Further preparations for more batch execution
>>>>>> optimization
>>>>>>>> from
>>>>>>>>>>> Blink
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Dedicated scheduler component [4]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Fine grained failover for batch [5]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Selectable input stream operator [6]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Pluggable Shuffle Service [7]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> FLIP-30: Unified Catalog API & Hive metastore
>>>>> integration
>>>>>> [8]
>>>>>>>>>>> - Heavily anticipated / discussed features in the
>>>>> community
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> FLIP-27: Source interface rework [9]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Savepoint connector [10]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> FLIP-34: Terminate / Suspend job with savepoint [11]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> FLIP-36: Interactive Programming [12]
>>>>>>>>>>> - Ecosystem
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Web UI rework [13]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Active K8s integration [14]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> Google PubSub connector [15]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> First-class Protobuf support [16]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> FLIP-38: Python support in Table API [17]
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>>> FLIP-39: Flink ML pipeline and libraries on top of
>>> Table
>>>>>> API
>>>>>>>> [18]
>>>>>>>>>>> *Suggested release timeline*
>>>>>>>>>>>
>>>>>>>>>>> Apache Flink 1.8.0 was released earlier this month, so
>> based
>>>>> on
>>>>>> our
>>>>>>>>> usual
>>>>>>>>>>> timely release schedule, we should aim for releasing 1.9.0
>>>>> around
>>>>>>> mid
>>>>>>>>> to
>>>>>>>>>>> end July.
>>>>>>>>>>>
>>>>>>>>>>> Since it seems that this is going to be a fairly large
>>>>> release,
>>>>>> to
>>>>>>>> give
>>>>>>>>>> the
>>>>>>>>>>> community enough testing time, I propose that the feature
>>>>> freeze
>>>>>> to
>>>>>>>> be
>>>>>>>>>> near
>>>>>>>>>>> the end of June (8-9 weeks from now, probable June 28).
>> This
>>>>> is
>>>>>> of
>>>>>>>>>> course a
>>>>>>>>>>> ballpark estimation for now; we should follow-up with a
>>>>> separate
>>>>>>>> thread
>>>>>>>>>>> later in the release cycle to prepare contributors with an
>>>>>> official
>>>>>>>>>> feature
>>>>>>>>>>> freeze date.
>>>>>>>>>>>
>>>>>>>>>>> I’d also like to use this opportunity to propose myself
>> and
>>>>> Kurt
>>>>>> as
>>>>>>>> the
>>>>>>>>>>> release managers for 1.9.
>>>>>>>>>>> AFAIK, we did not used to have 2 RMs for a single release
>> in
>>>>> the
>>>>>>>> past,
>>>>>>>>>> but
>>>>>>>>>>> 1.9.0 is definitely quite ambitious so it would not hurt
>> to
>>>>> have
>>>>>>> one
>>>>>>>>> more
>>>>>>>>>>> on board :) Cheers, Gordon [1]
>>>>>>>>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-11439
>>>>>>>>>>>
>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/FLINK-12251
>>>>>>>>>>>
>>>>>>>>>>> [4] https://issues.apache.org/jira/browse/FLINK-10429
>>>>>>>>>>>
>>>>>>>>>>> [5]
>>>>>>>>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
>>>>>>>>>>> [6]
>>>>>>>>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
>>>>>>>>>>> [7] https://issues.apache.org/jira/browse/FLINK-10653
>>>>>>>>>>>
>>>>>>>>>>> [8] https://issues.apache.org/jira/browse/FLINK-11275
>>>>>>>>>>>
>>>>>>>>>>> [9]
>>>>>>>>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
>>>>>>>>>>> [10] https://issues.apache.org/jira/browse/FLINK-12047
>>>>>>>>>>>
>>>>>>>>>>> [11]
>>>>>>>>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
>>>>>>>>>>> [12]
>>>>>>>>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
>>>>>>>>>>> [13] https://issues.apache.org/jira/browse/FLINK-10705
>>>>>>>>>>>
>>>>>>>>>>> [14] https://issues.apache.org/jira/browse/FLINK-9953
>>>>>>>>>>>
>>>>>>>>>>> [15] https://issues.apache.org/jira/browse/FLINK-9311
>>>>>>>>>>>
>>>>>>>>>>> [16] https://issues.apache.org/jira/browse/FLINK-11333
>>>>>>>>>>>
>>>>>>>>>>> [17]
>>>>>>>>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
>>>>>>>>>>> [18]
>>>>>>>>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by Kurt Young <yk...@gmail.com>.
Thanks Gordon for bringing this up.
I'm glad to say that blink planner merge work is almost done, and i will
follow up the work of
integrating blink planner with Table API to co-exist with current flink
planner.
In addition to this, the following features:
1. FLIP-32: Restructure flink-table for future contributions [1]
2. FLIP-37: Rework of the Table API Type System [2]
3. Hive integration work (including hive meta [3] and connectors)
are also going well, i will spend some time to keep track of them.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
[3]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
Best,
Kurt
On Mon, May 27, 2019 at 7:18 PM jincheng sun <su...@gmail.com>
wrote:
> Hi Gordon,
>
> Thanks for mention the feature freeze date for 1.9.0, that's very helpful
> for contributors to evaluate their dev plan！
>
> Regarding FLIP-29, we are glad to do our best to finish the dev of FLIP-29,
> then catch up with the release of 1.9.
>
> Thanks again for push the release of 1.9.0 forward!
>
> Cheers,
> Jincheng
>
>
>
> Tzu-Li (Gordon) Tai <tz...@apache.org> 于2019年5月27日周一 下午5:48写道：
>
> > Hi all,
> >
> > I want to kindly remind the community that we're now 5 weeks away from
> the
> > proposed feature freeze date for 1.9.0, which is June 28.
> >
> > This is not yet a final date we have agreed on, so I would like to start
> > collecting feedback on how the mentioned features are going, and in
> > general, whether or not the date sounds reasonable given the current
> status
> > of the ongoing efforts.
> > Please let me know what you think!
> >
> > Cheers,
> > Gordon
> >
> >
> > On Mon, May 27, 2019 at 5:40 PM Tzu-Li (Gordon) Tai <tzulitai@apache.org
> >
> > wrote:
> >
> > > @Hequn @Jincheng
> > >
> > > Thanks for bringing up FLIP-29 to attention.
> > > As previously mentioned, the original list is not a fixed feature set,
> so
> > > if FLIP-29 has ongoing efforts and can make it before the feature
> freeze,
> > > then of course it should be included!
> > >
> > > @himansh1306
> > >
> > > Concerning the ORC format for StreamingFileSink, is there already a
> JIRA
> > > ticket tracking that? If not, I suggest to first open one and see if
> > there
> > > are similar interests from committers in adding that.
> > >
> > >
> > > On Sun, May 5, 2019 at 11:19 PM Hequn Cheng <ch...@gmail.com>
> > wrote:
> > >
> > >> Hi,
> > >>
> > >> Great job, Gordon! Thanks a lot for driving this and wrapping features
> > up
> > >> to a detailed list. +1 on it!
> > >>
> > >> Would be great if we can also add flip29 to the list. @jincheng sun
> > >> <su...@gmail.com> and I are focusing on it these days. I
> > think
> > >> these features in flip29 would bring big enhancements to the Table
> API.
> > >> :-)
> > >>
> > >> Best, Hequn
> > >>
> > >> On Sun, May 5, 2019 at 10:41 PM Becket Qin <be...@gmail.com>
> > wrote:
> > >>
> > >> > Thanks for driving this release, Gordon. +1 on the feature list.
> > >> >
> > >> > This is a pretty exciting and ambitious release!
> > >> >
> > >> > Cheers,
> > >> >
> > >> > Jiangjie (Becket) Qin
> > >> >
> > >> > On Sun, May 5, 2019 at 4:28 PM jincheng sun <
> sunjincheng121@gmail.com
> > >
> > >> > wrote:
> > >> >
> > >> > > Thanks a lot for being our release manager, Great job!
> > >> > >
> > >> > > +1 for the feature list and It's better to add FLIP-29
> > >> > > <
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739
> > >> > > >(Support
> > >> > > map/flatMap/aggregate/flatAggregate on TableAPI), as the goals of
> > >> release
> > >> > > 1.9.
> > >> > >
> > >> > > What do you think?
> > >> > >
> > >> > > Best,
> > >> > > Jincheng
> > >> > >
> > >> > > Bowen Li <bo...@gmail.com> 于2019年5月5日周日 上午12:47写道：
> > >> > >
> > >> > > > +1, exciting and ambitious goals, the rough timeline looks
> > >> reasonable.
> > >> > > > Let's make it happen!
> > >> > > >
> > >> > > > On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com>
> wrote:
> > >> > > >
> > >> > > > > +1 for the 1.9.0 feature list. Excited to see it is happening.
> > >> > > > >
> > >> > > > > Regards,
> > >> > > > > Jark
> > >> > > > >
> > >> > > > > On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <
> > >> > > > himansh1306@gmail.com>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > +1 for Protobuf, Hive Metastore integration & features
> related
> > >> to
> > >> > > > > > savepoint.
> > >> > > > > >
> > >> > > > > > I was hoping if we could support for ORC File format can be
> > >> added
> > >> > in
> > >> > > > > > StreamingFileSink Writer, Currently only Parquet is
> supported
> > >> with
> > >> > > > > respect
> > >> > > > > > to columnar file formats
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <
> > >> tzulitai@apache.org
> > >> > >
> > >> > > > > > wrote:
> > >> > > > > > > Hi community,
> > >> > > > > > >
> > >> > > > > > > Apache Flink 1.8.0 has been released a few weeks ago, so
> > >> > naturally,
> > >> > > > > it’s
> > >> > > > > > > time to start thinking about what we want to aim for
> 1.9.0.
> > >> > > > > > >
> > >> > > > > > > Kurt and I had collected some features that would be
> > >> reasonable
> > >> > to
> > >> > > > > > consider
> > >> > > > > > > including for the next release, based on talking with
> > various
> > >> > > people
> > >> > > > as
> > >> > > > > > > well as observations from mailing list discussions and
> > >> questions.
> > >> > > > > > >
> > >> > > > > > > Note that having specific features listed here does not
> mean
> > >> that
> > >> > > no
> > >> > > > > > other
> > >> > > > > > > pull requests or topics will be reviewed. I am sure that
> > there
> > >> > are
> > >> > > > > other
> > >> > > > > > > ongoing efforts that we missed here and will likely make
> it
> > >> as an
> > >> > > > > > > improvement or new feature in the next release. This
> > >> discussion
> > >> > is
> > >> > > > > merely
> > >> > > > > > > for bootstrapping a discussion for 1.9, as well as to give
> > >> > > > contributors
> > >> > > > > > an
> > >> > > > > > > idea of what the community is looking to focus on in the
> > next
> > >> > > couple
> > >> > > > of
> > >> > > > > > > weeks.
> > >> > > > > > >
> > >> > > > > > > *Proposed features and focus*
> > >> > > > > > >
> > >> > > > > > > In the previous major release, Apache Flink 1.8.0, the
> > >> community
> > >> > > had
> > >> > > > > > > prepared for some major Table & SQL additions from the
> Blink
> > >> > > branch.
> > >> > > > > With
> > >> > > > > > > this in mind, for the next release, it would be great to
> > wind
> > >> up
> > >> > > > those
> > >> > > > > > > efforts by merging in the Blink-based Table / SQL planner
> > and
> > >> > > runtime
> > >> > > > > for
> > >> > > > > > > 1.9.
> > >> > > > > > >
> > >> > > > > > > Following Stephan’s previous thread [1] in the mailing
> list
> > >> about
> > >> > > > > > features
> > >> > > > > > > in Blink, we should also start focusing on preparing for
> > >> Blink’s
> > >> > > > other
> > >> > > > > > > several enhancements for batch execution. This includes
> > >> resource
> > >> > > > > > > optimization, fine-grained failover, pluggable shuffle
> > >> service,
> > >> > > > > adapting
> > >> > > > > > > stream operators for batch execution, as well as better
> > >> > integration
> > >> > > > > with
> > >> > > > > > > commonly used systems by batch executions such as Apache
> > Hive.
> > >> > > > > > >
> > >> > > > > > > Moreover, besides efforts related to the Blink merge, we
> > would
> > >> > also
> > >> > > > > like
> > >> > > > > > us
> > >> > > > > > > to work towards pushing forward some of the most discussed
> > and
> > >> > > > > > anticipated
> > >> > > > > > > features by the community. Most of these had discussions
> in
> > >> the
> > >> > > > mailing
> > >> > > > > > > lists that span multiple releases, and are also frequently
> > >> > brought
> > >> > > up
> > >> > > > > in
> > >> > > > > > > community events such as Flink Forward. This includes
> > features
> > >> > such
> > >> > > > as
> > >> > > > > > > source event-time alignment and the source interface
> > rework, a
> > >> > > > > savepoint
> > >> > > > > > > connector that allows users to manipulate and query state
> in
> > >> > > > > savepoints,
> > >> > > > > > > interactive programming, as well as terminating a job
> with a
> > >> > final
> > >> > > > > > > savepoint.
> > >> > > > > > >
> > >> > > > > > > Last but not least, we have several existing contributions
> > or
> > >> > > > > discussions
> > >> > > > > > > for the ecosystem surrounding Flink, which we think is
> also
> > >> very
> > >> > > > > valuable
> > >> > > > > > > to try to merge in for 1.9. This includes a web UI rework
> > >> > (recently
> > >> > > > > > already
> > >> > > > > > > merged), active K8s integration, Google PubSub connector,
> > >> native
> > >> > > > > support
> > >> > > > > > > for the Protobuf format, Python support in the Table API,
> as
> > >> well
> > >> > > as
> > >> > > > > > > reworking Flink’s support for machine learning.
> > >> > > > > > >
> > >> > > > > > > To wrap this up as a list of items, some of which already
> > have
> > >> > > JIRAs
> > >> > > > or
> > >> > > > > > > mailing list threads to track them:
> > >> > > > > > >
> > >> > > > > > > - Merge Blink runner for Table & SQL [2]
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > Restructure flink-table to separate API from core
> > >> runtime
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > Make table planners pluggable
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > Rework Table / SQL type system to integrate better
> > with
> > >> the
> > >> > > SQL
> > >> > > > > > > standard [3]
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > Merge Blink planner and runtime for Table / SQL
> > >> > > > > > > - Further preparations for more batch execution
> > >> > optimization
> > >> > > > from
> > >> > > > > > > Blink
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > Dedicated scheduler component [4]
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > Fine grained failover for batch [5]
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > Selectable input stream operator [6]
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > Pluggable Shuffle Service [7]
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > FLIP-30: Unified Catalog API & Hive metastore
> > >> integration
> > >> > [8]
> > >> > > > > > > - Heavily anticipated / discussed features in the
> > >> community
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > FLIP-27: Source interface rework [9]
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > Savepoint connector [10]
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > FLIP-34: Terminate / Suspend job with savepoint [11]
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > FLIP-36: Interactive Programming [12]
> > >> > > > > > > - Ecosystem
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > Web UI rework [13]
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > Active K8s integration [14]
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > Google PubSub connector [15]
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > First-class Protobuf support [16]
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > FLIP-38: Python support in Table API [17]
> > >> > > > > > > -
> > >> > > > > > >
> > >> > > > > > > FLIP-39: Flink ML pipeline and libraries on top of
> > Table
> > >> > API
> > >> > > > [18]
> > >> > > > > > >
> > >> > > > > > > *Suggested release timeline*
> > >> > > > > > >
> > >> > > > > > > Apache Flink 1.8.0 was released earlier this month, so
> based
> > >> on
> > >> > our
> > >> > > > > usual
> > >> > > > > > > timely release schedule, we should aim for releasing 1.9.0
> > >> around
> > >> > > mid
> > >> > > > > to
> > >> > > > > > > end July.
> > >> > > > > > >
> > >> > > > > > > Since it seems that this is going to be a fairly large
> > >> release,
> > >> > to
> > >> > > > give
> > >> > > > > > the
> > >> > > > > > > community enough testing time, I propose that the feature
> > >> freeze
> > >> > to
> > >> > > > be
> > >> > > > > > near
> > >> > > > > > > the end of June (8-9 weeks from now, probable June 28).
> This
> > >> is
> > >> > of
> > >> > > > > > course a
> > >> > > > > > > ballpark estimation for now; we should follow-up with a
> > >> separate
> > >> > > > thread
> > >> > > > > > > later in the release cycle to prepare contributors with an
> > >> > official
> > >> > > > > > feature
> > >> > > > > > > freeze date.
> > >> > > > > > >
> > >> > > > > > > I’d also like to use this opportunity to propose myself
> and
> > >> Kurt
> > >> > as
> > >> > > > the
> > >> > > > > > > release managers for 1.9.
> > >> > > > > > > AFAIK, we did not used to have 2 RMs for a single release
> in
> > >> the
> > >> > > > past,
> > >> > > > > > but
> > >> > > > > > > 1.9.0 is definitely quite ambitious so it would not hurt
> to
> > >> have
> > >> > > one
> > >> > > > > more
> > >> > > > > > > on board :) Cheers, Gordon [1]
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
> > >> > > > > > >
> > >> > > > > > > [2] https://issues.apache.org/jira/browse/FLINK-11439
> > >> > > > > > >
> > >> > > > > > > [3] https://issues.apache.org/jira/browse/FLINK-12251
> > >> > > > > > >
> > >> > > > > > > [4] https://issues.apache.org/jira/browse/FLINK-10429
> > >> > > > > > >
> > >> > > > > > > [5]
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
> > >> > > > > > >
> > >> > > > > > > [6]
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
> > >> > > > > > >
> > >> > > > > > > [7] https://issues.apache.org/jira/browse/FLINK-10653
> > >> > > > > > >
> > >> > > > > > > [8] https://issues.apache.org/jira/browse/FLINK-11275
> > >> > > > > > >
> > >> > > > > > > [9]
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
> > >> > > > > > >
> > >> > > > > > > [10] https://issues.apache.org/jira/browse/FLINK-12047
> > >> > > > > > >
> > >> > > > > > > [11]
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
> > >> > > > > > >
> > >> > > > > > > [12]
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> > >> > > > > > >
> > >> > > > > > > [13] https://issues.apache.org/jira/browse/FLINK-10705
> > >> > > > > > >
> > >> > > > > > > [14] https://issues.apache.org/jira/browse/FLINK-9953
> > >> > > > > > >
> > >> > > > > > > [15] https://issues.apache.org/jira/browse/FLINK-9311
> > >> > > > > > >
> > >> > > > > > > [16] https://issues.apache.org/jira/browse/FLINK-11333
> > >> > > > > > >
> > >> > > > > > > [17]
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> > >> > > > > > > [18]
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by jincheng sun <su...@gmail.com>.
Hi Gordon,
Thanks for mention the feature freeze date for 1.9.0, that's very helpful
for contributors to evaluate their dev plan！
Regarding FLIP-29, we are glad to do our best to finish the dev of FLIP-29,
then catch up with the release of 1.9.
Thanks again for push the release of 1.9.0 forward!
Cheers,
Jincheng
Tzu-Li (Gordon) Tai <tz...@apache.org> 于2019年5月27日周一 下午5:48写道：
> Hi all,
>
> I want to kindly remind the community that we're now 5 weeks away from the
> proposed feature freeze date for 1.9.0, which is June 28.
>
> This is not yet a final date we have agreed on, so I would like to start
> collecting feedback on how the mentioned features are going, and in
> general, whether or not the date sounds reasonable given the current status
> of the ongoing efforts.
> Please let me know what you think!
>
> Cheers,
> Gordon
>
>
> On Mon, May 27, 2019 at 5:40 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
> > @Hequn @Jincheng
> >
> > Thanks for bringing up FLIP-29 to attention.
> > As previously mentioned, the original list is not a fixed feature set, so
> > if FLIP-29 has ongoing efforts and can make it before the feature freeze,
> > then of course it should be included!
> >
> > @himansh1306
> >
> > Concerning the ORC format for StreamingFileSink, is there already a JIRA
> > ticket tracking that? If not, I suggest to first open one and see if
> there
> > are similar interests from committers in adding that.
> >
> >
> > On Sun, May 5, 2019 at 11:19 PM Hequn Cheng <ch...@gmail.com>
> wrote:
> >
> >> Hi,
> >>
> >> Great job, Gordon! Thanks a lot for driving this and wrapping features
> up
> >> to a detailed list. +1 on it!
> >>
> >> Would be great if we can also add flip29 to the list. @jincheng sun
> >> <su...@gmail.com> and I are focusing on it these days. I
> think
> >> these features in flip29 would bring big enhancements to the Table API.
> >> :-)
> >>
> >> Best, Hequn
> >>
> >> On Sun, May 5, 2019 at 10:41 PM Becket Qin <be...@gmail.com>
> wrote:
> >>
> >> > Thanks for driving this release, Gordon. +1 on the feature list.
> >> >
> >> > This is a pretty exciting and ambitious release!
> >> >
> >> > Cheers,
> >> >
> >> > Jiangjie (Becket) Qin
> >> >
> >> > On Sun, May 5, 2019 at 4:28 PM jincheng sun <sunjincheng121@gmail.com
> >
> >> > wrote:
> >> >
> >> > > Thanks a lot for being our release manager, Great job!
> >> > >
> >> > > +1 for the feature list and It's better to add FLIP-29
> >> > > <
> >> >
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739
> >> > > >(Support
> >> > > map/flatMap/aggregate/flatAggregate on TableAPI), as the goals of
> >> release
> >> > > 1.9.
> >> > >
> >> > > What do you think?
> >> > >
> >> > > Best,
> >> > > Jincheng
> >> > >
> >> > > Bowen Li <bo...@gmail.com> 于2019年5月5日周日 上午12:47写道：
> >> > >
> >> > > > +1, exciting and ambitious goals, the rough timeline looks
> >> reasonable.
> >> > > > Let's make it happen!
> >> > > >
> >> > > > On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com> wrote:
> >> > > >
> >> > > > > +1 for the 1.9.0 feature list. Excited to see it is happening.
> >> > > > >
> >> > > > > Regards,
> >> > > > > Jark
> >> > > > >
> >> > > > > On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <
> >> > > > himansh1306@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > +1 for Protobuf, Hive Metastore integration & features related
> >> to
> >> > > > > > savepoint.
> >> > > > > >
> >> > > > > > I was hoping if we could support for ORC File format can be
> >> added
> >> > in
> >> > > > > > StreamingFileSink Writer, Currently only Parquet is supported
> >> with
> >> > > > > respect
> >> > > > > > to columnar file formats
> >> > > > > >
> >> > > > > >
> >> > > > > > On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <
> >> tzulitai@apache.org
> >> > >
> >> > > > > > wrote:
> >> > > > > > > Hi community,
> >> > > > > > >
> >> > > > > > > Apache Flink 1.8.0 has been released a few weeks ago, so
> >> > naturally,
> >> > > > > it’s
> >> > > > > > > time to start thinking about what we want to aim for 1.9.0.
> >> > > > > > >
> >> > > > > > > Kurt and I had collected some features that would be
> >> reasonable
> >> > to
> >> > > > > > consider
> >> > > > > > > including for the next release, based on talking with
> various
> >> > > people
> >> > > > as
> >> > > > > > > well as observations from mailing list discussions and
> >> questions.
> >> > > > > > >
> >> > > > > > > Note that having specific features listed here does not mean
> >> that
> >> > > no
> >> > > > > > other
> >> > > > > > > pull requests or topics will be reviewed. I am sure that
> there
> >> > are
> >> > > > > other
> >> > > > > > > ongoing efforts that we missed here and will likely make it
> >> as an
> >> > > > > > > improvement or new feature in the next release. This
> >> discussion
> >> > is
> >> > > > > merely
> >> > > > > > > for bootstrapping a discussion for 1.9, as well as to give
> >> > > > contributors
> >> > > > > > an
> >> > > > > > > idea of what the community is looking to focus on in the
> next
> >> > > couple
> >> > > > of
> >> > > > > > > weeks.
> >> > > > > > >
> >> > > > > > > *Proposed features and focus*
> >> > > > > > >
> >> > > > > > > In the previous major release, Apache Flink 1.8.0, the
> >> community
> >> > > had
> >> > > > > > > prepared for some major Table & SQL additions from the Blink
> >> > > branch.
> >> > > > > With
> >> > > > > > > this in mind, for the next release, it would be great to
> wind
> >> up
> >> > > > those
> >> > > > > > > efforts by merging in the Blink-based Table / SQL planner
> and
> >> > > runtime
> >> > > > > for
> >> > > > > > > 1.9.
> >> > > > > > >
> >> > > > > > > Following Stephan’s previous thread [1] in the mailing list
> >> about
> >> > > > > > features
> >> > > > > > > in Blink, we should also start focusing on preparing for
> >> Blink’s
> >> > > > other
> >> > > > > > > several enhancements for batch execution. This includes
> >> resource
> >> > > > > > > optimization, fine-grained failover, pluggable shuffle
> >> service,
> >> > > > > adapting
> >> > > > > > > stream operators for batch execution, as well as better
> >> > integration
> >> > > > > with
> >> > > > > > > commonly used systems by batch executions such as Apache
> Hive.
> >> > > > > > >
> >> > > > > > > Moreover, besides efforts related to the Blink merge, we
> would
> >> > also
> >> > > > > like
> >> > > > > > us
> >> > > > > > > to work towards pushing forward some of the most discussed
> and
> >> > > > > > anticipated
> >> > > > > > > features by the community. Most of these had discussions in
> >> the
> >> > > > mailing
> >> > > > > > > lists that span multiple releases, and are also frequently
> >> > brought
> >> > > up
> >> > > > > in
> >> > > > > > > community events such as Flink Forward. This includes
> features
> >> > such
> >> > > > as
> >> > > > > > > source event-time alignment and the source interface
> rework, a
> >> > > > > savepoint
> >> > > > > > > connector that allows users to manipulate and query state in
> >> > > > > savepoints,
> >> > > > > > > interactive programming, as well as terminating a job with a
> >> > final
> >> > > > > > > savepoint.
> >> > > > > > >
> >> > > > > > > Last but not least, we have several existing contributions
> or
> >> > > > > discussions
> >> > > > > > > for the ecosystem surrounding Flink, which we think is also
> >> very
> >> > > > > valuable
> >> > > > > > > to try to merge in for 1.9. This includes a web UI rework
> >> > (recently
> >> > > > > > already
> >> > > > > > > merged), active K8s integration, Google PubSub connector,
> >> native
> >> > > > > support
> >> > > > > > > for the Protobuf format, Python support in the Table API, as
> >> well
> >> > > as
> >> > > > > > > reworking Flink’s support for machine learning.
> >> > > > > > >
> >> > > > > > > To wrap this up as a list of items, some of which already
> have
> >> > > JIRAs
> >> > > > or
> >> > > > > > > mailing list threads to track them:
> >> > > > > > >
> >> > > > > > > - Merge Blink runner for Table & SQL [2]
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > Restructure flink-table to separate API from core
> >> runtime
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > Make table planners pluggable
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > Rework Table / SQL type system to integrate better
> with
> >> the
> >> > > SQL
> >> > > > > > > standard [3]
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > Merge Blink planner and runtime for Table / SQL
> >> > > > > > > - Further preparations for more batch execution
> >> > optimization
> >> > > > from
> >> > > > > > > Blink
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > Dedicated scheduler component [4]
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > Fine grained failover for batch [5]
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > Selectable input stream operator [6]
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > Pluggable Shuffle Service [7]
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > FLIP-30: Unified Catalog API & Hive metastore
> >> integration
> >> > [8]
> >> > > > > > > - Heavily anticipated / discussed features in the
> >> community
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > FLIP-27: Source interface rework [9]
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > Savepoint connector [10]
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > FLIP-34: Terminate / Suspend job with savepoint [11]
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > FLIP-36: Interactive Programming [12]
> >> > > > > > > - Ecosystem
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > Web UI rework [13]
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > Active K8s integration [14]
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > Google PubSub connector [15]
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > First-class Protobuf support [16]
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > FLIP-38: Python support in Table API [17]
> >> > > > > > > -
> >> > > > > > >
> >> > > > > > > FLIP-39: Flink ML pipeline and libraries on top of
> Table
> >> > API
> >> > > > [18]
> >> > > > > > >
> >> > > > > > > *Suggested release timeline*
> >> > > > > > >
> >> > > > > > > Apache Flink 1.8.0 was released earlier this month, so based
> >> on
> >> > our
> >> > > > > usual
> >> > > > > > > timely release schedule, we should aim for releasing 1.9.0
> >> around
> >> > > mid
> >> > > > > to
> >> > > > > > > end July.
> >> > > > > > >
> >> > > > > > > Since it seems that this is going to be a fairly large
> >> release,
> >> > to
> >> > > > give
> >> > > > > > the
> >> > > > > > > community enough testing time, I propose that the feature
> >> freeze
> >> > to
> >> > > > be
> >> > > > > > near
> >> > > > > > > the end of June (8-9 weeks from now, probable June 28). This
> >> is
> >> > of
> >> > > > > > course a
> >> > > > > > > ballpark estimation for now; we should follow-up with a
> >> separate
> >> > > > thread
> >> > > > > > > later in the release cycle to prepare contributors with an
> >> > official
> >> > > > > > feature
> >> > > > > > > freeze date.
> >> > > > > > >
> >> > > > > > > I’d also like to use this opportunity to propose myself and
> >> Kurt
> >> > as
> >> > > > the
> >> > > > > > > release managers for 1.9.
> >> > > > > > > AFAIK, we did not used to have 2 RMs for a single release in
> >> the
> >> > > > past,
> >> > > > > > but
> >> > > > > > > 1.9.0 is definitely quite ambitious so it would not hurt to
> >> have
> >> > > one
> >> > > > > more
> >> > > > > > > on board :) Cheers, Gordon [1]
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
> >> > > > > > >
> >> > > > > > > [2] https://issues.apache.org/jira/browse/FLINK-11439
> >> > > > > > >
> >> > > > > > > [3] https://issues.apache.org/jira/browse/FLINK-12251
> >> > > > > > >
> >> > > > > > > [4] https://issues.apache.org/jira/browse/FLINK-10429
> >> > > > > > >
> >> > > > > > > [5]
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
> >> > > > > > >
> >> > > > > > > [6]
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
> >> > > > > > >
> >> > > > > > > [7] https://issues.apache.org/jira/browse/FLINK-10653
> >> > > > > > >
> >> > > > > > > [8] https://issues.apache.org/jira/browse/FLINK-11275
> >> > > > > > >
> >> > > > > > > [9]
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
> >> > > > > > >
> >> > > > > > > [10] https://issues.apache.org/jira/browse/FLINK-12047
> >> > > > > > >
> >> > > > > > > [11]
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
> >> > > > > > >
> >> > > > > > > [12]
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> >> > > > > > >
> >> > > > > > > [13] https://issues.apache.org/jira/browse/FLINK-10705
> >> > > > > > >
> >> > > > > > > [14] https://issues.apache.org/jira/browse/FLINK-9953
> >> > > > > > >
> >> > > > > > > [15] https://issues.apache.org/jira/browse/FLINK-9311
> >> > > > > > >
> >> > > > > > > [16] https://issues.apache.org/jira/browse/FLINK-11333
> >> > > > > > >
> >> > > > > > > [17]
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> >> > > > > > > [18]
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi all,
I want to kindly remind the community that we're now 5 weeks away from the
proposed feature freeze date for 1.9.0, which is June 28.
This is not yet a final date we have agreed on, so I would like to start
collecting feedback on how the mentioned features are going, and in
general, whether or not the date sounds reasonable given the current status
of the ongoing efforts.
Please let me know what you think!
Cheers,
Gordon
On Mon, May 27, 2019 at 5:40 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> @Hequn @Jincheng
>
> Thanks for bringing up FLIP-29 to attention.
> As previously mentioned, the original list is not a fixed feature set, so
> if FLIP-29 has ongoing efforts and can make it before the feature freeze,
> then of course it should be included!
>
> @himansh1306
>
> Concerning the ORC format for StreamingFileSink, is there already a JIRA
> ticket tracking that? If not, I suggest to first open one and see if there
> are similar interests from committers in adding that.
>
>
> On Sun, May 5, 2019 at 11:19 PM Hequn Cheng <ch...@gmail.com> wrote:
>
>> Hi,
>>
>> Great job, Gordon! Thanks a lot for driving this and wrapping features up
>> to a detailed list. +1 on it!
>>
>> Would be great if we can also add flip29 to the list. @jincheng sun
>> <su...@gmail.com> and I are focusing on it these days. I think
>> these features in flip29 would bring big enhancements to the Table API.
>> :-)
>>
>> Best, Hequn
>>
>> On Sun, May 5, 2019 at 10:41 PM Becket Qin <be...@gmail.com> wrote:
>>
>> > Thanks for driving this release, Gordon. +1 on the feature list.
>> >
>> > This is a pretty exciting and ambitious release!
>> >
>> > Cheers,
>> >
>> > Jiangjie (Becket) Qin
>> >
>> > On Sun, May 5, 2019 at 4:28 PM jincheng sun <su...@gmail.com>
>> > wrote:
>> >
>> > > Thanks a lot for being our release manager, Great job!
>> > >
>> > > +1 for the feature list and It's better to add FLIP-29
>> > > <
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739
>> > > >(Support
>> > > map/flatMap/aggregate/flatAggregate on TableAPI), as the goals of
>> release
>> > > 1.9.
>> > >
>> > > What do you think?
>> > >
>> > > Best,
>> > > Jincheng
>> > >
>> > > Bowen Li <bo...@gmail.com> 于2019年5月5日周日 上午12:47写道：
>> > >
>> > > > +1, exciting and ambitious goals, the rough timeline looks
>> reasonable.
>> > > > Let's make it happen!
>> > > >
>> > > > On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com> wrote:
>> > > >
>> > > > > +1 for the 1.9.0 feature list. Excited to see it is happening.
>> > > > >
>> > > > > Regards,
>> > > > > Jark
>> > > > >
>> > > > > On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <
>> > > > himansh1306@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > +1 for Protobuf, Hive Metastore integration & features related
>> to
>> > > > > > savepoint.
>> > > > > >
>> > > > > > I was hoping if we could support for ORC File format can be
>> added
>> > in
>> > > > > > StreamingFileSink Writer, Currently only Parquet is supported
>> with
>> > > > > respect
>> > > > > > to columnar file formats
>> > > > > >
>> > > > > >
>> > > > > > On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <
>> tzulitai@apache.org
>> > >
>> > > > > > wrote:
>> > > > > > > Hi community,
>> > > > > > >
>> > > > > > > Apache Flink 1.8.0 has been released a few weeks ago, so
>> > naturally,
>> > > > > it’s
>> > > > > > > time to start thinking about what we want to aim for 1.9.0.
>> > > > > > >
>> > > > > > > Kurt and I had collected some features that would be
>> reasonable
>> > to
>> > > > > > consider
>> > > > > > > including for the next release, based on talking with various
>> > > people
>> > > > as
>> > > > > > > well as observations from mailing list discussions and
>> questions.
>> > > > > > >
>> > > > > > > Note that having specific features listed here does not mean
>> that
>> > > no
>> > > > > > other
>> > > > > > > pull requests or topics will be reviewed. I am sure that there
>> > are
>> > > > > other
>> > > > > > > ongoing efforts that we missed here and will likely make it
>> as an
>> > > > > > > improvement or new feature in the next release. This
>> discussion
>> > is
>> > > > > merely
>> > > > > > > for bootstrapping a discussion for 1.9, as well as to give
>> > > > contributors
>> > > > > > an
>> > > > > > > idea of what the community is looking to focus on in the next
>> > > couple
>> > > > of
>> > > > > > > weeks.
>> > > > > > >
>> > > > > > > *Proposed features and focus*
>> > > > > > >
>> > > > > > > In the previous major release, Apache Flink 1.8.0, the
>> community
>> > > had
>> > > > > > > prepared for some major Table & SQL additions from the Blink
>> > > branch.
>> > > > > With
>> > > > > > > this in mind, for the next release, it would be great to wind
>> up
>> > > > those
>> > > > > > > efforts by merging in the Blink-based Table / SQL planner and
>> > > runtime
>> > > > > for
>> > > > > > > 1.9.
>> > > > > > >
>> > > > > > > Following Stephan’s previous thread [1] in the mailing list
>> about
>> > > > > > features
>> > > > > > > in Blink, we should also start focusing on preparing for
>> Blink’s
>> > > > other
>> > > > > > > several enhancements for batch execution. This includes
>> resource
>> > > > > > > optimization, fine-grained failover, pluggable shuffle
>> service,
>> > > > > adapting
>> > > > > > > stream operators for batch execution, as well as better
>> > integration
>> > > > > with
>> > > > > > > commonly used systems by batch executions such as Apache Hive.
>> > > > > > >
>> > > > > > > Moreover, besides efforts related to the Blink merge, we would
>> > also
>> > > > > like
>> > > > > > us
>> > > > > > > to work towards pushing forward some of the most discussed and
>> > > > > > anticipated
>> > > > > > > features by the community. Most of these had discussions in
>> the
>> > > > mailing
>> > > > > > > lists that span multiple releases, and are also frequently
>> > brought
>> > > up
>> > > > > in
>> > > > > > > community events such as Flink Forward. This includes features
>> > such
>> > > > as
>> > > > > > > source event-time alignment and the source interface rework, a
>> > > > > savepoint
>> > > > > > > connector that allows users to manipulate and query state in
>> > > > > savepoints,
>> > > > > > > interactive programming, as well as terminating a job with a
>> > final
>> > > > > > > savepoint.
>> > > > > > >
>> > > > > > > Last but not least, we have several existing contributions or
>> > > > > discussions
>> > > > > > > for the ecosystem surrounding Flink, which we think is also
>> very
>> > > > > valuable
>> > > > > > > to try to merge in for 1.9. This includes a web UI rework
>> > (recently
>> > > > > > already
>> > > > > > > merged), active K8s integration, Google PubSub connector,
>> native
>> > > > > support
>> > > > > > > for the Protobuf format, Python support in the Table API, as
>> well
>> > > as
>> > > > > > > reworking Flink’s support for machine learning.
>> > > > > > >
>> > > > > > > To wrap this up as a list of items, some of which already have
>> > > JIRAs
>> > > > or
>> > > > > > > mailing list threads to track them:
>> > > > > > >
>> > > > > > > - Merge Blink runner for Table & SQL [2]
>> > > > > > > -
>> > > > > > >
>> > > > > > > Restructure flink-table to separate API from core
>> runtime
>> > > > > > > -
>> > > > > > >
>> > > > > > > Make table planners pluggable
>> > > > > > > -
>> > > > > > >
>> > > > > > > Rework Table / SQL type system to integrate better with
>> the
>> > > SQL
>> > > > > > > standard [3]
>> > > > > > > -
>> > > > > > >
>> > > > > > > Merge Blink planner and runtime for Table / SQL
>> > > > > > > - Further preparations for more batch execution
>> > optimization
>> > > > from
>> > > > > > > Blink
>> > > > > > > -
>> > > > > > >
>> > > > > > > Dedicated scheduler component [4]
>> > > > > > > -
>> > > > > > >
>> > > > > > > Fine grained failover for batch [5]
>> > > > > > > -
>> > > > > > >
>> > > > > > > Selectable input stream operator [6]
>> > > > > > > -
>> > > > > > >
>> > > > > > > Pluggable Shuffle Service [7]
>> > > > > > > -
>> > > > > > >
>> > > > > > > FLIP-30: Unified Catalog API & Hive metastore
>> integration
>> > [8]
>> > > > > > > - Heavily anticipated / discussed features in the
>> community
>> > > > > > > -
>> > > > > > >
>> > > > > > > FLIP-27: Source interface rework [9]
>> > > > > > > -
>> > > > > > >
>> > > > > > > Savepoint connector [10]
>> > > > > > > -
>> > > > > > >
>> > > > > > > FLIP-34: Terminate / Suspend job with savepoint [11]
>> > > > > > > -
>> > > > > > >
>> > > > > > > FLIP-36: Interactive Programming [12]
>> > > > > > > - Ecosystem
>> > > > > > > -
>> > > > > > >
>> > > > > > > Web UI rework [13]
>> > > > > > > -
>> > > > > > >
>> > > > > > > Active K8s integration [14]
>> > > > > > > -
>> > > > > > >
>> > > > > > > Google PubSub connector [15]
>> > > > > > > -
>> > > > > > >
>> > > > > > > First-class Protobuf support [16]
>> > > > > > > -
>> > > > > > >
>> > > > > > > FLIP-38: Python support in Table API [17]
>> > > > > > > -
>> > > > > > >
>> > > > > > > FLIP-39: Flink ML pipeline and libraries on top of Table
>> > API
>> > > > [18]
>> > > > > > >
>> > > > > > > *Suggested release timeline*
>> > > > > > >
>> > > > > > > Apache Flink 1.8.0 was released earlier this month, so based
>> on
>> > our
>> > > > > usual
>> > > > > > > timely release schedule, we should aim for releasing 1.9.0
>> around
>> > > mid
>> > > > > to
>> > > > > > > end July.
>> > > > > > >
>> > > > > > > Since it seems that this is going to be a fairly large
>> release,
>> > to
>> > > > give
>> > > > > > the
>> > > > > > > community enough testing time, I propose that the feature
>> freeze
>> > to
>> > > > be
>> > > > > > near
>> > > > > > > the end of June (8-9 weeks from now, probable June 28). This
>> is
>> > of
>> > > > > > course a
>> > > > > > > ballpark estimation for now; we should follow-up with a
>> separate
>> > > > thread
>> > > > > > > later in the release cycle to prepare contributors with an
>> > official
>> > > > > > feature
>> > > > > > > freeze date.
>> > > > > > >
>> > > > > > > I’d also like to use this opportunity to propose myself and
>> Kurt
>> > as
>> > > > the
>> > > > > > > release managers for 1.9.
>> > > > > > > AFAIK, we did not used to have 2 RMs for a single release in
>> the
>> > > > past,
>> > > > > > but
>> > > > > > > 1.9.0 is definitely quite ambitious so it would not hurt to
>> have
>> > > one
>> > > > > more
>> > > > > > > on board :) Cheers, Gordon [1]
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
>> > > > > > >
>> > > > > > > [2] https://issues.apache.org/jira/browse/FLINK-11439
>> > > > > > >
>> > > > > > > [3] https://issues.apache.org/jira/browse/FLINK-12251
>> > > > > > >
>> > > > > > > [4] https://issues.apache.org/jira/browse/FLINK-10429
>> > > > > > >
>> > > > > > > [5]
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
>> > > > > > >
>> > > > > > > [6]
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
>> > > > > > >
>> > > > > > > [7] https://issues.apache.org/jira/browse/FLINK-10653
>> > > > > > >
>> > > > > > > [8] https://issues.apache.org/jira/browse/FLINK-11275
>> > > > > > >
>> > > > > > > [9]
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
>> > > > > > >
>> > > > > > > [10] https://issues.apache.org/jira/browse/FLINK-12047
>> > > > > > >
>> > > > > > > [11]
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
>> > > > > > >
>> > > > > > > [12]
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
>> > > > > > >
>> > > > > > > [13] https://issues.apache.org/jira/browse/FLINK-10705
>> > > > > > >
>> > > > > > > [14] https://issues.apache.org/jira/browse/FLINK-9953
>> > > > > > >
>> > > > > > > [15] https://issues.apache.org/jira/browse/FLINK-9311
>> > > > > > >
>> > > > > > > [16] https://issues.apache.org/jira/browse/FLINK-11333
>> > > > > > >
>> > > > > > > [17]
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
>> > > > > > > [18]
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
@Hequn @Jincheng
Thanks for bringing up FLIP-29 to attention.
As previously mentioned, the original list is not a fixed feature set, so
if FLIP-29 has ongoing efforts and can make it before the feature freeze,
then of course it should be included!
@himansh1306
Concerning the ORC format for StreamingFileSink, is there already a JIRA
ticket tracking that? If not, I suggest to first open one and see if there
are similar interests from committers in adding that.
On Sun, May 5, 2019 at 11:19 PM Hequn Cheng <ch...@gmail.com> wrote:
> Hi,
>
> Great job, Gordon! Thanks a lot for driving this and wrapping features up
> to a detailed list. +1 on it!
>
> Would be great if we can also add flip29 to the list. @jincheng sun
> <su...@gmail.com> and I are focusing on it these days. I think
> these features in flip29 would bring big enhancements to the Table API. :-)
>
> Best, Hequn
>
> On Sun, May 5, 2019 at 10:41 PM Becket Qin <be...@gmail.com> wrote:
>
> > Thanks for driving this release, Gordon. +1 on the feature list.
> >
> > This is a pretty exciting and ambitious release!
> >
> > Cheers,
> >
> > Jiangjie (Becket) Qin
> >
> > On Sun, May 5, 2019 at 4:28 PM jincheng sun <su...@gmail.com>
> > wrote:
> >
> > > Thanks a lot for being our release manager, Great job!
> > >
> > > +1 for the feature list and It's better to add FLIP-29
> > > <
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739
> > > >(Support
> > > map/flatMap/aggregate/flatAggregate on TableAPI), as the goals of
> release
> > > 1.9.
> > >
> > > What do you think?
> > >
> > > Best,
> > > Jincheng
> > >
> > > Bowen Li <bo...@gmail.com> 于2019年5月5日周日 上午12:47写道：
> > >
> > > > +1, exciting and ambitious goals, the rough timeline looks
> reasonable.
> > > > Let's make it happen!
> > > >
> > > > On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com> wrote:
> > > >
> > > > > +1 for the 1.9.0 feature list. Excited to see it is happening.
> > > > >
> > > > > Regards,
> > > > > Jark
> > > > >
> > > > > On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <
> > > > himansh1306@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > +1 for Protobuf, Hive Metastore integration & features related to
> > > > > > savepoint.
> > > > > >
> > > > > > I was hoping if we could support for ORC File format can be added
> > in
> > > > > > StreamingFileSink Writer, Currently only Parquet is supported
> with
> > > > > respect
> > > > > > to columnar file formats
> > > > > >
> > > > > >
> > > > > > On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <
> tzulitai@apache.org
> > >
> > > > > > wrote:
> > > > > > > Hi community,
> > > > > > >
> > > > > > > Apache Flink 1.8.0 has been released a few weeks ago, so
> > naturally,
> > > > > it’s
> > > > > > > time to start thinking about what we want to aim for 1.9.0.
> > > > > > >
> > > > > > > Kurt and I had collected some features that would be reasonable
> > to
> > > > > > consider
> > > > > > > including for the next release, based on talking with various
> > > people
> > > > as
> > > > > > > well as observations from mailing list discussions and
> questions.
> > > > > > >
> > > > > > > Note that having specific features listed here does not mean
> that
> > > no
> > > > > > other
> > > > > > > pull requests or topics will be reviewed. I am sure that there
> > are
> > > > > other
> > > > > > > ongoing efforts that we missed here and will likely make it as
> an
> > > > > > > improvement or new feature in the next release. This discussion
> > is
> > > > > merely
> > > > > > > for bootstrapping a discussion for 1.9, as well as to give
> > > > contributors
> > > > > > an
> > > > > > > idea of what the community is looking to focus on in the next
> > > couple
> > > > of
> > > > > > > weeks.
> > > > > > >
> > > > > > > *Proposed features and focus*
> > > > > > >
> > > > > > > In the previous major release, Apache Flink 1.8.0, the
> community
> > > had
> > > > > > > prepared for some major Table & SQL additions from the Blink
> > > branch.
> > > > > With
> > > > > > > this in mind, for the next release, it would be great to wind
> up
> > > > those
> > > > > > > efforts by merging in the Blink-based Table / SQL planner and
> > > runtime
> > > > > for
> > > > > > > 1.9.
> > > > > > >
> > > > > > > Following Stephan’s previous thread [1] in the mailing list
> about
> > > > > > features
> > > > > > > in Blink, we should also start focusing on preparing for
> Blink’s
> > > > other
> > > > > > > several enhancements for batch execution. This includes
> resource
> > > > > > > optimization, fine-grained failover, pluggable shuffle service,
> > > > > adapting
> > > > > > > stream operators for batch execution, as well as better
> > integration
> > > > > with
> > > > > > > commonly used systems by batch executions such as Apache Hive.
> > > > > > >
> > > > > > > Moreover, besides efforts related to the Blink merge, we would
> > also
> > > > > like
> > > > > > us
> > > > > > > to work towards pushing forward some of the most discussed and
> > > > > > anticipated
> > > > > > > features by the community. Most of these had discussions in the
> > > > mailing
> > > > > > > lists that span multiple releases, and are also frequently
> > brought
> > > up
> > > > > in
> > > > > > > community events such as Flink Forward. This includes features
> > such
> > > > as
> > > > > > > source event-time alignment and the source interface rework, a
> > > > > savepoint
> > > > > > > connector that allows users to manipulate and query state in
> > > > > savepoints,
> > > > > > > interactive programming, as well as terminating a job with a
> > final
> > > > > > > savepoint.
> > > > > > >
> > > > > > > Last but not least, we have several existing contributions or
> > > > > discussions
> > > > > > > for the ecosystem surrounding Flink, which we think is also
> very
> > > > > valuable
> > > > > > > to try to merge in for 1.9. This includes a web UI rework
> > (recently
> > > > > > already
> > > > > > > merged), active K8s integration, Google PubSub connector,
> native
> > > > > support
> > > > > > > for the Protobuf format, Python support in the Table API, as
> well
> > > as
> > > > > > > reworking Flink’s support for machine learning.
> > > > > > >
> > > > > > > To wrap this up as a list of items, some of which already have
> > > JIRAs
> > > > or
> > > > > > > mailing list threads to track them:
> > > > > > >
> > > > > > > - Merge Blink runner for Table & SQL [2]
> > > > > > > -
> > > > > > >
> > > > > > > Restructure flink-table to separate API from core runtime
> > > > > > > -
> > > > > > >
> > > > > > > Make table planners pluggable
> > > > > > > -
> > > > > > >
> > > > > > > Rework Table / SQL type system to integrate better with
> the
> > > SQL
> > > > > > > standard [3]
> > > > > > > -
> > > > > > >
> > > > > > > Merge Blink planner and runtime for Table / SQL
> > > > > > > - Further preparations for more batch execution
> > optimization
> > > > from
> > > > > > > Blink
> > > > > > > -
> > > > > > >
> > > > > > > Dedicated scheduler component [4]
> > > > > > > -
> > > > > > >
> > > > > > > Fine grained failover for batch [5]
> > > > > > > -
> > > > > > >
> > > > > > > Selectable input stream operator [6]
> > > > > > > -
> > > > > > >
> > > > > > > Pluggable Shuffle Service [7]
> > > > > > > -
> > > > > > >
> > > > > > > FLIP-30: Unified Catalog API & Hive metastore integration
> > [8]
> > > > > > > - Heavily anticipated / discussed features in the
> community
> > > > > > > -
> > > > > > >
> > > > > > > FLIP-27: Source interface rework [9]
> > > > > > > -
> > > > > > >
> > > > > > > Savepoint connector [10]
> > > > > > > -
> > > > > > >
> > > > > > > FLIP-34: Terminate / Suspend job with savepoint [11]
> > > > > > > -
> > > > > > >
> > > > > > > FLIP-36: Interactive Programming [12]
> > > > > > > - Ecosystem
> > > > > > > -
> > > > > > >
> > > > > > > Web UI rework [13]
> > > > > > > -
> > > > > > >
> > > > > > > Active K8s integration [14]
> > > > > > > -
> > > > > > >
> > > > > > > Google PubSub connector [15]
> > > > > > > -
> > > > > > >
> > > > > > > First-class Protobuf support [16]
> > > > > > > -
> > > > > > >
> > > > > > > FLIP-38: Python support in Table API [17]
> > > > > > > -
> > > > > > >
> > > > > > > FLIP-39: Flink ML pipeline and libraries on top of Table
> > API
> > > > [18]
> > > > > > >
> > > > > > > *Suggested release timeline*
> > > > > > >
> > > > > > > Apache Flink 1.8.0 was released earlier this month, so based on
> > our
> > > > > usual
> > > > > > > timely release schedule, we should aim for releasing 1.9.0
> around
> > > mid
> > > > > to
> > > > > > > end July.
> > > > > > >
> > > > > > > Since it seems that this is going to be a fairly large release,
> > to
> > > > give
> > > > > > the
> > > > > > > community enough testing time, I propose that the feature
> freeze
> > to
> > > > be
> > > > > > near
> > > > > > > the end of June (8-9 weeks from now, probable June 28). This is
> > of
> > > > > > course a
> > > > > > > ballpark estimation for now; we should follow-up with a
> separate
> > > > thread
> > > > > > > later in the release cycle to prepare contributors with an
> > official
> > > > > > feature
> > > > > > > freeze date.
> > > > > > >
> > > > > > > I’d also like to use this opportunity to propose myself and
> Kurt
> > as
> > > > the
> > > > > > > release managers for 1.9.
> > > > > > > AFAIK, we did not used to have 2 RMs for a single release in
> the
> > > > past,
> > > > > > but
> > > > > > > 1.9.0 is definitely quite ambitious so it would not hurt to
> have
> > > one
> > > > > more
> > > > > > > on board :) Cheers, Gordon [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
> > > > > > >
> > > > > > > [2] https://issues.apache.org/jira/browse/FLINK-11439
> > > > > > >
> > > > > > > [3] https://issues.apache.org/jira/browse/FLINK-12251
> > > > > > >
> > > > > > > [4] https://issues.apache.org/jira/browse/FLINK-10429
> > > > > > >
> > > > > > > [5]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
> > > > > > >
> > > > > > > [6]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
> > > > > > >
> > > > > > > [7] https://issues.apache.org/jira/browse/FLINK-10653
> > > > > > >
> > > > > > > [8] https://issues.apache.org/jira/browse/FLINK-11275
> > > > > > >
> > > > > > > [9]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
> > > > > > >
> > > > > > > [10] https://issues.apache.org/jira/browse/FLINK-12047
> > > > > > >
> > > > > > > [11]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
> > > > > > >
> > > > > > > [12]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> > > > > > >
> > > > > > > [13] https://issues.apache.org/jira/browse/FLINK-10705
> > > > > > >
> > > > > > > [14] https://issues.apache.org/jira/browse/FLINK-9953
> > > > > > >
> > > > > > > [15] https://issues.apache.org/jira/browse/FLINK-9311
> > > > > > >
> > > > > > > [16] https://issues.apache.org/jira/browse/FLINK-11333
> > > > > > >
> > > > > > > [17]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> > > > > > > [18]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by Hequn Cheng <ch...@gmail.com>.
Hi,
Great job, Gordon! Thanks a lot for driving this and wrapping features up
to a detailed list. +1 on it!
Would be great if we can also add flip29 to the list. @jincheng sun
<su...@gmail.com> and I are focusing on it these days. I think
these features in flip29 would bring big enhancements to the Table API. :-)
Best, Hequn
On Sun, May 5, 2019 at 10:41 PM Becket Qin <be...@gmail.com> wrote:
> Thanks for driving this release, Gordon. +1 on the feature list.
>
> This is a pretty exciting and ambitious release!
>
> Cheers,
>
> Jiangjie (Becket) Qin
>
> On Sun, May 5, 2019 at 4:28 PM jincheng sun <su...@gmail.com>
> wrote:
>
> > Thanks a lot for being our release manager, Great job!
> >
> > +1 for the feature list and It's better to add FLIP-29
> > <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739
> > >(Support
> > map/flatMap/aggregate/flatAggregate on TableAPI), as the goals of release
> > 1.9.
> >
> > What do you think?
> >
> > Best,
> > Jincheng
> >
> > Bowen Li <bo...@gmail.com> 于2019年5月5日周日 上午12:47写道：
> >
> > > +1, exciting and ambitious goals, the rough timeline looks reasonable.
> > > Let's make it happen!
> > >
> > > On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com> wrote:
> > >
> > > > +1 for the 1.9.0 feature list. Excited to see it is happening.
> > > >
> > > > Regards,
> > > > Jark
> > > >
> > > > On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <
> > > himansh1306@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 for Protobuf, Hive Metastore integration & features related to
> > > > > savepoint.
> > > > >
> > > > > I was hoping if we could support for ORC File format can be added
> in
> > > > > StreamingFileSink Writer, Currently only Parquet is supported with
> > > > respect
> > > > > to columnar file formats
> > > > >
> > > > >
> > > > > On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <tzulitai@apache.org
> >
> > > > > wrote:
> > > > > > Hi community,
> > > > > >
> > > > > > Apache Flink 1.8.0 has been released a few weeks ago, so
> naturally,
> > > > it’s
> > > > > > time to start thinking about what we want to aim for 1.9.0.
> > > > > >
> > > > > > Kurt and I had collected some features that would be reasonable
> to
> > > > > consider
> > > > > > including for the next release, based on talking with various
> > people
> > > as
> > > > > > well as observations from mailing list discussions and questions.
> > > > > >
> > > > > > Note that having specific features listed here does not mean that
> > no
> > > > > other
> > > > > > pull requests or topics will be reviewed. I am sure that there
> are
> > > > other
> > > > > > ongoing efforts that we missed here and will likely make it as an
> > > > > > improvement or new feature in the next release. This discussion
> is
> > > > merely
> > > > > > for bootstrapping a discussion for 1.9, as well as to give
> > > contributors
> > > > > an
> > > > > > idea of what the community is looking to focus on in the next
> > couple
> > > of
> > > > > > weeks.
> > > > > >
> > > > > > *Proposed features and focus*
> > > > > >
> > > > > > In the previous major release, Apache Flink 1.8.0, the community
> > had
> > > > > > prepared for some major Table & SQL additions from the Blink
> > branch.
> > > > With
> > > > > > this in mind, for the next release, it would be great to wind up
> > > those
> > > > > > efforts by merging in the Blink-based Table / SQL planner and
> > runtime
> > > > for
> > > > > > 1.9.
> > > > > >
> > > > > > Following Stephan’s previous thread [1] in the mailing list about
> > > > > features
> > > > > > in Blink, we should also start focusing on preparing for Blink’s
> > > other
> > > > > > several enhancements for batch execution. This includes resource
> > > > > > optimization, fine-grained failover, pluggable shuffle service,
> > > > adapting
> > > > > > stream operators for batch execution, as well as better
> integration
> > > > with
> > > > > > commonly used systems by batch executions such as Apache Hive.
> > > > > >
> > > > > > Moreover, besides efforts related to the Blink merge, we would
> also
> > > > like
> > > > > us
> > > > > > to work towards pushing forward some of the most discussed and
> > > > > anticipated
> > > > > > features by the community. Most of these had discussions in the
> > > mailing
> > > > > > lists that span multiple releases, and are also frequently
> brought
> > up
> > > > in
> > > > > > community events such as Flink Forward. This includes features
> such
> > > as
> > > > > > source event-time alignment and the source interface rework, a
> > > > savepoint
> > > > > > connector that allows users to manipulate and query state in
> > > > savepoints,
> > > > > > interactive programming, as well as terminating a job with a
> final
> > > > > > savepoint.
> > > > > >
> > > > > > Last but not least, we have several existing contributions or
> > > > discussions
> > > > > > for the ecosystem surrounding Flink, which we think is also very
> > > > valuable
> > > > > > to try to merge in for 1.9. This includes a web UI rework
> (recently
> > > > > already
> > > > > > merged), active K8s integration, Google PubSub connector, native
> > > > support
> > > > > > for the Protobuf format, Python support in the Table API, as well
> > as
> > > > > > reworking Flink’s support for machine learning.
> > > > > >
> > > > > > To wrap this up as a list of items, some of which already have
> > JIRAs
> > > or
> > > > > > mailing list threads to track them:
> > > > > >
> > > > > > - Merge Blink runner for Table & SQL [2]
> > > > > > -
> > > > > >
> > > > > > Restructure flink-table to separate API from core runtime
> > > > > > -
> > > > > >
> > > > > > Make table planners pluggable
> > > > > > -
> > > > > >
> > > > > > Rework Table / SQL type system to integrate better with the
> > SQL
> > > > > > standard [3]
> > > > > > -
> > > > > >
> > > > > > Merge Blink planner and runtime for Table / SQL
> > > > > > - Further preparations for more batch execution
> optimization
> > > from
> > > > > > Blink
> > > > > > -
> > > > > >
> > > > > > Dedicated scheduler component [4]
> > > > > > -
> > > > > >
> > > > > > Fine grained failover for batch [5]
> > > > > > -
> > > > > >
> > > > > > Selectable input stream operator [6]
> > > > > > -
> > > > > >
> > > > > > Pluggable Shuffle Service [7]
> > > > > > -
> > > > > >
> > > > > > FLIP-30: Unified Catalog API & Hive metastore integration
> [8]
> > > > > > - Heavily anticipated / discussed features in the community
> > > > > > -
> > > > > >
> > > > > > FLIP-27: Source interface rework [9]
> > > > > > -
> > > > > >
> > > > > > Savepoint connector [10]
> > > > > > -
> > > > > >
> > > > > > FLIP-34: Terminate / Suspend job with savepoint [11]
> > > > > > -
> > > > > >
> > > > > > FLIP-36: Interactive Programming [12]
> > > > > > - Ecosystem
> > > > > > -
> > > > > >
> > > > > > Web UI rework [13]
> > > > > > -
> > > > > >
> > > > > > Active K8s integration [14]
> > > > > > -
> > > > > >
> > > > > > Google PubSub connector [15]
> > > > > > -
> > > > > >
> > > > > > First-class Protobuf support [16]
> > > > > > -
> > > > > >
> > > > > > FLIP-38: Python support in Table API [17]
> > > > > > -
> > > > > >
> > > > > > FLIP-39: Flink ML pipeline and libraries on top of Table
> API
> > > [18]
> > > > > >
> > > > > > *Suggested release timeline*
> > > > > >
> > > > > > Apache Flink 1.8.0 was released earlier this month, so based on
> our
> > > > usual
> > > > > > timely release schedule, we should aim for releasing 1.9.0 around
> > mid
> > > > to
> > > > > > end July.
> > > > > >
> > > > > > Since it seems that this is going to be a fairly large release,
> to
> > > give
> > > > > the
> > > > > > community enough testing time, I propose that the feature freeze
> to
> > > be
> > > > > near
> > > > > > the end of June (8-9 weeks from now, probable June 28). This is
> of
> > > > > course a
> > > > > > ballpark estimation for now; we should follow-up with a separate
> > > thread
> > > > > > later in the release cycle to prepare contributors with an
> official
> > > > > feature
> > > > > > freeze date.
> > > > > >
> > > > > > I’d also like to use this opportunity to propose myself and Kurt
> as
> > > the
> > > > > > release managers for 1.9.
> > > > > > AFAIK, we did not used to have 2 RMs for a single release in the
> > > past,
> > > > > but
> > > > > > 1.9.0 is definitely quite ambitious so it would not hurt to have
> > one
> > > > more
> > > > > > on board :) Cheers, Gordon [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
> > > > > >
> > > > > > [2] https://issues.apache.org/jira/browse/FLINK-11439
> > > > > >
> > > > > > [3] https://issues.apache.org/jira/browse/FLINK-12251
> > > > > >
> > > > > > [4] https://issues.apache.org/jira/browse/FLINK-10429
> > > > > >
> > > > > > [5]
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
> > > > > >
> > > > > > [6]
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
> > > > > >
> > > > > > [7] https://issues.apache.org/jira/browse/FLINK-10653
> > > > > >
> > > > > > [8] https://issues.apache.org/jira/browse/FLINK-11275
> > > > > >
> > > > > > [9]
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
> > > > > >
> > > > > > [10] https://issues.apache.org/jira/browse/FLINK-12047
> > > > > >
> > > > > > [11]
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
> > > > > >
> > > > > > [12]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> > > > > >
> > > > > > [13] https://issues.apache.org/jira/browse/FLINK-10705
> > > > > >
> > > > > > [14] https://issues.apache.org/jira/browse/FLINK-9953
> > > > > >
> > > > > > [15] https://issues.apache.org/jira/browse/FLINK-9311
> > > > > >
> > > > > > [16] https://issues.apache.org/jira/browse/FLINK-11333
> > > > > >
> > > > > > [17]
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> > > > > > [18]
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by Becket Qin <be...@gmail.com>.
Thanks for driving this release, Gordon. +1 on the feature list.
This is a pretty exciting and ambitious release!
Cheers,
Jiangjie (Becket) Qin
On Sun, May 5, 2019 at 4:28 PM jincheng sun <su...@gmail.com>
wrote:
> Thanks a lot for being our release manager, Great job!
>
> +1 for the feature list and It's better to add FLIP-29
> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739
> >(Support
> map/flatMap/aggregate/flatAggregate on TableAPI), as the goals of release
> 1.9.
>
> What do you think?
>
> Best,
> Jincheng
>
> Bowen Li <bo...@gmail.com> 于2019年5月5日周日 上午12:47写道：
>
> > +1, exciting and ambitious goals, the rough timeline looks reasonable.
> > Let's make it happen!
> >
> > On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com> wrote:
> >
> > > +1 for the 1.9.0 feature list. Excited to see it is happening.
> > >
> > > Regards,
> > > Jark
> > >
> > > On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <
> > himansh1306@gmail.com>
> > > wrote:
> > >
> > > > +1 for Protobuf, Hive Metastore integration & features related to
> > > > savepoint.
> > > >
> > > > I was hoping if we could support for ORC File format can be added in
> > > > StreamingFileSink Writer, Currently only Parquet is supported with
> > > respect
> > > > to columnar file formats
> > > >
> > > >
> > > > On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <tz...@apache.org>
> > > > wrote:
> > > > > Hi community,
> > > > >
> > > > > Apache Flink 1.8.0 has been released a few weeks ago, so naturally,
> > > it’s
> > > > > time to start thinking about what we want to aim for 1.9.0.
> > > > >
> > > > > Kurt and I had collected some features that would be reasonable to
> > > > consider
> > > > > including for the next release, based on talking with various
> people
> > as
> > > > > well as observations from mailing list discussions and questions.
> > > > >
> > > > > Note that having specific features listed here does not mean that
> no
> > > > other
> > > > > pull requests or topics will be reviewed. I am sure that there are
> > > other
> > > > > ongoing efforts that we missed here and will likely make it as an
> > > > > improvement or new feature in the next release. This discussion is
> > > merely
> > > > > for bootstrapping a discussion for 1.9, as well as to give
> > contributors
> > > > an
> > > > > idea of what the community is looking to focus on in the next
> couple
> > of
> > > > > weeks.
> > > > >
> > > > > *Proposed features and focus*
> > > > >
> > > > > In the previous major release, Apache Flink 1.8.0, the community
> had
> > > > > prepared for some major Table & SQL additions from the Blink
> branch.
> > > With
> > > > > this in mind, for the next release, it would be great to wind up
> > those
> > > > > efforts by merging in the Blink-based Table / SQL planner and
> runtime
> > > for
> > > > > 1.9.
> > > > >
> > > > > Following Stephan’s previous thread [1] in the mailing list about
> > > > features
> > > > > in Blink, we should also start focusing on preparing for Blink’s
> > other
> > > > > several enhancements for batch execution. This includes resource
> > > > > optimization, fine-grained failover, pluggable shuffle service,
> > > adapting
> > > > > stream operators for batch execution, as well as better integration
> > > with
> > > > > commonly used systems by batch executions such as Apache Hive.
> > > > >
> > > > > Moreover, besides efforts related to the Blink merge, we would also
> > > like
> > > > us
> > > > > to work towards pushing forward some of the most discussed and
> > > > anticipated
> > > > > features by the community. Most of these had discussions in the
> > mailing
> > > > > lists that span multiple releases, and are also frequently brought
> up
> > > in
> > > > > community events such as Flink Forward. This includes features such
> > as
> > > > > source event-time alignment and the source interface rework, a
> > > savepoint
> > > > > connector that allows users to manipulate and query state in
> > > savepoints,
> > > > > interactive programming, as well as terminating a job with a final
> > > > > savepoint.
> > > > >
> > > > > Last but not least, we have several existing contributions or
> > > discussions
> > > > > for the ecosystem surrounding Flink, which we think is also very
> > > valuable
> > > > > to try to merge in for 1.9. This includes a web UI rework (recently
> > > > already
> > > > > merged), active K8s integration, Google PubSub connector, native
> > > support
> > > > > for the Protobuf format, Python support in the Table API, as well
> as
> > > > > reworking Flink’s support for machine learning.
> > > > >
> > > > > To wrap this up as a list of items, some of which already have
> JIRAs
> > or
> > > > > mailing list threads to track them:
> > > > >
> > > > > - Merge Blink runner for Table & SQL [2]
> > > > > -
> > > > >
> > > > > Restructure flink-table to separate API from core runtime
> > > > > -
> > > > >
> > > > > Make table planners pluggable
> > > > > -
> > > > >
> > > > > Rework Table / SQL type system to integrate better with the
> SQL
> > > > > standard [3]
> > > > > -
> > > > >
> > > > > Merge Blink planner and runtime for Table / SQL
> > > > > - Further preparations for more batch execution optimization
> > from
> > > > > Blink
> > > > > -
> > > > >
> > > > > Dedicated scheduler component [4]
> > > > > -
> > > > >
> > > > > Fine grained failover for batch [5]
> > > > > -
> > > > >
> > > > > Selectable input stream operator [6]
> > > > > -
> > > > >
> > > > > Pluggable Shuffle Service [7]
> > > > > -
> > > > >
> > > > > FLIP-30: Unified Catalog API & Hive metastore integration [8]
> > > > > - Heavily anticipated / discussed features in the community
> > > > > -
> > > > >
> > > > > FLIP-27: Source interface rework [9]
> > > > > -
> > > > >
> > > > > Savepoint connector [10]
> > > > > -
> > > > >
> > > > > FLIP-34: Terminate / Suspend job with savepoint [11]
> > > > > -
> > > > >
> > > > > FLIP-36: Interactive Programming [12]
> > > > > - Ecosystem
> > > > > -
> > > > >
> > > > > Web UI rework [13]
> > > > > -
> > > > >
> > > > > Active K8s integration [14]
> > > > > -
> > > > >
> > > > > Google PubSub connector [15]
> > > > > -
> > > > >
> > > > > First-class Protobuf support [16]
> > > > > -
> > > > >
> > > > > FLIP-38: Python support in Table API [17]
> > > > > -
> > > > >
> > > > > FLIP-39: Flink ML pipeline and libraries on top of Table API
> > [18]
> > > > >
> > > > > *Suggested release timeline*
> > > > >
> > > > > Apache Flink 1.8.0 was released earlier this month, so based on our
> > > usual
> > > > > timely release schedule, we should aim for releasing 1.9.0 around
> mid
> > > to
> > > > > end July.
> > > > >
> > > > > Since it seems that this is going to be a fairly large release, to
> > give
> > > > the
> > > > > community enough testing time, I propose that the feature freeze to
> > be
> > > > near
> > > > > the end of June (8-9 weeks from now, probable June 28). This is of
> > > > course a
> > > > > ballpark estimation for now; we should follow-up with a separate
> > thread
> > > > > later in the release cycle to prepare contributors with an official
> > > > feature
> > > > > freeze date.
> > > > >
> > > > > I’d also like to use this opportunity to propose myself and Kurt as
> > the
> > > > > release managers for 1.9.
> > > > > AFAIK, we did not used to have 2 RMs for a single release in the
> > past,
> > > > but
> > > > > 1.9.0 is definitely quite ambitious so it would not hurt to have
> one
> > > more
> > > > > on board :) Cheers, Gordon [1]
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
> > > > >
> > > > > [2] https://issues.apache.org/jira/browse/FLINK-11439
> > > > >
> > > > > [3] https://issues.apache.org/jira/browse/FLINK-12251
> > > > >
> > > > > [4] https://issues.apache.org/jira/browse/FLINK-10429
> > > > >
> > > > > [5]
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
> > > > >
> > > > > [6]
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
> > > > >
> > > > > [7] https://issues.apache.org/jira/browse/FLINK-10653
> > > > >
> > > > > [8] https://issues.apache.org/jira/browse/FLINK-11275
> > > > >
> > > > > [9]
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
> > > > >
> > > > > [10] https://issues.apache.org/jira/browse/FLINK-12047
> > > > >
> > > > > [11]
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
> > > > >
> > > > > [12]
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> > > > >
> > > > > [13] https://issues.apache.org/jira/browse/FLINK-10705
> > > > >
> > > > > [14] https://issues.apache.org/jira/browse/FLINK-9953
> > > > >
> > > > > [15] https://issues.apache.org/jira/browse/FLINK-9311
> > > > >
> > > > > [16] https://issues.apache.org/jira/browse/FLINK-11333
> > > > >
> > > > > [17]
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> > > > > [18]
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by jincheng sun <su...@gmail.com>.
Thanks a lot for being our release manager, Great job!
+1 for the feature list and It's better to add FLIP-29
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739>(Support
map/flatMap/aggregate/flatAggregate on TableAPI), as the goals of release
1.9.
What do you think?
Best,
Jincheng
Bowen Li <bo...@gmail.com> 于2019年5月5日周日 上午12:47写道：
> +1, exciting and ambitious goals, the rough timeline looks reasonable.
> Let's make it happen!
>
> On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com> wrote:
>
> > +1 for the 1.9.0 feature list. Excited to see it is happening.
> >
> > Regards,
> > Jark
> >
> > On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <
> himansh1306@gmail.com>
> > wrote:
> >
> > > +1 for Protobuf, Hive Metastore integration & features related to
> > > savepoint.
> > >
> > > I was hoping if we could support for ORC File format can be added in
> > > StreamingFileSink Writer, Currently only Parquet is supported with
> > respect
> > > to columnar file formats
> > >
> > >
> > > On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <tz...@apache.org>
> > > wrote:
> > > > Hi community,
> > > >
> > > > Apache Flink 1.8.0 has been released a few weeks ago, so naturally,
> > it’s
> > > > time to start thinking about what we want to aim for 1.9.0.
> > > >
> > > > Kurt and I had collected some features that would be reasonable to
> > > consider
> > > > including for the next release, based on talking with various people
> as
> > > > well as observations from mailing list discussions and questions.
> > > >
> > > > Note that having specific features listed here does not mean that no
> > > other
> > > > pull requests or topics will be reviewed. I am sure that there are
> > other
> > > > ongoing efforts that we missed here and will likely make it as an
> > > > improvement or new feature in the next release. This discussion is
> > merely
> > > > for bootstrapping a discussion for 1.9, as well as to give
> contributors
> > > an
> > > > idea of what the community is looking to focus on in the next couple
> of
> > > > weeks.
> > > >
> > > > *Proposed features and focus*
> > > >
> > > > In the previous major release, Apache Flink 1.8.0, the community had
> > > > prepared for some major Table & SQL additions from the Blink branch.
> > With
> > > > this in mind, for the next release, it would be great to wind up
> those
> > > > efforts by merging in the Blink-based Table / SQL planner and runtime
> > for
> > > > 1.9.
> > > >
> > > > Following Stephan’s previous thread [1] in the mailing list about
> > > features
> > > > in Blink, we should also start focusing on preparing for Blink’s
> other
> > > > several enhancements for batch execution. This includes resource
> > > > optimization, fine-grained failover, pluggable shuffle service,
> > adapting
> > > > stream operators for batch execution, as well as better integration
> > with
> > > > commonly used systems by batch executions such as Apache Hive.
> > > >
> > > > Moreover, besides efforts related to the Blink merge, we would also
> > like
> > > us
> > > > to work towards pushing forward some of the most discussed and
> > > anticipated
> > > > features by the community. Most of these had discussions in the
> mailing
> > > > lists that span multiple releases, and are also frequently brought up
> > in
> > > > community events such as Flink Forward. This includes features such
> as
> > > > source event-time alignment and the source interface rework, a
> > savepoint
> > > > connector that allows users to manipulate and query state in
> > savepoints,
> > > > interactive programming, as well as terminating a job with a final
> > > > savepoint.
> > > >
> > > > Last but not least, we have several existing contributions or
> > discussions
> > > > for the ecosystem surrounding Flink, which we think is also very
> > valuable
> > > > to try to merge in for 1.9. This includes a web UI rework (recently
> > > already
> > > > merged), active K8s integration, Google PubSub connector, native
> > support
> > > > for the Protobuf format, Python support in the Table API, as well as
> > > > reworking Flink’s support for machine learning.
> > > >
> > > > To wrap this up as a list of items, some of which already have JIRAs
> or
> > > > mailing list threads to track them:
> > > >
> > > > - Merge Blink runner for Table & SQL [2]
> > > > -
> > > >
> > > > Restructure flink-table to separate API from core runtime
> > > > -
> > > >
> > > > Make table planners pluggable
> > > > -
> > > >
> > > > Rework Table / SQL type system to integrate better with the SQL
> > > > standard [3]
> > > > -
> > > >
> > > > Merge Blink planner and runtime for Table / SQL
> > > > - Further preparations for more batch execution optimization
> from
> > > > Blink
> > > > -
> > > >
> > > > Dedicated scheduler component [4]
> > > > -
> > > >
> > > > Fine grained failover for batch [5]
> > > > -
> > > >
> > > > Selectable input stream operator [6]
> > > > -
> > > >
> > > > Pluggable Shuffle Service [7]
> > > > -
> > > >
> > > > FLIP-30: Unified Catalog API & Hive metastore integration [8]
> > > > - Heavily anticipated / discussed features in the community
> > > > -
> > > >
> > > > FLIP-27: Source interface rework [9]
> > > > -
> > > >
> > > > Savepoint connector [10]
> > > > -
> > > >
> > > > FLIP-34: Terminate / Suspend job with savepoint [11]
> > > > -
> > > >
> > > > FLIP-36: Interactive Programming [12]
> > > > - Ecosystem
> > > > -
> > > >
> > > > Web UI rework [13]
> > > > -
> > > >
> > > > Active K8s integration [14]
> > > > -
> > > >
> > > > Google PubSub connector [15]
> > > > -
> > > >
> > > > First-class Protobuf support [16]
> > > > -
> > > >
> > > > FLIP-38: Python support in Table API [17]
> > > > -
> > > >
> > > > FLIP-39: Flink ML pipeline and libraries on top of Table API
> [18]
> > > >
> > > > *Suggested release timeline*
> > > >
> > > > Apache Flink 1.8.0 was released earlier this month, so based on our
> > usual
> > > > timely release schedule, we should aim for releasing 1.9.0 around mid
> > to
> > > > end July.
> > > >
> > > > Since it seems that this is going to be a fairly large release, to
> give
> > > the
> > > > community enough testing time, I propose that the feature freeze to
> be
> > > near
> > > > the end of June (8-9 weeks from now, probable June 28). This is of
> > > course a
> > > > ballpark estimation for now; we should follow-up with a separate
> thread
> > > > later in the release cycle to prepare contributors with an official
> > > feature
> > > > freeze date.
> > > >
> > > > I’d also like to use this opportunity to propose myself and Kurt as
> the
> > > > release managers for 1.9.
> > > > AFAIK, we did not used to have 2 RMs for a single release in the
> past,
> > > but
> > > > 1.9.0 is definitely quite ambitious so it would not hurt to have one
> > more
> > > > on board :) Cheers, Gordon [1]
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
> > > >
> > > > [2] https://issues.apache.org/jira/browse/FLINK-11439
> > > >
> > > > [3] https://issues.apache.org/jira/browse/FLINK-12251
> > > >
> > > > [4] https://issues.apache.org/jira/browse/FLINK-10429
> > > >
> > > > [5]
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
> > > >
> > > > [6]
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
> > > >
> > > > [7] https://issues.apache.org/jira/browse/FLINK-10653
> > > >
> > > > [8] https://issues.apache.org/jira/browse/FLINK-11275
> > > >
> > > > [9]
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
> > > >
> > > > [10] https://issues.apache.org/jira/browse/FLINK-12047
> > > >
> > > > [11]
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
> > > >
> > > > [12]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> > > >
> > > > [13] https://issues.apache.org/jira/browse/FLINK-10705
> > > >
> > > > [14] https://issues.apache.org/jira/browse/FLINK-9953
> > > >
> > > > [15] https://issues.apache.org/jira/browse/FLINK-9311
> > > >
> > > > [16] https://issues.apache.org/jira/browse/FLINK-11333
> > > >
> > > > [17]
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> > > > [18]
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
> > > >
> > >
> >
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by Bowen Li <bo...@gmail.com>.
+1, exciting and ambitious goals, the rough timeline looks reasonable.
Let's make it happen!
On Sat, May 4, 2019 at 2:47 AM Jark Wu <im...@gmail.com> wrote:
> +1 for the 1.9.0 feature list. Excited to see it is happening.
>
> Regards,
> Jark
>
> On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <hi...@gmail.com>
> wrote:
>
> > +1 for Protobuf, Hive Metastore integration & features related to
> > savepoint.
> >
> > I was hoping if we could support for ORC File format can be added in
> > StreamingFileSink Writer, Currently only Parquet is supported with
> respect
> > to columnar file formats
> >
> >
> > On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <tz...@apache.org>
> > wrote:
> > > Hi community,
> > >
> > > Apache Flink 1.8.0 has been released a few weeks ago, so naturally,
> it’s
> > > time to start thinking about what we want to aim for 1.9.0.
> > >
> > > Kurt and I had collected some features that would be reasonable to
> > consider
> > > including for the next release, based on talking with various people as
> > > well as observations from mailing list discussions and questions.
> > >
> > > Note that having specific features listed here does not mean that no
> > other
> > > pull requests or topics will be reviewed. I am sure that there are
> other
> > > ongoing efforts that we missed here and will likely make it as an
> > > improvement or new feature in the next release. This discussion is
> merely
> > > for bootstrapping a discussion for 1.9, as well as to give contributors
> > an
> > > idea of what the community is looking to focus on in the next couple of
> > > weeks.
> > >
> > > *Proposed features and focus*
> > >
> > > In the previous major release, Apache Flink 1.8.0, the community had
> > > prepared for some major Table & SQL additions from the Blink branch.
> With
> > > this in mind, for the next release, it would be great to wind up those
> > > efforts by merging in the Blink-based Table / SQL planner and runtime
> for
> > > 1.9.
> > >
> > > Following Stephan’s previous thread [1] in the mailing list about
> > features
> > > in Blink, we should also start focusing on preparing for Blink’s other
> > > several enhancements for batch execution. This includes resource
> > > optimization, fine-grained failover, pluggable shuffle service,
> adapting
> > > stream operators for batch execution, as well as better integration
> with
> > > commonly used systems by batch executions such as Apache Hive.
> > >
> > > Moreover, besides efforts related to the Blink merge, we would also
> like
> > us
> > > to work towards pushing forward some of the most discussed and
> > anticipated
> > > features by the community. Most of these had discussions in the mailing
> > > lists that span multiple releases, and are also frequently brought up
> in
> > > community events such as Flink Forward. This includes features such as
> > > source event-time alignment and the source interface rework, a
> savepoint
> > > connector that allows users to manipulate and query state in
> savepoints,
> > > interactive programming, as well as terminating a job with a final
> > > savepoint.
> > >
> > > Last but not least, we have several existing contributions or
> discussions
> > > for the ecosystem surrounding Flink, which we think is also very
> valuable
> > > to try to merge in for 1.9. This includes a web UI rework (recently
> > already
> > > merged), active K8s integration, Google PubSub connector, native
> support
> > > for the Protobuf format, Python support in the Table API, as well as
> > > reworking Flink’s support for machine learning.
> > >
> > > To wrap this up as a list of items, some of which already have JIRAs or
> > > mailing list threads to track them:
> > >
> > > - Merge Blink runner for Table & SQL [2]
> > > -
> > >
> > > Restructure flink-table to separate API from core runtime
> > > -
> > >
> > > Make table planners pluggable
> > > -
> > >
> > > Rework Table / SQL type system to integrate better with the SQL
> > > standard [3]
> > > -
> > >
> > > Merge Blink planner and runtime for Table / SQL
> > > - Further preparations for more batch execution optimization from
> > > Blink
> > > -
> > >
> > > Dedicated scheduler component [4]
> > > -
> > >
> > > Fine grained failover for batch [5]
> > > -
> > >
> > > Selectable input stream operator [6]
> > > -
> > >
> > > Pluggable Shuffle Service [7]
> > > -
> > >
> > > FLIP-30: Unified Catalog API & Hive metastore integration [8]
> > > - Heavily anticipated / discussed features in the community
> > > -
> > >
> > > FLIP-27: Source interface rework [9]
> > > -
> > >
> > > Savepoint connector [10]
> > > -
> > >
> > > FLIP-34: Terminate / Suspend job with savepoint [11]
> > > -
> > >
> > > FLIP-36: Interactive Programming [12]
> > > - Ecosystem
> > > -
> > >
> > > Web UI rework [13]
> > > -
> > >
> > > Active K8s integration [14]
> > > -
> > >
> > > Google PubSub connector [15]
> > > -
> > >
> > > First-class Protobuf support [16]
> > > -
> > >
> > > FLIP-38: Python support in Table API [17]
> > > -
> > >
> > > FLIP-39: Flink ML pipeline and libraries on top of Table API [18]
> > >
> > > *Suggested release timeline*
> > >
> > > Apache Flink 1.8.0 was released earlier this month, so based on our
> usual
> > > timely release schedule, we should aim for releasing 1.9.0 around mid
> to
> > > end July.
> > >
> > > Since it seems that this is going to be a fairly large release, to give
> > the
> > > community enough testing time, I propose that the feature freeze to be
> > near
> > > the end of June (8-9 weeks from now, probable June 28). This is of
> > course a
> > > ballpark estimation for now; we should follow-up with a separate thread
> > > later in the release cycle to prepare contributors with an official
> > feature
> > > freeze date.
> > >
> > > I’d also like to use this opportunity to propose myself and Kurt as the
> > > release managers for 1.9.
> > > AFAIK, we did not used to have 2 RMs for a single release in the past,
> > but
> > > 1.9.0 is definitely quite ambitious so it would not hurt to have one
> more
> > > on board :) Cheers, Gordon [1]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
> > >
> > > [2] https://issues.apache.org/jira/browse/FLINK-11439
> > >
> > > [3] https://issues.apache.org/jira/browse/FLINK-12251
> > >
> > > [4] https://issues.apache.org/jira/browse/FLINK-10429
> > >
> > > [5]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
> > >
> > > [6]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
> > >
> > > [7] https://issues.apache.org/jira/browse/FLINK-10653
> > >
> > > [8] https://issues.apache.org/jira/browse/FLINK-11275
> > >
> > > [9]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
> > >
> > > [10] https://issues.apache.org/jira/browse/FLINK-12047
> > >
> > > [11]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
> > >
> > > [12]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> > >
> > > [13] https://issues.apache.org/jira/browse/FLINK-10705
> > >
> > > [14] https://issues.apache.org/jira/browse/FLINK-9953
> > >
> > > [15] https://issues.apache.org/jira/browse/FLINK-9311
> > >
> > > [16] https://issues.apache.org/jira/browse/FLINK-11333
> > >
> > > [17]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> > > [18]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
> > >
> >
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by Jark Wu <im...@gmail.com>.
+1 for the 1.9.0 feature list. Excited to see it is happening.
Regards,
Jark
On Thu, 2 May 2019 at 17:07, himansh1306@gmail.com <hi...@gmail.com>
wrote:
> +1 for Protobuf, Hive Metastore integration & features related to
> savepoint.
>
> I was hoping if we could support for ORC File format can be added in
> StreamingFileSink Writer, Currently only Parquet is supported with respect
> to columnar file formats
>
>
> On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <tz...@apache.org>
> wrote:
> > Hi community,
> >
> > Apache Flink 1.8.0 has been released a few weeks ago, so naturally, it’s
> > time to start thinking about what we want to aim for 1.9.0.
> >
> > Kurt and I had collected some features that would be reasonable to
> consider
> > including for the next release, based on talking with various people as
> > well as observations from mailing list discussions and questions.
> >
> > Note that having specific features listed here does not mean that no
> other
> > pull requests or topics will be reviewed. I am sure that there are other
> > ongoing efforts that we missed here and will likely make it as an
> > improvement or new feature in the next release. This discussion is merely
> > for bootstrapping a discussion for 1.9, as well as to give contributors
> an
> > idea of what the community is looking to focus on in the next couple of
> > weeks.
> >
> > *Proposed features and focus*
> >
> > In the previous major release, Apache Flink 1.8.0, the community had
> > prepared for some major Table & SQL additions from the Blink branch. With
> > this in mind, for the next release, it would be great to wind up those
> > efforts by merging in the Blink-based Table / SQL planner and runtime for
> > 1.9.
> >
> > Following Stephan’s previous thread [1] in the mailing list about
> features
> > in Blink, we should also start focusing on preparing for Blink’s other
> > several enhancements for batch execution. This includes resource
> > optimization, fine-grained failover, pluggable shuffle service, adapting
> > stream operators for batch execution, as well as better integration with
> > commonly used systems by batch executions such as Apache Hive.
> >
> > Moreover, besides efforts related to the Blink merge, we would also like
> us
> > to work towards pushing forward some of the most discussed and
> anticipated
> > features by the community. Most of these had discussions in the mailing
> > lists that span multiple releases, and are also frequently brought up in
> > community events such as Flink Forward. This includes features such as
> > source event-time alignment and the source interface rework, a savepoint
> > connector that allows users to manipulate and query state in savepoints,
> > interactive programming, as well as terminating a job with a final
> > savepoint.
> >
> > Last but not least, we have several existing contributions or discussions
> > for the ecosystem surrounding Flink, which we think is also very valuable
> > to try to merge in for 1.9. This includes a web UI rework (recently
> already
> > merged), active K8s integration, Google PubSub connector, native support
> > for the Protobuf format, Python support in the Table API, as well as
> > reworking Flink’s support for machine learning.
> >
> > To wrap this up as a list of items, some of which already have JIRAs or
> > mailing list threads to track them:
> >
> > - Merge Blink runner for Table & SQL [2]
> > -
> >
> > Restructure flink-table to separate API from core runtime
> > -
> >
> > Make table planners pluggable
> > -
> >
> > Rework Table / SQL type system to integrate better with the SQL
> > standard [3]
> > -
> >
> > Merge Blink planner and runtime for Table / SQL
> > - Further preparations for more batch execution optimization from
> > Blink
> > -
> >
> > Dedicated scheduler component [4]
> > -
> >
> > Fine grained failover for batch [5]
> > -
> >
> > Selectable input stream operator [6]
> > -
> >
> > Pluggable Shuffle Service [7]
> > -
> >
> > FLIP-30: Unified Catalog API & Hive metastore integration [8]
> > - Heavily anticipated / discussed features in the community
> > -
> >
> > FLIP-27: Source interface rework [9]
> > -
> >
> > Savepoint connector [10]
> > -
> >
> > FLIP-34: Terminate / Suspend job with savepoint [11]
> > -
> >
> > FLIP-36: Interactive Programming [12]
> > - Ecosystem
> > -
> >
> > Web UI rework [13]
> > -
> >
> > Active K8s integration [14]
> > -
> >
> > Google PubSub connector [15]
> > -
> >
> > First-class Protobuf support [16]
> > -
> >
> > FLIP-38: Python support in Table API [17]
> > -
> >
> > FLIP-39: Flink ML pipeline and libraries on top of Table API [18]
> >
> > *Suggested release timeline*
> >
> > Apache Flink 1.8.0 was released earlier this month, so based on our usual
> > timely release schedule, we should aim for releasing 1.9.0 around mid to
> > end July.
> >
> > Since it seems that this is going to be a fairly large release, to give
> the
> > community enough testing time, I propose that the feature freeze to be
> near
> > the end of June (8-9 weeks from now, probable June 28). This is of
> course a
> > ballpark estimation for now; we should follow-up with a separate thread
> > later in the release cycle to prepare contributors with an official
> feature
> > freeze date.
> >
> > I’d also like to use this opportunity to propose myself and Kurt as the
> > release managers for 1.9.
> > AFAIK, we did not used to have 2 RMs for a single release in the past,
> but
> > 1.9.0 is definitely quite ambitious so it would not hurt to have one more
> > on board :) Cheers, Gordon [1]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
> >
> > [2] https://issues.apache.org/jira/browse/FLINK-11439
> >
> > [3] https://issues.apache.org/jira/browse/FLINK-12251
> >
> > [4] https://issues.apache.org/jira/browse/FLINK-10429
> >
> > [5]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
> >
> > [6]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
> >
> > [7] https://issues.apache.org/jira/browse/FLINK-10653
> >
> > [8] https://issues.apache.org/jira/browse/FLINK-11275
> >
> > [9]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
> >
> > [10] https://issues.apache.org/jira/browse/FLINK-12047
> >
> > [11]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
> >
> > [12]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> >
> > [13] https://issues.apache.org/jira/browse/FLINK-10705
> >
> > [14] https://issues.apache.org/jira/browse/FLINK-9953
> >
> > [15] https://issues.apache.org/jira/browse/FLINK-9311
> >
> > [16] https://issues.apache.org/jira/browse/FLINK-11333
> >
> > [17]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> > [18]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
> >
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by hi...@gmail.com,
hi...@gmail.com.
+1 for Protobuf, Hive Metastore integration & features related to savepoint.
I was hoping if we could support for ORC File format can be added in StreamingFileSink Writer, Currently only Parquet is supported with respect to columnar file formats
On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <tz...@apache.org> wrote:
> Hi community,
>
> Apache Flink 1.8.0 has been released a few weeks ago, so naturally, it’s
> time to start thinking about what we want to aim for 1.9.0.
>
> Kurt and I had collected some features that would be reasonable to consider
> including for the next release, based on talking with various people as
> well as observations from mailing list discussions and questions.
>
> Note that having specific features listed here does not mean that no other
> pull requests or topics will be reviewed. I am sure that there are other
> ongoing efforts that we missed here and will likely make it as an
> improvement or new feature in the next release. This discussion is merely
> for bootstrapping a discussion for 1.9, as well as to give contributors an
> idea of what the community is looking to focus on in the next couple of
> weeks.
>
> *Proposed features and focus*
>
> In the previous major release, Apache Flink 1.8.0, the community had
> prepared for some major Table & SQL additions from the Blink branch. With
> this in mind, for the next release, it would be great to wind up those
> efforts by merging in the Blink-based Table / SQL planner and runtime for
> 1.9.
>
> Following Stephan’s previous thread [1] in the mailing list about features
> in Blink, we should also start focusing on preparing for Blink’s other
> several enhancements for batch execution. This includes resource
> optimization, fine-grained failover, pluggable shuffle service, adapting
> stream operators for batch execution, as well as better integration with
> commonly used systems by batch executions such as Apache Hive.
>
> Moreover, besides efforts related to the Blink merge, we would also like us
> to work towards pushing forward some of the most discussed and anticipated
> features by the community. Most of these had discussions in the mailing
> lists that span multiple releases, and are also frequently brought up in
> community events such as Flink Forward. This includes features such as
> source event-time alignment and the source interface rework, a savepoint
> connector that allows users to manipulate and query state in savepoints,
> interactive programming, as well as terminating a job with a final
> savepoint.
>
> Last but not least, we have several existing contributions or discussions
> for the ecosystem surrounding Flink, which we think is also very valuable
> to try to merge in for 1.9. This includes a web UI rework (recently already
> merged), active K8s integration, Google PubSub connector, native support
> for the Protobuf format, Python support in the Table API, as well as
> reworking Flink’s support for machine learning.
>
> To wrap this up as a list of items, some of which already have JIRAs or
> mailing list threads to track them:
>
> - Merge Blink runner for Table & SQL [2]
> -
>
> Restructure flink-table to separate API from core runtime
> -
>
> Make table planners pluggable
> -
>
> Rework Table / SQL type system to integrate better with the SQL
> standard [3]
> -
>
> Merge Blink planner and runtime for Table / SQL
> - Further preparations for more batch execution optimization from
> Blink
> -
>
> Dedicated scheduler component [4]
> -
>
> Fine grained failover for batch [5]
> -
>
> Selectable input stream operator [6]
> -
>
> Pluggable Shuffle Service [7]
> -
>
> FLIP-30: Unified Catalog API & Hive metastore integration [8]
> - Heavily anticipated / discussed features in the community
> -
>
> FLIP-27: Source interface rework [9]
> -
>
> Savepoint connector [10]
> -
>
> FLIP-34: Terminate / Suspend job with savepoint [11]
> -
>
> FLIP-36: Interactive Programming [12]
> - Ecosystem
> -
>
> Web UI rework [13]
> -
>
> Active K8s integration [14]
> -
>
> Google PubSub connector [15]
> -
>
> First-class Protobuf support [16]
> -
>
> FLIP-38: Python support in Table API [17]
> -
>
> FLIP-39: Flink ML pipeline and libraries on top of Table API [18]
>
> *Suggested release timeline*
>
> Apache Flink 1.8.0 was released earlier this month, so based on our usual
> timely release schedule, we should aim for releasing 1.9.0 around mid to
> end July.
>
> Since it seems that this is going to be a fairly large release, to give the
> community enough testing time, I propose that the feature freeze to be near
> the end of June (8-9 weeks from now, probable June 28). This is of course a
> ballpark estimation for now; we should follow-up with a separate thread
> later in the release cycle to prepare contributors with an official feature
> freeze date.
>
> I’d also like to use this opportunity to propose myself and Kurt as the
> release managers for 1.9.
> AFAIK, we did not used to have 2 RMs for a single release in the past, but
> 1.9.0 is definitely quite ambitious so it would not hurt to have one more
> on board :) Cheers, Gordon [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
>
> [2] https://issues.apache.org/jira/browse/FLINK-11439
>
> [3] https://issues.apache.org/jira/browse/FLINK-12251
>
> [4] https://issues.apache.org/jira/browse/FLINK-10429
>
> [5]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
>
> [6]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
>
> [7] https://issues.apache.org/jira/browse/FLINK-10653
>
> [8] https://issues.apache.org/jira/browse/FLINK-11275
>
> [9]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
>
> [10] https://issues.apache.org/jira/browse/FLINK-12047
>
> [11]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
>
> [12]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
>
> [13] https://issues.apache.org/jira/browse/FLINK-10705
>
> [14] https://issues.apache.org/jira/browse/FLINK-9953
>
> [15] https://issues.apache.org/jira/browse/FLINK-9311
>
> [16] https://issues.apache.org/jira/browse/FLINK-11333
>
> [17]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> [18]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
>
Re: [DISCUSS] Features for Apache Flink 1.9.0
Posted by hi...@gmail.com,
hi...@gmail.com.
+1 for Protobuf support, Hive Metastore Integration & features related to savepoint.
I was hoping if we could support for ORC File format can be added in StreamingFileSink Writer, Currently only Parquet is supported with respect to columnar file formats
On 2019/05/01 05:15:23, "Tzu-Li (Gordon) Tai" <tz...@apache.org> wrote:
> Hi community,
>
> Apache Flink 1.8.0 has been released a few weeks ago, so naturally, it’s
> time to start thinking about what we want to aim for 1.9.0.
>
> Kurt and I had collected some features that would be reasonable to consider
> including for the next release, based on talking with various people as
> well as observations from mailing list discussions and questions.
>
> Note that having specific features listed here does not mean that no other
> pull requests or topics will be reviewed. I am sure that there are other
> ongoing efforts that we missed here and will likely make it as an
> improvement or new feature in the next release. This discussion is merely
> for bootstrapping a discussion for 1.9, as well as to give contributors an
> idea of what the community is looking to focus on in the next couple of
> weeks.
>
> *Proposed features and focus*
>
> In the previous major release, Apache Flink 1.8.0, the community had
> prepared for some major Table & SQL additions from the Blink branch. With
> this in mind, for the next release, it would be great to wind up those
> efforts by merging in the Blink-based Table / SQL planner and runtime for
> 1.9.
>
> Following Stephan’s previous thread [1] in the mailing list about features
> in Blink, we should also start focusing on preparing for Blink’s other
> several enhancements for batch execution. This includes resource
> optimization, fine-grained failover, pluggable shuffle service, adapting
> stream operators for batch execution, as well as better integration with
> commonly used systems by batch executions such as Apache Hive.
>
> Moreover, besides efforts related to the Blink merge, we would also like us
> to work towards pushing forward some of the most discussed and anticipated
> features by the community. Most of these had discussions in the mailing
> lists that span multiple releases, and are also frequently brought up in
> community events such as Flink Forward. This includes features such as
> source event-time alignment and the source interface rework, a savepoint
> connector that allows users to manipulate and query state in savepoints,
> interactive programming, as well as terminating a job with a final
> savepoint.
>
> Last but not least, we have several existing contributions or discussions
> for the ecosystem surrounding Flink, which we think is also very valuable
> to try to merge in for 1.9. This includes a web UI rework (recently already
> merged), active K8s integration, Google PubSub connector, native support
> for the Protobuf format, Python support in the Table API, as well as
> reworking Flink’s support for machine learning.
>
> To wrap this up as a list of items, some of which already have JIRAs or
> mailing list threads to track them:
>
> - Merge Blink runner for Table & SQL [2]
> -
>
> Restructure flink-table to separate API from core runtime
> -
>
> Make table planners pluggable
> -
>
> Rework Table / SQL type system to integrate better with the SQL
> standard [3]
> -
>
> Merge Blink planner and runtime for Table / SQL
> - Further preparations for more batch execution optimization from
> Blink
> -
>
> Dedicated scheduler component [4]
> -
>
> Fine grained failover for batch [5]
> -
>
> Selectable input stream operator [6]
> -
>
> Pluggable Shuffle Service [7]
> -
>
> FLIP-30: Unified Catalog API & Hive metastore integration [8]
> - Heavily anticipated / discussed features in the community
> -
>
> FLIP-27: Source interface rework [9]
> -
>
> Savepoint connector [10]
> -
>
> FLIP-34: Terminate / Suspend job with savepoint [11]
> -
>
> FLIP-36: Interactive Programming [12]
> - Ecosystem
> -
>
> Web UI rework [13]
> -
>
> Active K8s integration [14]
> -
>
> Google PubSub connector [15]
> -
>
> First-class Protobuf support [16]
> -
>
> FLIP-38: Python support in Table API [17]
> -
>
> FLIP-39: Flink ML pipeline and libraries on top of Table API [18]
>
> *Suggested release timeline*
>
> Apache Flink 1.8.0 was released earlier this month, so based on our usual
> timely release schedule, we should aim for releasing 1.9.0 around mid to
> end July.
>
> Since it seems that this is going to be a fairly large release, to give the
> community enough testing time, I propose that the feature freeze to be near
> the end of June (8-9 weeks from now, probable June 28). This is of course a
> ballpark estimation for now; we should follow-up with a separate thread
> later in the release cycle to prepare contributors with an official feature
> freeze date.
>
> I’d also like to use this opportunity to propose myself and Kurt as the
> release managers for 1.9.
> AFAIK, we did not used to have 2 RMs for a single release in the past, but
> 1.9.0 is definitely quite ambitious so it would not hurt to have one more
> on board :) Cheers, Gordon [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html
>
> [2] https://issues.apache.org/jira/browse/FLINK-11439
>
> [3] https://issues.apache.org/jira/browse/FLINK-12251
>
> [4] https://issues.apache.org/jira/browse/FLINK-10429
>
> [5]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backtracking-for-failover-regions-td28293.html
>
> [6]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html
>
> [7] https://issues.apache.org/jira/browse/FLINK-10653
>
> [8] https://issues.apache.org/jira/browse/FLINK-11275
>
> [9]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952i20.html
>
> [10] https://issues.apache.org/jira/browse/FLINK-12047
>
> [11]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Terminate-Suspend-Job-with-Savepoint-td26927.html
>
> [12]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
>
> [13] https://issues.apache.org/jira/browse/FLINK-10705
>
> [14] https://issues.apache.org/jira/browse/FLINK-9953
>
> [15] https://issues.apache.org/jira/browse/FLINK-9311
>
> [16] https://issues.apache.org/jira/browse/FLINK-11333
>
> [17]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> [18]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html
>