You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Zhijiang <wa...@aliyun.com.INVALID> on 2020/03/11 03:16:38 UTC

[DISCUSS] Features of Apache Flink 1.11

Hi community,


Not more than one month ago we have released Flink 1.10. We are now heading for the Flink 1.11 release and we, as release managers, would like to share with you what are the features that the community is currently working on and we are hoping that will be part of the Flink 1.11 release. Currently we are aiming with the feature freeze to happen in late April.

As for now, some of the features are in the very early stages of the development or even brainstorming. Because of that, some of them do not have associated JIRA tickets or FLIP documents. For the next progress announcement we are hoping that this will be no longer the case.

Please also note that because we are still relatively at the beginning of the release cycle, some of the FLIPs haven’t yet been voted. 

- SQL / Table
- FLIP-42: Restructure documentation [1]
- FLIP-65: New type inference for Table API UDFs [2]
- FLIP-84: Improve TableEnv’s interface [3]
- FLIP-91 Introduce SQL client gateway and provide JDBC driver [4]
- FLIP-93: Introduce JDBC catalog and Postgres catalog [5]
- FLIP-105: Support to interpret and emit changelog in Flink SQL [6]
- FLIP-107: Reading table columns from different parts of source records [7]
- [FLINK-14807] Add Table#collect API for fetching data [8]
- Support query and table hints
- ML / Connectors
- FLIP-27: New source API [9]
- [FLINK-15670] Wrap a source/sink pair to persist intermediate result for subgraph failure recovery [10]
- Pulsar source / sink / catalog
- Update ML Pipeline API interface to better support Flink ML lib algorithms
- PyFlink
- FLIP-58: Debugging and monitoring of Python UDF [11]
- FLIP-106: Expand the usage scope of Python UDF [12]
- Integration with most popular Python libraries (Pandas)
- Performance improvements of Python UDF
- Support running python UDF in docker workers
- Add Python ML API
- Fully support all kinds of Python UDF
- Web UI
- FLIP-98: Better back pressure detection [13]
- FLIP-99: Make max exception configurable [14]
- FLIP-100: Add attempt information [15]
- FLIP-102: Add more metrics to TaskManager [16]
- FLIP-103: Better TM/JM log display [17]
- [FLINK-14816] Add thread dump feature for TaskManager [18]
- Runtime
- FLIP-56: Support for dynamic slots on the TaskExecutor [19]
- FLIP-67: Support for cluster partitions [20]
- FLIP-76: Unaligned checkpoints [21]
- FLIP-83: Flink e2e performance testing framework [22]
- FLIP-85: Support cluster deploy mode [23]
- FLIP-92: Add N-Ary input stream operator in Flink [24]
- FLIP-108: Add GPU to the resource management (specifically for UDTF & UDF) [25]
- FLIP-111: Consolidate docker images [26]
- Unified memory configuration for JobManager 
- Specify upper bound for number of allocated TaskManagers 
- [FLINK-9407] ORC format for StreamingFileSink [27]
- [FLINK-10742] Let Netty use Flink's buffers on downstream side [28]
- [FLINK-10934] Support per-job mode for Kubernetes integration [29]
- [FLINK-11395] Avro writer for StreamingFileSink [30]
- [FLINK-11427] Protobuf parquet writer for StreamingFileSink [31]
- [FLINK-11499] Extend StreamingFileSink BulkFormats to support arbitrary roll policies [32]
- [FLINK-14106] Make SlotManager pluggable [33]
- [FLINK-15672] Switch to Log4j2 by default [34]
- [FLINK-15674] Consolidate Java and Scala type extraction stack [35]
- [FLINK-15679] Improve Flink’s ID system [36]
- [FLINK-15786] Use the separated classloader to load connectors’ jar [37]
- [FLINK-15788] Various Kubernetes improvements [38]
- [FLINK-15911][FLINK-15154] Support Flink work over NAT [39]
- [FLINK-16408] Bind user code class loader to lifetime of a slot [40]
- [FLINK-16428] Network memory management for backpressure [41]
- [FLINK-16430] Pipelined region scheduling [42]
- Calculate required shuffle memory before allocating slots
- State Backend:
- [FLINK-5763] Make savepoint self-contained / relocatable [43]
- [FLINK-8871] Complete checkpoint cancellation messages [44]
- [FLINK-12692] Support disk spilling in HeapKeyedStateBackend [45]
- [FLINK-15012] Cleanup of leftover files in HDFS/OSS/S3 [46]
- Enable local recovery by default
- Support getCustomizedState in KeyedStateStore [47]
- [FLINK-15532] Enable strict capacity limit for memory usage for RocksDB [48]
Your release managers,
Zhijiang & Piotr Nowojski 

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Documentation
[2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-65%3A+New+type+inference+for+Table+API+UDFs
[3] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878
[4] https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
[5] https://cwiki.apache.org/confluence/display/FLINK/FLIP-93%3A+JDBC+catalog+and+Postgres+catalog
[6] https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL
[7] https://cwiki.apache.org/confluence/display/FLINK/FLIP-107%3A+Reading+table+columns+from+different+parts+of+source+records
[8] https://issues.apache.org/jira/browse/FLINK-14807
[9] https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
[10] https://issues.apache.org/jira/browse/FLINK-15670
[11] https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
[12] https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
[13] https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
[14] https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable
[15] https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
[16] https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager
[17] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
[18] https://issues.apache.org/jira/browse/FLINK-14816
[19] https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation
[20] https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
[21] https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
[22] https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework
[23] https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode
[24] https://cwiki.apache.org/confluence/display/FLINK/FLIP-92%3A+Add+N-Ary+Stream+Operator+in+Flink
[25] https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink
[26] https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification
[27] https://issues.apache.org/jira/browse/FLINK-9407
[28] https://issues.apache.org/jira/browse/FLINK-10742
[29] https://issues.apache.org/jira/browse/FLINK-10934
[30] https://issues.apache.org/jira/browse/FLINK-11395
[31] https://issues.apache.org/jira/browse/FLINK-11427
[32] https://issues.apache.org/jira/browse/FLINK-11499
[33] https://issues.apache.org/jira/browse/FLINK-14106
[34] https://issues.apache.org/jira/browse/FLINK-15672
[35] https://issues.apache.org/jira/browse/FLINK-15674
[36] https://issues.apache.org/jira/browse/FLINK-15679
[37] https://issues.apache.org/jira/browse/FLINK-15786
[38] https://issues.apache.org/jira/browse/FLINK-15788
[39] https://issues.apache.org/jira/browse/FLINK-15911
[39] https://issues.apache.org/jira/browse/FLINK-15154
[40] https://issues.apache.org/jira/browse/FLINK-16408
[41] https://issues.apache.org/jira/browse/FLINK-16428
[42] https://issues.apache.org/jira/browse/FLINK-16430
[43] https://issues.apache.org/jira/browse/FLINK-5763
[44] https://issues.apache.org/jira/browse/FLINK-8871
[45] https://issues.apache.org/jira/browse/FLINK-12692
[46] https://issues.apache.org/jira/browse/FLINK-15012
[47] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-customize-state-in-customized-KeyedStateBackend-td32771.html
[48] https://issues.apache.org/jira/browse/FLINK-15532

Re: [DISCUSS] Features of Apache Flink 1.11

Posted by Zhijiang <wa...@aliyun.com.INVALID>.

Thanks for all the feedbacks @Hequn @Timo @Godfrey !

We will note your updates and adjust the feature list accordingly in the next progress announcement.

Best,
Zhijiang


------------------------------------------------------------------
From:godfrey he <go...@gmail.com>
Send Time:2020 Mar. 11 (Wed.) 17:20
To:dev <de...@flink.apache.org>
Subject:Re: [DISCUSS] Features of Apache Flink 1.11

Hi Zhijiang and Piotr,

I think we can remove "FLIP-91 Introduce SQL client gateway and provide
JDBC driver" from the list, because we have decided the first step to
support sql gateway and jdbc driver as ecosystem in ververica, we are not
going to put more effort on it now.

Thanks for updating the list!

Bests,
Godfrey


Timo Walther <tw...@apache.org> 于2020年3月11日周三 下午4:13写道：

> Hi Zhijiang and Piotr,
>
> from the SQL side we also plan to rework the source and sink interfaces
> in 1.11. The FLIP is not yet published but already reserved and
> requirement for FLIP-105:
>
> FLIP-95: New TableSource and TableSink interfaces
>
> Thanks for compiling the list!
>
> Regards,
> Timo
>
>
> On 11.03.20 09:05, Hequn Cheng wrote:
> > Thanks Zhijiang and Piotr for kicking off the discussion and providing
> the
> > detailed list.
> > This would be very helpful for tracking the features.
> >
> > BTW, as for PyFlink, it would be great if the feature list can also
> include
> > the following features:
> > - FLIP-112: Support User-Defined Metrics in Python UDF
> > - FLIP-114: Support Python UDF in SQL Client
> >
> > Looking forward to the release!
> >
> > Best,
> > Hequn
> >
> >
> >
> > On Wed, Mar 11, 2020 at 1:02 PM Yu Li <ca...@gmail.com> wrote:
> >
> >> Thanks for compiling the list of 1.11 efforts Zhijiang and Piotr! This
> >> helps a lot to better understand what the community is currently working
> >> on. Looking forward to another successful release.
> >>
> >> Best Regards,
> >> Yu
> >>
> >>
> >> On Wed, 11 Mar 2020 at 11:17, Zhijiang <wangzhijiang999@aliyun.com
> >> .invalid>
> >> wrote:
> >>
> >>> Hi community,
> >>>
> >>>
> >>> Not more than one month ago we have released Flink 1.10. We are now
> >>> heading for the Flink 1.11 release and we, as release managers, would
> >> like
> >>> to share with you what are the features that the community is currently
> >>> working on and we are hoping that will be part of the Flink 1.11
> release.
> >>> Currently we are aiming with the feature freeze to happen in late
> April.
> >>>
> >>> As for now, some of the features are in the very early stages of the
> >>> development or even brainstorming. Because of that, some of them do not
> >>> have associated JIRA tickets or FLIP documents. For the next progress
> >>> announcement we are hoping that this will be no longer the case.
> >>>
> >>> Please also note that because we are still relatively at the beginning
> of
> >>> the release cycle, some of the FLIPs haven’t yet been voted.
> >>>
> >>> - SQL / Table
> >>> - FLIP-42: Restructure documentation [1]
> >>> - FLIP-65: New type inference for Table API UDFs [2]
> >>> - FLIP-84: Improve TableEnv’s interface [3]
> >>> - FLIP-91 Introduce SQL client gateway and provide JDBC driver [4]
> >>> - FLIP-93: Introduce JDBC catalog and Postgres catalog [5]
> >>> - FLIP-105: Support to interpret and emit changelog in Flink SQL [6]
> >>> - FLIP-107: Reading table columns from different parts of source
> records
> >>> [7]
> >>> - [FLINK-14807] Add Table#collect API for fetching data [8]
> >>> - Support query and table hints
> >>> - ML / Connectors
> >>> - FLIP-27: New source API [9]
> >>> - [FLINK-15670] Wrap a source/sink pair to persist intermediate result
> >> for
> >>> subgraph failure recovery [10]
> >>> - Pulsar source / sink / catalog
> >>> - Update ML Pipeline API interface to better support Flink ML lib
> >>> algorithms
> >>> - PyFlink
> >>> - FLIP-58: Debugging and monitoring of Python UDF [11]
> >>> - FLIP-106: Expand the usage scope of Python UDF [12]
> >>> - Integration with most popular Python libraries (Pandas)
> >>> - Performance improvements of Python UDF
> >>> - Support running python UDF in docker workers
> >>> - Add Python ML API
> >>> - Fully support all kinds of Python UDF
> >>> - Web UI
> >>> - FLIP-98: Better back pressure detection [13]
> >>> - FLIP-99: Make max exception configurable [14]
> >>> - FLIP-100: Add attempt information [15]
> >>> - FLIP-102: Add more metrics to TaskManager [16]
> >>> - FLIP-103: Better TM/JM log display [17]
> >>> - [FLINK-14816] Add thread dump feature for TaskManager [18]
> >>> - Runtime
> >>> - FLIP-56: Support for dynamic slots on the TaskExecutor [19]
> >>> - FLIP-67: Support for cluster partitions [20]
> >>> - FLIP-76: Unaligned checkpoints [21]
> >>> - FLIP-83: Flink e2e performance testing framework [22]
> >>> - FLIP-85: Support cluster deploy mode [23]
> >>> - FLIP-92: Add N-Ary input stream operator in Flink [24]
> >>> - FLIP-108: Add GPU to the resource management (specifically for UDTF &
> >>> UDF) [25]
> >>> - FLIP-111: Consolidate docker images [26]
> >>> - Unified memory configuration for JobManager
> >>> - Specify upper bound for number of allocated TaskManagers
> >>> - [FLINK-9407] ORC format for StreamingFileSink [27]
> >>> - [FLINK-10742] Let Netty use Flink's buffers on downstream side [28]
> >>> - [FLINK-10934] Support per-job mode for Kubernetes integration [29]
> >>> - [FLINK-11395] Avro writer for StreamingFileSink [30]
> >>> - [FLINK-11427] Protobuf parquet writer for StreamingFileSink [31]
> >>> - [FLINK-11499] Extend StreamingFileSink BulkFormats to support
> arbitrary
> >>> roll policies [32]
> >>> - [FLINK-14106] Make SlotManager pluggable [33]
> >>> - [FLINK-15672] Switch to Log4j2 by default [34]
> >>> - [FLINK-15674] Consolidate Java and Scala type extraction stack [35]
> >>> - [FLINK-15679] Improve Flink’s ID system [36]
> >>> - [FLINK-15786] Use the separated classloader to load connectors’ jar
> >> [37]
> >>> - [FLINK-15788] Various Kubernetes improvements [38]
> >>> - [FLINK-15911][FLINK-15154] Support Flink work over NAT [39]
> >>> - [FLINK-16408] Bind user code class loader to lifetime of a slot [40]
> >>> - [FLINK-16428] Network memory management for backpressure [41]
> >>> - [FLINK-16430] Pipelined region scheduling [42]
> >>> - Calculate required shuffle memory before allocating slots
> >>> - State Backend:
> >>> - [FLINK-5763] Make savepoint self-contained / relocatable [43]
> >>> - [FLINK-8871] Complete checkpoint cancellation messages [44]
> >>> - [FLINK-12692] Support disk spilling in HeapKeyedStateBackend [45]
> >>> - [FLINK-15012] Cleanup of leftover files in HDFS/OSS/S3 [46]
> >>> - Enable local recovery by default
> >>> - Support getCustomizedState in KeyedStateStore [47]
> >>> - [FLINK-15532] Enable strict capacity limit for memory usage for
> RocksDB
> >>> [48]
> >>> Your release managers,
> >>> Zhijiang & Piotr Nowojski
> >>>
> >>> [1]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Documentation
> >>> [2]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-65%3A+New+type+inference+for+Table+API+UDFs
> >>> [3]
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878
> >>> [4]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> >>> [5]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-93%3A+JDBC+catalog+and+Postgres+catalog
> >>> [6]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL
> >>> [7]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-107%3A+Reading+table+columns+from+different+parts+of+source+records
> >>> [8] https://issues.apache.org/jira/browse/FLINK-14807
> >>> [9]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> >>> [10] https://issues.apache.org/jira/browse/FLINK-15670
> >>> [11]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> >>> [12]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
> >>> [13]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
> >>> [14]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable
> >>> [15]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
> >>> [16]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager
> >>> [17]
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
> >>> [18] https://issues.apache.org/jira/browse/FLINK-14816
> >>> [19]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation
> >>> [20]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
> >>> [21]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
> >>> [22]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework
> >>> [23]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode
> >>> [24]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-92%3A+Add+N-Ary+Stream+Operator+in+Flink
> >>> [25]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink
> >>> [26]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification
> >>> [27] https://issues.apache.org/jira/browse/FLINK-9407
> >>> [28] https://issues.apache.org/jira/browse/FLINK-10742
> >>> [29] https://issues.apache.org/jira/browse/FLINK-10934
> >>> [30] https://issues.apache.org/jira/browse/FLINK-11395
> >>> [31] https://issues.apache.org/jira/browse/FLINK-11427
> >>> [32] https://issues.apache.org/jira/browse/FLINK-11499
> >>> [33] https://issues.apache.org/jira/browse/FLINK-14106
> >>> [34] https://issues.apache.org/jira/browse/FLINK-15672
> >>> [35] https://issues.apache.org/jira/browse/FLINK-15674
> >>> [36] https://issues.apache.org/jira/browse/FLINK-15679
> >>> [37] https://issues.apache.org/jira/browse/FLINK-15786
> >>> [38] https://issues.apache.org/jira/browse/FLINK-15788
> >>> [39] https://issues.apache.org/jira/browse/FLINK-15911
> >>> [39] https://issues.apache.org/jira/browse/FLINK-15154
> >>> [40] https://issues.apache.org/jira/browse/FLINK-16408
> >>> [41] https://issues.apache.org/jira/browse/FLINK-16428
> >>> [42] https://issues.apache.org/jira/browse/FLINK-16430
> >>> [43] https://issues.apache.org/jira/browse/FLINK-5763
> >>> [44] https://issues.apache.org/jira/browse/FLINK-8871
> >>> [45] https://issues.apache.org/jira/browse/FLINK-12692
> >>> [46] https://issues.apache.org/jira/browse/FLINK-15012
> >>> [47]
> >>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-customize-state-in-customized-KeyedStateBackend-td32771.html
> >>> [48] https://issues.apache.org/jira/browse/FLINK-15532
> >>
> >
>
>

Re: [DISCUSS] Features of Apache Flink 1.11

Posted by godfrey he <go...@gmail.com>.

Hi Zhijiang and Piotr,

I think we can remove "FLIP-91 Introduce SQL client gateway and provide
JDBC driver" from the list, because we have decided the first step to
support sql gateway and jdbc driver as ecosystem in ververica, we are not
going to put more effort on it now.

Thanks for updating the list!

Bests,
Godfrey


Timo Walther <tw...@apache.org> 于2020年3月11日周三 下午4:13写道：

> Hi Zhijiang and Piotr,
>
> from the SQL side we also plan to rework the source and sink interfaces
> in 1.11. The FLIP is not yet published but already reserved and
> requirement for FLIP-105:
>
> FLIP-95: New TableSource and TableSink interfaces
>
> Thanks for compiling the list!
>
> Regards,
> Timo
>
>
> On 11.03.20 09:05, Hequn Cheng wrote:
> > Thanks Zhijiang and Piotr for kicking off the discussion and providing
> the
> > detailed list.
> > This would be very helpful for tracking the features.
> >
> > BTW, as for PyFlink, it would be great if the feature list can also
> include
> > the following features:
> > - FLIP-112: Support User-Defined Metrics in Python UDF
> > - FLIP-114: Support Python UDF in SQL Client
> >
> > Looking forward to the release!
> >
> > Best,
> > Hequn
> >
> >
> >
> > On Wed, Mar 11, 2020 at 1:02 PM Yu Li <ca...@gmail.com> wrote:
> >
> >> Thanks for compiling the list of 1.11 efforts Zhijiang and Piotr! This
> >> helps a lot to better understand what the community is currently working
> >> on. Looking forward to another successful release.
> >>
> >> Best Regards,
> >> Yu
> >>
> >>
> >> On Wed, 11 Mar 2020 at 11:17, Zhijiang <wangzhijiang999@aliyun.com
> >> .invalid>
> >> wrote:
> >>
> >>> Hi community,
> >>>
> >>>
> >>> Not more than one month ago we have released Flink 1.10. We are now
> >>> heading for the Flink 1.11 release and we, as release managers, would
> >> like
> >>> to share with you what are the features that the community is currently
> >>> working on and we are hoping that will be part of the Flink 1.11
> release.
> >>> Currently we are aiming with the feature freeze to happen in late
> April.
> >>>
> >>> As for now, some of the features are in the very early stages of the
> >>> development or even brainstorming. Because of that, some of them do not
> >>> have associated JIRA tickets or FLIP documents. For the next progress
> >>> announcement we are hoping that this will be no longer the case.
> >>>
> >>> Please also note that because we are still relatively at the beginning
> of
> >>> the release cycle, some of the FLIPs haven’t yet been voted.
> >>>
> >>> - SQL / Table
> >>> - FLIP-42: Restructure documentation [1]
> >>> - FLIP-65: New type inference for Table API UDFs [2]
> >>> - FLIP-84: Improve TableEnv’s interface [3]
> >>> - FLIP-91 Introduce SQL client gateway and provide JDBC driver [4]
> >>> - FLIP-93: Introduce JDBC catalog and Postgres catalog [5]
> >>> - FLIP-105: Support to interpret and emit changelog in Flink SQL [6]
> >>> - FLIP-107: Reading table columns from different parts of source
> records
> >>> [7]
> >>> - [FLINK-14807] Add Table#collect API for fetching data [8]
> >>> - Support query and table hints
> >>> - ML / Connectors
> >>> - FLIP-27: New source API [9]
> >>> - [FLINK-15670] Wrap a source/sink pair to persist intermediate result
> >> for
> >>> subgraph failure recovery [10]
> >>> - Pulsar source / sink / catalog
> >>> - Update ML Pipeline API interface to better support Flink ML lib
> >>> algorithms
> >>> - PyFlink
> >>> - FLIP-58: Debugging and monitoring of Python UDF [11]
> >>> - FLIP-106: Expand the usage scope of Python UDF [12]
> >>> - Integration with most popular Python libraries (Pandas)
> >>> - Performance improvements of Python UDF
> >>> - Support running python UDF in docker workers
> >>> - Add Python ML API
> >>> - Fully support all kinds of Python UDF
> >>> - Web UI
> >>> - FLIP-98: Better back pressure detection [13]
> >>> - FLIP-99: Make max exception configurable [14]
> >>> - FLIP-100: Add attempt information [15]
> >>> - FLIP-102: Add more metrics to TaskManager [16]
> >>> - FLIP-103: Better TM/JM log display [17]
> >>> - [FLINK-14816] Add thread dump feature for TaskManager [18]
> >>> - Runtime
> >>> - FLIP-56: Support for dynamic slots on the TaskExecutor [19]
> >>> - FLIP-67: Support for cluster partitions [20]
> >>> - FLIP-76: Unaligned checkpoints [21]
> >>> - FLIP-83: Flink e2e performance testing framework [22]
> >>> - FLIP-85: Support cluster deploy mode [23]
> >>> - FLIP-92: Add N-Ary input stream operator in Flink [24]
> >>> - FLIP-108: Add GPU to the resource management (specifically for UDTF &
> >>> UDF) [25]
> >>> - FLIP-111: Consolidate docker images [26]
> >>> - Unified memory configuration for JobManager
> >>> - Specify upper bound for number of allocated TaskManagers
> >>> - [FLINK-9407] ORC format for StreamingFileSink [27]
> >>> - [FLINK-10742] Let Netty use Flink's buffers on downstream side [28]
> >>> - [FLINK-10934] Support per-job mode for Kubernetes integration [29]
> >>> - [FLINK-11395] Avro writer for StreamingFileSink [30]
> >>> - [FLINK-11427] Protobuf parquet writer for StreamingFileSink [31]
> >>> - [FLINK-11499] Extend StreamingFileSink BulkFormats to support
> arbitrary
> >>> roll policies [32]
> >>> - [FLINK-14106] Make SlotManager pluggable [33]
> >>> - [FLINK-15672] Switch to Log4j2 by default [34]
> >>> - [FLINK-15674] Consolidate Java and Scala type extraction stack [35]
> >>> - [FLINK-15679] Improve Flink’s ID system [36]
> >>> - [FLINK-15786] Use the separated classloader to load connectors’ jar
> >> [37]
> >>> - [FLINK-15788] Various Kubernetes improvements [38]
> >>> - [FLINK-15911][FLINK-15154] Support Flink work over NAT [39]
> >>> - [FLINK-16408] Bind user code class loader to lifetime of a slot [40]
> >>> - [FLINK-16428] Network memory management for backpressure [41]
> >>> - [FLINK-16430] Pipelined region scheduling [42]
> >>> - Calculate required shuffle memory before allocating slots
> >>> - State Backend:
> >>> - [FLINK-5763] Make savepoint self-contained / relocatable [43]
> >>> - [FLINK-8871] Complete checkpoint cancellation messages [44]
> >>> - [FLINK-12692] Support disk spilling in HeapKeyedStateBackend [45]
> >>> - [FLINK-15012] Cleanup of leftover files in HDFS/OSS/S3 [46]
> >>> - Enable local recovery by default
> >>> - Support getCustomizedState in KeyedStateStore [47]
> >>> - [FLINK-15532] Enable strict capacity limit for memory usage for
> RocksDB
> >>> [48]
> >>> Your release managers,
> >>> Zhijiang & Piotr Nowojski
> >>>
> >>> [1]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Documentation
> >>> [2]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-65%3A+New+type+inference+for+Table+API+UDFs
> >>> [3]
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878
> >>> [4]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> >>> [5]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-93%3A+JDBC+catalog+and+Postgres+catalog
> >>> [6]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL
> >>> [7]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-107%3A+Reading+table+columns+from+different+parts+of+source+records
> >>> [8] https://issues.apache.org/jira/browse/FLINK-14807
> >>> [9]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> >>> [10] https://issues.apache.org/jira/browse/FLINK-15670
> >>> [11]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> >>> [12]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
> >>> [13]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
> >>> [14]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable
> >>> [15]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
> >>> [16]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager
> >>> [17]
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
> >>> [18] https://issues.apache.org/jira/browse/FLINK-14816
> >>> [19]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation
> >>> [20]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
> >>> [21]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
> >>> [22]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework
> >>> [23]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode
> >>> [24]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-92%3A+Add+N-Ary+Stream+Operator+in+Flink
> >>> [25]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink
> >>> [26]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification
> >>> [27] https://issues.apache.org/jira/browse/FLINK-9407
> >>> [28] https://issues.apache.org/jira/browse/FLINK-10742
> >>> [29] https://issues.apache.org/jira/browse/FLINK-10934
> >>> [30] https://issues.apache.org/jira/browse/FLINK-11395
> >>> [31] https://issues.apache.org/jira/browse/FLINK-11427
> >>> [32] https://issues.apache.org/jira/browse/FLINK-11499
> >>> [33] https://issues.apache.org/jira/browse/FLINK-14106
> >>> [34] https://issues.apache.org/jira/browse/FLINK-15672
> >>> [35] https://issues.apache.org/jira/browse/FLINK-15674
> >>> [36] https://issues.apache.org/jira/browse/FLINK-15679
> >>> [37] https://issues.apache.org/jira/browse/FLINK-15786
> >>> [38] https://issues.apache.org/jira/browse/FLINK-15788
> >>> [39] https://issues.apache.org/jira/browse/FLINK-15911
> >>> [39] https://issues.apache.org/jira/browse/FLINK-15154
> >>> [40] https://issues.apache.org/jira/browse/FLINK-16408
> >>> [41] https://issues.apache.org/jira/browse/FLINK-16428
> >>> [42] https://issues.apache.org/jira/browse/FLINK-16430
> >>> [43] https://issues.apache.org/jira/browse/FLINK-5763
> >>> [44] https://issues.apache.org/jira/browse/FLINK-8871
> >>> [45] https://issues.apache.org/jira/browse/FLINK-12692
> >>> [46] https://issues.apache.org/jira/browse/FLINK-15012
> >>> [47]
> >>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-customize-state-in-customized-KeyedStateBackend-td32771.html
> >>> [48] https://issues.apache.org/jira/browse/FLINK-15532
> >>
> >
>
>

Re: [DISCUSS] Features of Apache Flink 1.11

Posted by Timo Walther <tw...@apache.org>.

Hi Zhijiang and Piotr,

from the SQL side we also plan to rework the source and sink interfaces 
in 1.11. The FLIP is not yet published but already reserved and 
requirement for FLIP-105:

FLIP-95: New TableSource and TableSink interfaces

Thanks for compiling the list!

Regards,
Timo


On 11.03.20 09:05, Hequn Cheng wrote:
> Thanks Zhijiang and Piotr for kicking off the discussion and providing the
> detailed list.
> This would be very helpful for tracking the features.
> 
> BTW, as for PyFlink, it would be great if the feature list can also include
> the following features:
> - FLIP-112: Support User-Defined Metrics in Python UDF
> - FLIP-114: Support Python UDF in SQL Client
> 
> Looking forward to the release!
> 
> Best,
> Hequn
> 
> 
> 
> On Wed, Mar 11, 2020 at 1:02 PM Yu Li <ca...@gmail.com> wrote:
> 
>> Thanks for compiling the list of 1.11 efforts Zhijiang and Piotr! This
>> helps a lot to better understand what the community is currently working
>> on. Looking forward to another successful release.
>>
>> Best Regards,
>> Yu
>>
>>
>> On Wed, 11 Mar 2020 at 11:17, Zhijiang <wangzhijiang999@aliyun.com
>> .invalid>
>> wrote:
>>
>>> Hi community,
>>>
>>>
>>> Not more than one month ago we have released Flink 1.10. We are now
>>> heading for the Flink 1.11 release and we, as release managers, would
>> like
>>> to share with you what are the features that the community is currently
>>> working on and we are hoping that will be part of the Flink 1.11 release.
>>> Currently we are aiming with the feature freeze to happen in late April.
>>>
>>> As for now, some of the features are in the very early stages of the
>>> development or even brainstorming. Because of that, some of them do not
>>> have associated JIRA tickets or FLIP documents. For the next progress
>>> announcement we are hoping that this will be no longer the case.
>>>
>>> Please also note that because we are still relatively at the beginning of
>>> the release cycle, some of the FLIPs haven’t yet been voted.
>>>
>>> - SQL / Table
>>> - FLIP-42: Restructure documentation [1]
>>> - FLIP-65: New type inference for Table API UDFs [2]
>>> - FLIP-84: Improve TableEnv’s interface [3]
>>> - FLIP-91 Introduce SQL client gateway and provide JDBC driver [4]
>>> - FLIP-93: Introduce JDBC catalog and Postgres catalog [5]
>>> - FLIP-105: Support to interpret and emit changelog in Flink SQL [6]
>>> - FLIP-107: Reading table columns from different parts of source records
>>> [7]
>>> - [FLINK-14807] Add Table#collect API for fetching data [8]
>>> - Support query and table hints
>>> - ML / Connectors
>>> - FLIP-27: New source API [9]
>>> - [FLINK-15670] Wrap a source/sink pair to persist intermediate result
>> for
>>> subgraph failure recovery [10]
>>> - Pulsar source / sink / catalog
>>> - Update ML Pipeline API interface to better support Flink ML lib
>>> algorithms
>>> - PyFlink
>>> - FLIP-58: Debugging and monitoring of Python UDF [11]
>>> - FLIP-106: Expand the usage scope of Python UDF [12]
>>> - Integration with most popular Python libraries (Pandas)
>>> - Performance improvements of Python UDF
>>> - Support running python UDF in docker workers
>>> - Add Python ML API
>>> - Fully support all kinds of Python UDF
>>> - Web UI
>>> - FLIP-98: Better back pressure detection [13]
>>> - FLIP-99: Make max exception configurable [14]
>>> - FLIP-100: Add attempt information [15]
>>> - FLIP-102: Add more metrics to TaskManager [16]
>>> - FLIP-103: Better TM/JM log display [17]
>>> - [FLINK-14816] Add thread dump feature for TaskManager [18]
>>> - Runtime
>>> - FLIP-56: Support for dynamic slots on the TaskExecutor [19]
>>> - FLIP-67: Support for cluster partitions [20]
>>> - FLIP-76: Unaligned checkpoints [21]
>>> - FLIP-83: Flink e2e performance testing framework [22]
>>> - FLIP-85: Support cluster deploy mode [23]
>>> - FLIP-92: Add N-Ary input stream operator in Flink [24]
>>> - FLIP-108: Add GPU to the resource management (specifically for UDTF &
>>> UDF) [25]
>>> - FLIP-111: Consolidate docker images [26]
>>> - Unified memory configuration for JobManager
>>> - Specify upper bound for number of allocated TaskManagers
>>> - [FLINK-9407] ORC format for StreamingFileSink [27]
>>> - [FLINK-10742] Let Netty use Flink's buffers on downstream side [28]
>>> - [FLINK-10934] Support per-job mode for Kubernetes integration [29]
>>> - [FLINK-11395] Avro writer for StreamingFileSink [30]
>>> - [FLINK-11427] Protobuf parquet writer for StreamingFileSink [31]
>>> - [FLINK-11499] Extend StreamingFileSink BulkFormats to support arbitrary
>>> roll policies [32]
>>> - [FLINK-14106] Make SlotManager pluggable [33]
>>> - [FLINK-15672] Switch to Log4j2 by default [34]
>>> - [FLINK-15674] Consolidate Java and Scala type extraction stack [35]
>>> - [FLINK-15679] Improve Flink’s ID system [36]
>>> - [FLINK-15786] Use the separated classloader to load connectors’ jar
>> [37]
>>> - [FLINK-15788] Various Kubernetes improvements [38]
>>> - [FLINK-15911][FLINK-15154] Support Flink work over NAT [39]
>>> - [FLINK-16408] Bind user code class loader to lifetime of a slot [40]
>>> - [FLINK-16428] Network memory management for backpressure [41]
>>> - [FLINK-16430] Pipelined region scheduling [42]
>>> - Calculate required shuffle memory before allocating slots
>>> - State Backend:
>>> - [FLINK-5763] Make savepoint self-contained / relocatable [43]
>>> - [FLINK-8871] Complete checkpoint cancellation messages [44]
>>> - [FLINK-12692] Support disk spilling in HeapKeyedStateBackend [45]
>>> - [FLINK-15012] Cleanup of leftover files in HDFS/OSS/S3 [46]
>>> - Enable local recovery by default
>>> - Support getCustomizedState in KeyedStateStore [47]
>>> - [FLINK-15532] Enable strict capacity limit for memory usage for RocksDB
>>> [48]
>>> Your release managers,
>>> Zhijiang & Piotr Nowojski
>>>
>>> [1]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Documentation
>>> [2]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-65%3A+New+type+inference+for+Table+API+UDFs
>>> [3]
>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878
>>> [4]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
>>> [5]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-93%3A+JDBC+catalog+and+Postgres+catalog
>>> [6]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL
>>> [7]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-107%3A+Reading+table+columns+from+different+parts+of+source+records
>>> [8] https://issues.apache.org/jira/browse/FLINK-14807
>>> [9]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
>>> [10] https://issues.apache.org/jira/browse/FLINK-15670
>>> [11]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>>> [12]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
>>> [13]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
>>> [14]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable
>>> [15]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
>>> [16]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager
>>> [17]
>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
>>> [18] https://issues.apache.org/jira/browse/FLINK-14816
>>> [19]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation
>>> [20]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
>>> [21]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
>>> [22]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework
>>> [23]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode
>>> [24]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-92%3A+Add+N-Ary+Stream+Operator+in+Flink
>>> [25]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink
>>> [26]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification
>>> [27] https://issues.apache.org/jira/browse/FLINK-9407
>>> [28] https://issues.apache.org/jira/browse/FLINK-10742
>>> [29] https://issues.apache.org/jira/browse/FLINK-10934
>>> [30] https://issues.apache.org/jira/browse/FLINK-11395
>>> [31] https://issues.apache.org/jira/browse/FLINK-11427
>>> [32] https://issues.apache.org/jira/browse/FLINK-11499
>>> [33] https://issues.apache.org/jira/browse/FLINK-14106
>>> [34] https://issues.apache.org/jira/browse/FLINK-15672
>>> [35] https://issues.apache.org/jira/browse/FLINK-15674
>>> [36] https://issues.apache.org/jira/browse/FLINK-15679
>>> [37] https://issues.apache.org/jira/browse/FLINK-15786
>>> [38] https://issues.apache.org/jira/browse/FLINK-15788
>>> [39] https://issues.apache.org/jira/browse/FLINK-15911
>>> [39] https://issues.apache.org/jira/browse/FLINK-15154
>>> [40] https://issues.apache.org/jira/browse/FLINK-16408
>>> [41] https://issues.apache.org/jira/browse/FLINK-16428
>>> [42] https://issues.apache.org/jira/browse/FLINK-16430
>>> [43] https://issues.apache.org/jira/browse/FLINK-5763
>>> [44] https://issues.apache.org/jira/browse/FLINK-8871
>>> [45] https://issues.apache.org/jira/browse/FLINK-12692
>>> [46] https://issues.apache.org/jira/browse/FLINK-15012
>>> [47]
>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-customize-state-in-customized-KeyedStateBackend-td32771.html
>>> [48] https://issues.apache.org/jira/browse/FLINK-15532
>>
>

Re: [DISCUSS] Features of Apache Flink 1.11

Posted by Hequn Cheng <he...@apache.org>.

Thanks Zhijiang and Piotr for kicking off the discussion and providing the
detailed list.
This would be very helpful for tracking the features.

BTW, as for PyFlink, it would be great if the feature list can also include
the following features:
- FLIP-112: Support User-Defined Metrics in Python UDF
- FLIP-114: Support Python UDF in SQL Client

Looking forward to the release!

Best,
Hequn



On Wed, Mar 11, 2020 at 1:02 PM Yu Li <ca...@gmail.com> wrote:

> Thanks for compiling the list of 1.11 efforts Zhijiang and Piotr! This
> helps a lot to better understand what the community is currently working
> on. Looking forward to another successful release.
>
> Best Regards,
> Yu
>
>
> On Wed, 11 Mar 2020 at 11:17, Zhijiang <wangzhijiang999@aliyun.com
> .invalid>
> wrote:
>
> > Hi community,
> >
> >
> > Not more than one month ago we have released Flink 1.10. We are now
> > heading for the Flink 1.11 release and we, as release managers, would
> like
> > to share with you what are the features that the community is currently
> > working on and we are hoping that will be part of the Flink 1.11 release.
> > Currently we are aiming with the feature freeze to happen in late April.
> >
> > As for now, some of the features are in the very early stages of the
> > development or even brainstorming. Because of that, some of them do not
> > have associated JIRA tickets or FLIP documents. For the next progress
> > announcement we are hoping that this will be no longer the case.
> >
> > Please also note that because we are still relatively at the beginning of
> > the release cycle, some of the FLIPs haven’t yet been voted.
> >
> > - SQL / Table
> > - FLIP-42: Restructure documentation [1]
> > - FLIP-65: New type inference for Table API UDFs [2]
> > - FLIP-84: Improve TableEnv’s interface [3]
> > - FLIP-91 Introduce SQL client gateway and provide JDBC driver [4]
> > - FLIP-93: Introduce JDBC catalog and Postgres catalog [5]
> > - FLIP-105: Support to interpret and emit changelog in Flink SQL [6]
> > - FLIP-107: Reading table columns from different parts of source records
> > [7]
> > - [FLINK-14807] Add Table#collect API for fetching data [8]
> > - Support query and table hints
> > - ML / Connectors
> > - FLIP-27: New source API [9]
> > - [FLINK-15670] Wrap a source/sink pair to persist intermediate result
> for
> > subgraph failure recovery [10]
> > - Pulsar source / sink / catalog
> > - Update ML Pipeline API interface to better support Flink ML lib
> > algorithms
> > - PyFlink
> > - FLIP-58: Debugging and monitoring of Python UDF [11]
> > - FLIP-106: Expand the usage scope of Python UDF [12]
> > - Integration with most popular Python libraries (Pandas)
> > - Performance improvements of Python UDF
> > - Support running python UDF in docker workers
> > - Add Python ML API
> > - Fully support all kinds of Python UDF
> > - Web UI
> > - FLIP-98: Better back pressure detection [13]
> > - FLIP-99: Make max exception configurable [14]
> > - FLIP-100: Add attempt information [15]
> > - FLIP-102: Add more metrics to TaskManager [16]
> > - FLIP-103: Better TM/JM log display [17]
> > - [FLINK-14816] Add thread dump feature for TaskManager [18]
> > - Runtime
> > - FLIP-56: Support for dynamic slots on the TaskExecutor [19]
> > - FLIP-67: Support for cluster partitions [20]
> > - FLIP-76: Unaligned checkpoints [21]
> > - FLIP-83: Flink e2e performance testing framework [22]
> > - FLIP-85: Support cluster deploy mode [23]
> > - FLIP-92: Add N-Ary input stream operator in Flink [24]
> > - FLIP-108: Add GPU to the resource management (specifically for UDTF &
> > UDF) [25]
> > - FLIP-111: Consolidate docker images [26]
> > - Unified memory configuration for JobManager
> > - Specify upper bound for number of allocated TaskManagers
> > - [FLINK-9407] ORC format for StreamingFileSink [27]
> > - [FLINK-10742] Let Netty use Flink's buffers on downstream side [28]
> > - [FLINK-10934] Support per-job mode for Kubernetes integration [29]
> > - [FLINK-11395] Avro writer for StreamingFileSink [30]
> > - [FLINK-11427] Protobuf parquet writer for StreamingFileSink [31]
> > - [FLINK-11499] Extend StreamingFileSink BulkFormats to support arbitrary
> > roll policies [32]
> > - [FLINK-14106] Make SlotManager pluggable [33]
> > - [FLINK-15672] Switch to Log4j2 by default [34]
> > - [FLINK-15674] Consolidate Java and Scala type extraction stack [35]
> > - [FLINK-15679] Improve Flink’s ID system [36]
> > - [FLINK-15786] Use the separated classloader to load connectors’ jar
> [37]
> > - [FLINK-15788] Various Kubernetes improvements [38]
> > - [FLINK-15911][FLINK-15154] Support Flink work over NAT [39]
> > - [FLINK-16408] Bind user code class loader to lifetime of a slot [40]
> > - [FLINK-16428] Network memory management for backpressure [41]
> > - [FLINK-16430] Pipelined region scheduling [42]
> > - Calculate required shuffle memory before allocating slots
> > - State Backend:
> > - [FLINK-5763] Make savepoint self-contained / relocatable [43]
> > - [FLINK-8871] Complete checkpoint cancellation messages [44]
> > - [FLINK-12692] Support disk spilling in HeapKeyedStateBackend [45]
> > - [FLINK-15012] Cleanup of leftover files in HDFS/OSS/S3 [46]
> > - Enable local recovery by default
> > - Support getCustomizedState in KeyedStateStore [47]
> > - [FLINK-15532] Enable strict capacity limit for memory usage for RocksDB
> > [48]
> > Your release managers,
> > Zhijiang & Piotr Nowojski
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Documentation
> > [2]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-65%3A+New+type+inference+for+Table+API+UDFs
> > [3]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878
> > [4]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> > [5]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-93%3A+JDBC+catalog+and+Postgres+catalog
> > [6]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL
> > [7]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-107%3A+Reading+table+columns+from+different+parts+of+source+records
> > [8] https://issues.apache.org/jira/browse/FLINK-14807
> > [9]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> > [10] https://issues.apache.org/jira/browse/FLINK-15670
> > [11]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> > [12]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
> > [13]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
> > [14]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable
> > [15]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
> > [16]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager
> > [17]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
> > [18] https://issues.apache.org/jira/browse/FLINK-14816
> > [19]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation
> > [20]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
> > [21]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
> > [22]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework
> > [23]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode
> > [24]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-92%3A+Add+N-Ary+Stream+Operator+in+Flink
> > [25]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink
> > [26]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification
> > [27] https://issues.apache.org/jira/browse/FLINK-9407
> > [28] https://issues.apache.org/jira/browse/FLINK-10742
> > [29] https://issues.apache.org/jira/browse/FLINK-10934
> > [30] https://issues.apache.org/jira/browse/FLINK-11395
> > [31] https://issues.apache.org/jira/browse/FLINK-11427
> > [32] https://issues.apache.org/jira/browse/FLINK-11499
> > [33] https://issues.apache.org/jira/browse/FLINK-14106
> > [34] https://issues.apache.org/jira/browse/FLINK-15672
> > [35] https://issues.apache.org/jira/browse/FLINK-15674
> > [36] https://issues.apache.org/jira/browse/FLINK-15679
> > [37] https://issues.apache.org/jira/browse/FLINK-15786
> > [38] https://issues.apache.org/jira/browse/FLINK-15788
> > [39] https://issues.apache.org/jira/browse/FLINK-15911
> > [39] https://issues.apache.org/jira/browse/FLINK-15154
> > [40] https://issues.apache.org/jira/browse/FLINK-16408
> > [41] https://issues.apache.org/jira/browse/FLINK-16428
> > [42] https://issues.apache.org/jira/browse/FLINK-16430
> > [43] https://issues.apache.org/jira/browse/FLINK-5763
> > [44] https://issues.apache.org/jira/browse/FLINK-8871
> > [45] https://issues.apache.org/jira/browse/FLINK-12692
> > [46] https://issues.apache.org/jira/browse/FLINK-15012
> > [47]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-customize-state-in-customized-KeyedStateBackend-td32771.html
> > [48] https://issues.apache.org/jira/browse/FLINK-15532
>

Re: [DISCUSS] Features of Apache Flink 1.11

Posted by Yu Li <ca...@gmail.com>.

Thanks for compiling the list of 1.11 efforts Zhijiang and Piotr! This
helps a lot to better understand what the community is currently working
on. Looking forward to another successful release.

Best Regards,
Yu


On Wed, 11 Mar 2020 at 11:17, Zhijiang <wa...@aliyun.com.invalid>
wrote:

> Hi community,
>
>
> Not more than one month ago we have released Flink 1.10. We are now
> heading for the Flink 1.11 release and we, as release managers, would like
> to share with you what are the features that the community is currently
> working on and we are hoping that will be part of the Flink 1.11 release.
> Currently we are aiming with the feature freeze to happen in late April.
>
> As for now, some of the features are in the very early stages of the
> development or even brainstorming. Because of that, some of them do not
> have associated JIRA tickets or FLIP documents. For the next progress
> announcement we are hoping that this will be no longer the case.
>
> Please also note that because we are still relatively at the beginning of
> the release cycle, some of the FLIPs haven’t yet been voted.
>
> - SQL / Table
> - FLIP-42: Restructure documentation [1]
> - FLIP-65: New type inference for Table API UDFs [2]
> - FLIP-84: Improve TableEnv’s interface [3]
> - FLIP-91 Introduce SQL client gateway and provide JDBC driver [4]
> - FLIP-93: Introduce JDBC catalog and Postgres catalog [5]
> - FLIP-105: Support to interpret and emit changelog in Flink SQL [6]
> - FLIP-107: Reading table columns from different parts of source records
> [7]
> - [FLINK-14807] Add Table#collect API for fetching data [8]
> - Support query and table hints
> - ML / Connectors
> - FLIP-27: New source API [9]
> - [FLINK-15670] Wrap a source/sink pair to persist intermediate result for
> subgraph failure recovery [10]
> - Pulsar source / sink / catalog
> - Update ML Pipeline API interface to better support Flink ML lib
> algorithms
> - PyFlink
> - FLIP-58: Debugging and monitoring of Python UDF [11]
> - FLIP-106: Expand the usage scope of Python UDF [12]
> - Integration with most popular Python libraries (Pandas)
> - Performance improvements of Python UDF
> - Support running python UDF in docker workers
> - Add Python ML API
> - Fully support all kinds of Python UDF
> - Web UI
> - FLIP-98: Better back pressure detection [13]
> - FLIP-99: Make max exception configurable [14]
> - FLIP-100: Add attempt information [15]
> - FLIP-102: Add more metrics to TaskManager [16]
> - FLIP-103: Better TM/JM log display [17]
> - [FLINK-14816] Add thread dump feature for TaskManager [18]
> - Runtime
> - FLIP-56: Support for dynamic slots on the TaskExecutor [19]
> - FLIP-67: Support for cluster partitions [20]
> - FLIP-76: Unaligned checkpoints [21]
> - FLIP-83: Flink e2e performance testing framework [22]
> - FLIP-85: Support cluster deploy mode [23]
> - FLIP-92: Add N-Ary input stream operator in Flink [24]
> - FLIP-108: Add GPU to the resource management (specifically for UDTF &
> UDF) [25]
> - FLIP-111: Consolidate docker images [26]
> - Unified memory configuration for JobManager
> - Specify upper bound for number of allocated TaskManagers
> - [FLINK-9407] ORC format for StreamingFileSink [27]
> - [FLINK-10742] Let Netty use Flink's buffers on downstream side [28]
> - [FLINK-10934] Support per-job mode for Kubernetes integration [29]
> - [FLINK-11395] Avro writer for StreamingFileSink [30]
> - [FLINK-11427] Protobuf parquet writer for StreamingFileSink [31]
> - [FLINK-11499] Extend StreamingFileSink BulkFormats to support arbitrary
> roll policies [32]
> - [FLINK-14106] Make SlotManager pluggable [33]
> - [FLINK-15672] Switch to Log4j2 by default [34]
> - [FLINK-15674] Consolidate Java and Scala type extraction stack [35]
> - [FLINK-15679] Improve Flink’s ID system [36]
> - [FLINK-15786] Use the separated classloader to load connectors’ jar [37]
> - [FLINK-15788] Various Kubernetes improvements [38]
> - [FLINK-15911][FLINK-15154] Support Flink work over NAT [39]
> - [FLINK-16408] Bind user code class loader to lifetime of a slot [40]
> - [FLINK-16428] Network memory management for backpressure [41]
> - [FLINK-16430] Pipelined region scheduling [42]
> - Calculate required shuffle memory before allocating slots
> - State Backend:
> - [FLINK-5763] Make savepoint self-contained / relocatable [43]
> - [FLINK-8871] Complete checkpoint cancellation messages [44]
> - [FLINK-12692] Support disk spilling in HeapKeyedStateBackend [45]
> - [FLINK-15012] Cleanup of leftover files in HDFS/OSS/S3 [46]
> - Enable local recovery by default
> - Support getCustomizedState in KeyedStateStore [47]
> - [FLINK-15532] Enable strict capacity limit for memory usage for RocksDB
> [48]
> Your release managers,
> Zhijiang & Piotr Nowojski
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Documentation
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-65%3A+New+type+inference+for+Table+API+UDFs
> [3]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878
> [4]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> [5]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-93%3A+JDBC+catalog+and+Postgres+catalog
> [6]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL
> [7]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-107%3A+Reading+table+columns+from+different+parts+of+source+records
> [8] https://issues.apache.org/jira/browse/FLINK-14807
> [9]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> [10] https://issues.apache.org/jira/browse/FLINK-15670
> [11]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> [12]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
> [13]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
> [14]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable
> [15]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
> [16]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager
> [17]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
> [18] https://issues.apache.org/jira/browse/FLINK-14816
> [19]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation
> [20]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
> [21]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
> [22]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework
> [23]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode
> [24]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-92%3A+Add+N-Ary+Stream+Operator+in+Flink
> [25]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink
> [26]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification
> [27] https://issues.apache.org/jira/browse/FLINK-9407
> [28] https://issues.apache.org/jira/browse/FLINK-10742
> [29] https://issues.apache.org/jira/browse/FLINK-10934
> [30] https://issues.apache.org/jira/browse/FLINK-11395
> [31] https://issues.apache.org/jira/browse/FLINK-11427
> [32] https://issues.apache.org/jira/browse/FLINK-11499
> [33] https://issues.apache.org/jira/browse/FLINK-14106
> [34] https://issues.apache.org/jira/browse/FLINK-15672
> [35] https://issues.apache.org/jira/browse/FLINK-15674
> [36] https://issues.apache.org/jira/browse/FLINK-15679
> [37] https://issues.apache.org/jira/browse/FLINK-15786
> [38] https://issues.apache.org/jira/browse/FLINK-15788
> [39] https://issues.apache.org/jira/browse/FLINK-15911
> [39] https://issues.apache.org/jira/browse/FLINK-15154
> [40] https://issues.apache.org/jira/browse/FLINK-16408
> [41] https://issues.apache.org/jira/browse/FLINK-16428
> [42] https://issues.apache.org/jira/browse/FLINK-16430
> [43] https://issues.apache.org/jira/browse/FLINK-5763
> [44] https://issues.apache.org/jira/browse/FLINK-8871
> [45] https://issues.apache.org/jira/browse/FLINK-12692
> [46] https://issues.apache.org/jira/browse/FLINK-15012
> [47]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-customize-state-in-customized-KeyedStateBackend-td32771.html
> [48] https://issues.apache.org/jira/browse/FLINK-15532