You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Gary Yao <ga...@ververica.com> on 2019/11/01 13:41:13 UTC
[ANNOUNCE] Progress of Apache Flink 1.10 #2
Hi community,
Because we have approximately one month of development time left until the
targeted Flink 1.10 feature freeze, we thought now would be a good time to
give another progress update. Below we have included a list of the ongoing
efforts that have made progress since our last release progress update [1].
As
always, if you are working on something that is not included here, feel free
to use this thread to share your progress.
- Support Java 11 [2]
- Implementation is in progress (18/21 subtasks resolved)
- Table API improvements
- Full Data Type Support in Planner [3]
- Implementing (1/8 subtasks resolved)
- FLIP-66 Support Time Attribute in SQL DDL [4]
- Implementation is in progress (1/7 subtasks resolved).
- FLIP-70 Support Computed Column [5]
- FLIP voting [6]
- FLIP-63 Rework Table Partition Support [7]
- Implementation is in progress (3/15 subtasks resolved).
- FLIP-51 Rework of Expression Design [8]
- Implementation is in progress (2/12 subtasks resolved).
- FLIP-64 Support for Temporary Objects in Table Module [9]
- Implementation is in progress
- Hive compatibility completion (DDL/UDF) to support full Hive integration
- FLIP-57 Rework FunctionCatalog [10]
- Implementation is in progress (6/9 subtasks resolved)
- FLIP-68 Extend Core Table System with Modular Plugins [11]
- Implementation is in progress (2/8 subtasks resolved)
- Finer grained resource management
- FLIP-49: Unified Memory Configuration for TaskExecutors [12]
- Implementation is in progress (6/10 subtasks resolved)
- FLIP-53: Fine Grained Operator Resource Management [13]
- Implementation is in progress (1/9 subtasks resolved)
- Finish scheduler re-architecture [14]
- Integration tests are being enabled for new scheduler
- Executor/Client refactoring [15]
- FLIP-81: Executor-related new ConfigOptions [16]
- done
- FLIP-73: Introducing Executors for job submission [17]
- Implementation is in progress
- FLIP-36 Support Interactive Programming [18]
- Is built on top of FLIP-67 [19], which has been accepted
- Implementation in progress
- FLIP-58: Flink Python User-Defined Stateless Function for Table [20]
- Implementation is in progress (12/22 subtask resolved)
- FLIP-50: Spill-able Heap Keyed State Backend [21]
- Implementation is in progress (2/11 subtasks resolved)
- RocksDB Backend Memory Control [22]
- FLIP for resource management on state backend will be opened soon
- Write Buffer Manager will be backported to FRocksDB due to
performance regression [23] in new RocksDB versions
- Unaligned Checkpoints
- FLIP-76 [24] was published and received positive feedback
- Implementation is in progress
- Separate framework and user class loader in per-job mode [25]
- First PR is almost done. Remaining PRs will be ready next week
- Active Kubernetes Integration [26]
- Implementation is in progress (6/11 in review, 3/11 in progress, 2/11
todo)
- FLIP-39 Flink ML pipeline and ML libs [27]
- A few abstract ML classes have been merged (FLINK-13339, FLINK-13513)
- Starting review of algorithms
Again, the feature freeze is targeted to be at the end of November. Please
make sure that all important work threads can be completed until that date.
Feel free to use this thread to communicate any concerns about features that
might not be finished until then. We will send another announcement later in
the release cycle to make the date of the feature freeze official.
Best,
Yu & Gary
[1] https://s.apache.org/wc0dc
[2] https://issues.apache.org/jira/browse/FLINK-10725
[3] https://issues.apache.org/jira/browse/FLINK-14079
[4]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL
[5]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design
[6]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-70-Flink-SQL-Computed-Column-Design-td34385.html
[7]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
[8]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
[9]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module
[10]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog
[11]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
[12]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
[13]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
[14] https://issues.apache.org/jira/browse/FLINK-10429
[15]
https://lists.apache.org/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@%3Cdev.flink.apache.org%3E
[16]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524
[17]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission
[18]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
[19]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
[20]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
[21]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
[22] https://issues.apache.org/jira/browse/FLINK-7289
[23] https://github.com/facebook/rocksdb/issues/5774
[24]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
[25] https://issues.apache.org/jira/browse/FLINK-13993
[26] https://issues.apache.org/jira/browse/FLINK-9953
[27]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2
Posted by Becket Qin <be...@gmail.com>.
Hi Thomas,
Event time alignment is absolutely one of the important considerations of
FLIP-27. That said, we are not implementing that in FLIP-27, but just make
sure such feature can be easily added in the future. The design was to make
the communication between SplitEnumerator and SourceReader quite extensible
so we can put more features on top of it. More specifically, we will have
two tiers of events flowing between the SourceReader and Enumerator -
OperatorEvent and SourceEvent.
The OperatorEvents are "lower" level events that are not visible to the
user provided SourceReader / SplitEnumerator directly. Instead they are
handled by the SourceOperator / SourceCoordinator that run SourceReader /
SplitEnumerator.
The SourceEvents are a subclass of OperatorEvents. They are "higher" level
events that each connector implementation can have their own events so the
SourceReader and SplitEnumerator can exchange arbitrary information.
The event time alignment can be implemented on top of this communication
mechanism:
1. Each SourceReader periodically reports the current watermark to the
SplitEnumerator via a SourceEvent.
2. The SplitEnumerator collects the global watermark frontier and decide to
apply some flow control to certain splits by sending another SourceEvent to
relevant SourceReader.
3. The SourceReaders will then refrain from reading the throttled splits.
This is just one implementation, the SplitEnumerator may also just
broadcast the current global watermark to each SourceReader and let
themselves make decision. With some API change, there is also a possibility
to make this alignment handled at the SourceOperator / SourceCoordinator
level. But in any case, after FLIP-27 such feature addition should be
fairly easy.
FLIP-27 is already quite a huge change by itself. I think there will be at
least 10K lines of additions. Because of that, it might worth confine the
scope a little bit. So I do not plan to implement event time alignment in
FLIP-27. But we did think about it and I think it can be implemented easily
afterwards.
Thanks,
Jiangjie (Becket) Qin
On Mon, Nov 4, 2019 at 10:18 PM Thomas Weise <th...@apache.org> wrote:
> Hi Becket,
>
> Thanks for the reply, it is good to know that there is activity on FLIP-27.
>
> A while ago I was wondering if event time alignment is on the radar [1],
> can you please clarify that?
>
> There is a parallel discussion of adding it to the existing Kafka consumer
> [2], could you please take a look and help figuring out the path forward?
>
> Thanks,
> Thomas
>
> [1]
> https://lists.apache.org/thread.html/c7336f259bbbb4c482de0580ecf2472a6b3e5a0fe2b7dd9258b159f2@%3Cdev.flink.apache.org%3E
> [2] https://issues.apache.org/jira/browse/FLINK-12675
>
>
>
> On Mon, Nov 4, 2019 at 5:48 AM Becket Qin <be...@gmail.com> wrote:
>
>> Hi Steven and Thomas,
>>
>> Sorry about missing the update of FLIP-27.
>>
>> I am working on the implementation of FLIP-27 at this point. It is about
>> 70% done. Right now I am integrating the source coordinator to the job
>> master. Hopefully I can get the basics of Kafka connector work from end to
>> end by this Sunday. Then I'll update FLIP-27 wiki page.
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>>
>> On Sat, Nov 2, 2019 at 8:18 AM Thomas Weise <th...@apache.org> wrote:
>>
>>> Is there any activity on FLIP-27 that would make it relevant for 1.10
>>> release?
>>>
>>> Thanks Gary for the update, it provides excellent visibility on current
>>> activity and what we can expect with the release.
>>>
>>>
>>> On Fri, Nov 1, 2019 at 1:52 PM Steven Wu <st...@gmail.com> wrote:
>>>
>>>> Gary, FLIP-27 seems to get omitted in the 2nd update. below is the info
>>>> from update #1.
>>>>
>>>> - FLIP-27: Refactor Source Interface [20]
>>>> - FLIP accepted. Implementation is in progress.
>>>>
>>>>
>>>>
>>>> On Fri, Nov 1, 2019 at 7:01 AM Gary Yao <ga...@ververica.com> wrote:
>>>>
>>>> > Hi community,
>>>> >
>>>> > Because we have approximately one month of development time left
>>>> until the
>>>> > targeted Flink 1.10 feature freeze, we thought now would be a good
>>>> time to
>>>> > give another progress update. Below we have included a list of the
>>>> ongoing
>>>> > efforts that have made progress since our last release progress update
>>>> > [1]. As
>>>> > always, if you are working on something that is not included here,
>>>> feel
>>>> > free
>>>> > to use this thread to share your progress.
>>>> >
>>>> > - Support Java 11 [2]
>>>> > - Implementation is in progress (18/21 subtasks resolved)
>>>> >
>>>> > - Table API improvements
>>>> > - Full Data Type Support in Planner [3]
>>>> > - Implementing (1/8 subtasks resolved)
>>>> > - FLIP-66 Support Time Attribute in SQL DDL [4]
>>>> > - Implementation is in progress (1/7 subtasks resolved).
>>>> > - FLIP-70 Support Computed Column [5]
>>>> > - FLIP voting [6]
>>>> > - FLIP-63 Rework Table Partition Support [7]
>>>> > - Implementation is in progress (3/15 subtasks resolved).
>>>> > - FLIP-51 Rework of Expression Design [8]
>>>> > - Implementation is in progress (2/12 subtasks resolved).
>>>> > - FLIP-64 Support for Temporary Objects in Table Module [9]
>>>> > - Implementation is in progress
>>>> >
>>>> > - Hive compatibility completion (DDL/UDF) to support full Hive
>>>> integration
>>>> > - FLIP-57 Rework FunctionCatalog [10]
>>>> > - Implementation is in progress (6/9 subtasks resolved)
>>>> > - FLIP-68 Extend Core Table System with Modular Plugins [11]
>>>> > - Implementation is in progress (2/8 subtasks resolved)
>>>> >
>>>> > - Finer grained resource management
>>>> > - FLIP-49: Unified Memory Configuration for TaskExecutors [12]
>>>> > - Implementation is in progress (6/10 subtasks resolved)
>>>> > - FLIP-53: Fine Grained Operator Resource Management [13]
>>>> > - Implementation is in progress (1/9 subtasks resolved)
>>>> >
>>>> > - Finish scheduler re-architecture [14]
>>>> > - Integration tests are being enabled for new scheduler
>>>> >
>>>> > - Executor/Client refactoring [15]
>>>> > - FLIP-81: Executor-related new ConfigOptions [16]
>>>> > - done
>>>> > - FLIP-73: Introducing Executors for job submission [17]
>>>> > - Implementation is in progress
>>>> >
>>>> > - FLIP-36 Support Interactive Programming [18]
>>>> > - Is built on top of FLIP-67 [19], which has been accepted
>>>> > - Implementation in progress
>>>> >
>>>> > - FLIP-58: Flink Python User-Defined Stateless Function for Table [20]
>>>> > - Implementation is in progress (12/22 subtask resolved)
>>>> > - FLIP-50: Spill-able Heap Keyed State Backend [21]
>>>> > - Implementation is in progress (2/11 subtasks resolved)
>>>> >
>>>> > - RocksDB Backend Memory Control [22]
>>>> > - FLIP for resource management on state backend will be opened
>>>> soon
>>>> > - Write Buffer Manager will be backported to FRocksDB due to
>>>> > performance regression [23] in new RocksDB versions
>>>> >
>>>> > - Unaligned Checkpoints
>>>> > - FLIP-76 [24] was published and received positive feedback
>>>> > - Implementation is in progress
>>>> >
>>>> > - Separate framework and user class loader in per-job mode [25]
>>>> > - First PR is almost done. Remaining PRs will be ready next week
>>>> >
>>>> > - Active Kubernetes Integration [26]
>>>> > - Implementation is in progress (6/11 in review, 3/11 in progress,
>>>> > 2/11 todo)
>>>> >
>>>> > - FLIP-39 Flink ML pipeline and ML libs [27]
>>>> > - A few abstract ML classes have been merged (FLINK-13339,
>>>> > FLINK-13513)
>>>> > - Starting review of algorithms
>>>> >
>>>> > Again, the feature freeze is targeted to be at the end of November.
>>>> Please
>>>> > make sure that all important work threads can be completed until that
>>>> date.
>>>> > Feel free to use this thread to communicate any concerns about
>>>> features
>>>> > that
>>>> > might not be finished until then. We will send another announcement
>>>> later
>>>> > in
>>>> > the release cycle to make the date of the feature freeze official.
>>>> >
>>>> > Best,
>>>> > Yu & Gary
>>>> >
>>>> > [1] https://s.apache.org/wc0dc
>>>> > [2] https://issues.apache.org/jira/browse/FLINK-10725
>>>> > [3] https://issues.apache.org/jira/browse/FLINK-14079
>>>> > [4]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL
>>>> > [5]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design
>>>> > [6]
>>>> >
>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-70-Flink-SQL-Computed-Column-Design-td34385.html
>>>> > [7]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
>>>> > [8]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
>>>> > [9]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module
>>>> > [10]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog
>>>> > [11]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
>>>> > [12]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
>>>> > [13]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
>>>> > [14] https://issues.apache.org/jira/browse/FLINK-10429
>>>> > [15]
>>>> >
>>>> https://lists.apache.org/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@%3Cdev.flink.apache.org%3E
>>>> > [16]
>>>> >
>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524
>>>> > [17]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission
>>>> > [18]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
>>>> > [19]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
>>>> > [20]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>>>> > [21]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
>>>> > [22] https://issues.apache.org/jira/browse/FLINK-7289
>>>> > [23] https://github.com/facebook/rocksdb/issues/5774
>>>> > [24]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
>>>> > [25] https://issues.apache.org/jira/browse/FLINK-13993
>>>> > [26] https://issues.apache.org/jira/browse/FLINK-9953
>>>> > [27]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
>>>> >
>>>>
>>>
Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2
Posted by Becket Qin <be...@gmail.com>.
Hi Thomas,
Event time alignment is absolutely one of the important considerations of
FLIP-27. That said, we are not implementing that in FLIP-27, but just make
sure such feature can be easily added in the future. The design was to make
the communication between SplitEnumerator and SourceReader quite extensible
so we can put more features on top of it. More specifically, we will have
two tiers of events flowing between the SourceReader and Enumerator -
OperatorEvent and SourceEvent.
The OperatorEvents are "lower" level events that are not visible to the
user provided SourceReader / SplitEnumerator directly. Instead they are
handled by the SourceOperator / SourceCoordinator that run SourceReader /
SplitEnumerator.
The SourceEvents are a subclass of OperatorEvents. They are "higher" level
events that each connector implementation can have their own events so the
SourceReader and SplitEnumerator can exchange arbitrary information.
The event time alignment can be implemented on top of this communication
mechanism:
1. Each SourceReader periodically reports the current watermark to the
SplitEnumerator via a SourceEvent.
2. The SplitEnumerator collects the global watermark frontier and decide to
apply some flow control to certain splits by sending another SourceEvent to
relevant SourceReader.
3. The SourceReaders will then refrain from reading the throttled splits.
This is just one implementation, the SplitEnumerator may also just
broadcast the current global watermark to each SourceReader and let
themselves make decision. With some API change, there is also a possibility
to make this alignment handled at the SourceOperator / SourceCoordinator
level. But in any case, after FLIP-27 such feature addition should be
fairly easy.
FLIP-27 is already quite a huge change by itself. I think there will be at
least 10K lines of additions. Because of that, it might worth confine the
scope a little bit. So I do not plan to implement event time alignment in
FLIP-27. But we did think about it and I think it can be implemented easily
afterwards.
Thanks,
Jiangjie (Becket) Qin
On Mon, Nov 4, 2019 at 10:18 PM Thomas Weise <th...@apache.org> wrote:
> Hi Becket,
>
> Thanks for the reply, it is good to know that there is activity on FLIP-27.
>
> A while ago I was wondering if event time alignment is on the radar [1],
> can you please clarify that?
>
> There is a parallel discussion of adding it to the existing Kafka consumer
> [2], could you please take a look and help figuring out the path forward?
>
> Thanks,
> Thomas
>
> [1]
> https://lists.apache.org/thread.html/c7336f259bbbb4c482de0580ecf2472a6b3e5a0fe2b7dd9258b159f2@%3Cdev.flink.apache.org%3E
> [2] https://issues.apache.org/jira/browse/FLINK-12675
>
>
>
> On Mon, Nov 4, 2019 at 5:48 AM Becket Qin <be...@gmail.com> wrote:
>
>> Hi Steven and Thomas,
>>
>> Sorry about missing the update of FLIP-27.
>>
>> I am working on the implementation of FLIP-27 at this point. It is about
>> 70% done. Right now I am integrating the source coordinator to the job
>> master. Hopefully I can get the basics of Kafka connector work from end to
>> end by this Sunday. Then I'll update FLIP-27 wiki page.
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>>
>> On Sat, Nov 2, 2019 at 8:18 AM Thomas Weise <th...@apache.org> wrote:
>>
>>> Is there any activity on FLIP-27 that would make it relevant for 1.10
>>> release?
>>>
>>> Thanks Gary for the update, it provides excellent visibility on current
>>> activity and what we can expect with the release.
>>>
>>>
>>> On Fri, Nov 1, 2019 at 1:52 PM Steven Wu <st...@gmail.com> wrote:
>>>
>>>> Gary, FLIP-27 seems to get omitted in the 2nd update. below is the info
>>>> from update #1.
>>>>
>>>> - FLIP-27: Refactor Source Interface [20]
>>>> - FLIP accepted. Implementation is in progress.
>>>>
>>>>
>>>>
>>>> On Fri, Nov 1, 2019 at 7:01 AM Gary Yao <ga...@ververica.com> wrote:
>>>>
>>>> > Hi community,
>>>> >
>>>> > Because we have approximately one month of development time left
>>>> until the
>>>> > targeted Flink 1.10 feature freeze, we thought now would be a good
>>>> time to
>>>> > give another progress update. Below we have included a list of the
>>>> ongoing
>>>> > efforts that have made progress since our last release progress update
>>>> > [1]. As
>>>> > always, if you are working on something that is not included here,
>>>> feel
>>>> > free
>>>> > to use this thread to share your progress.
>>>> >
>>>> > - Support Java 11 [2]
>>>> > - Implementation is in progress (18/21 subtasks resolved)
>>>> >
>>>> > - Table API improvements
>>>> > - Full Data Type Support in Planner [3]
>>>> > - Implementing (1/8 subtasks resolved)
>>>> > - FLIP-66 Support Time Attribute in SQL DDL [4]
>>>> > - Implementation is in progress (1/7 subtasks resolved).
>>>> > - FLIP-70 Support Computed Column [5]
>>>> > - FLIP voting [6]
>>>> > - FLIP-63 Rework Table Partition Support [7]
>>>> > - Implementation is in progress (3/15 subtasks resolved).
>>>> > - FLIP-51 Rework of Expression Design [8]
>>>> > - Implementation is in progress (2/12 subtasks resolved).
>>>> > - FLIP-64 Support for Temporary Objects in Table Module [9]
>>>> > - Implementation is in progress
>>>> >
>>>> > - Hive compatibility completion (DDL/UDF) to support full Hive
>>>> integration
>>>> > - FLIP-57 Rework FunctionCatalog [10]
>>>> > - Implementation is in progress (6/9 subtasks resolved)
>>>> > - FLIP-68 Extend Core Table System with Modular Plugins [11]
>>>> > - Implementation is in progress (2/8 subtasks resolved)
>>>> >
>>>> > - Finer grained resource management
>>>> > - FLIP-49: Unified Memory Configuration for TaskExecutors [12]
>>>> > - Implementation is in progress (6/10 subtasks resolved)
>>>> > - FLIP-53: Fine Grained Operator Resource Management [13]
>>>> > - Implementation is in progress (1/9 subtasks resolved)
>>>> >
>>>> > - Finish scheduler re-architecture [14]
>>>> > - Integration tests are being enabled for new scheduler
>>>> >
>>>> > - Executor/Client refactoring [15]
>>>> > - FLIP-81: Executor-related new ConfigOptions [16]
>>>> > - done
>>>> > - FLIP-73: Introducing Executors for job submission [17]
>>>> > - Implementation is in progress
>>>> >
>>>> > - FLIP-36 Support Interactive Programming [18]
>>>> > - Is built on top of FLIP-67 [19], which has been accepted
>>>> > - Implementation in progress
>>>> >
>>>> > - FLIP-58: Flink Python User-Defined Stateless Function for Table [20]
>>>> > - Implementation is in progress (12/22 subtask resolved)
>>>> > - FLIP-50: Spill-able Heap Keyed State Backend [21]
>>>> > - Implementation is in progress (2/11 subtasks resolved)
>>>> >
>>>> > - RocksDB Backend Memory Control [22]
>>>> > - FLIP for resource management on state backend will be opened
>>>> soon
>>>> > - Write Buffer Manager will be backported to FRocksDB due to
>>>> > performance regression [23] in new RocksDB versions
>>>> >
>>>> > - Unaligned Checkpoints
>>>> > - FLIP-76 [24] was published and received positive feedback
>>>> > - Implementation is in progress
>>>> >
>>>> > - Separate framework and user class loader in per-job mode [25]
>>>> > - First PR is almost done. Remaining PRs will be ready next week
>>>> >
>>>> > - Active Kubernetes Integration [26]
>>>> > - Implementation is in progress (6/11 in review, 3/11 in progress,
>>>> > 2/11 todo)
>>>> >
>>>> > - FLIP-39 Flink ML pipeline and ML libs [27]
>>>> > - A few abstract ML classes have been merged (FLINK-13339,
>>>> > FLINK-13513)
>>>> > - Starting review of algorithms
>>>> >
>>>> > Again, the feature freeze is targeted to be at the end of November.
>>>> Please
>>>> > make sure that all important work threads can be completed until that
>>>> date.
>>>> > Feel free to use this thread to communicate any concerns about
>>>> features
>>>> > that
>>>> > might not be finished until then. We will send another announcement
>>>> later
>>>> > in
>>>> > the release cycle to make the date of the feature freeze official.
>>>> >
>>>> > Best,
>>>> > Yu & Gary
>>>> >
>>>> > [1] https://s.apache.org/wc0dc
>>>> > [2] https://issues.apache.org/jira/browse/FLINK-10725
>>>> > [3] https://issues.apache.org/jira/browse/FLINK-14079
>>>> > [4]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL
>>>> > [5]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design
>>>> > [6]
>>>> >
>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-70-Flink-SQL-Computed-Column-Design-td34385.html
>>>> > [7]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
>>>> > [8]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
>>>> > [9]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module
>>>> > [10]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog
>>>> > [11]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
>>>> > [12]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
>>>> > [13]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
>>>> > [14] https://issues.apache.org/jira/browse/FLINK-10429
>>>> > [15]
>>>> >
>>>> https://lists.apache.org/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@%3Cdev.flink.apache.org%3E
>>>> > [16]
>>>> >
>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524
>>>> > [17]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission
>>>> > [18]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
>>>> > [19]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
>>>> > [20]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>>>> > [21]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
>>>> > [22] https://issues.apache.org/jira/browse/FLINK-7289
>>>> > [23] https://github.com/facebook/rocksdb/issues/5774
>>>> > [24]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
>>>> > [25] https://issues.apache.org/jira/browse/FLINK-13993
>>>> > [26] https://issues.apache.org/jira/browse/FLINK-9953
>>>> > [27]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
>>>> >
>>>>
>>>
Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2
Posted by Thomas Weise <th...@apache.org>.
Hi Becket,
Thanks for the reply, it is good to know that there is activity on FLIP-27.
A while ago I was wondering if event time alignment is on the radar [1],
can you please clarify that?
There is a parallel discussion of adding it to the existing Kafka consumer
[2], could you please take a look and help figuring out the path forward?
Thanks,
Thomas
[1]
https://lists.apache.org/thread.html/c7336f259bbbb4c482de0580ecf2472a6b3e5a0fe2b7dd9258b159f2@%3Cdev.flink.apache.org%3E
[2] https://issues.apache.org/jira/browse/FLINK-12675
On Mon, Nov 4, 2019 at 5:48 AM Becket Qin <be...@gmail.com> wrote:
> Hi Steven and Thomas,
>
> Sorry about missing the update of FLIP-27.
>
> I am working on the implementation of FLIP-27 at this point. It is about
> 70% done. Right now I am integrating the source coordinator to the job
> master. Hopefully I can get the basics of Kafka connector work from end to
> end by this Sunday. Then I'll update FLIP-27 wiki page.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
> On Sat, Nov 2, 2019 at 8:18 AM Thomas Weise <th...@apache.org> wrote:
>
>> Is there any activity on FLIP-27 that would make it relevant for 1.10
>> release?
>>
>> Thanks Gary for the update, it provides excellent visibility on current
>> activity and what we can expect with the release.
>>
>>
>> On Fri, Nov 1, 2019 at 1:52 PM Steven Wu <st...@gmail.com> wrote:
>>
>>> Gary, FLIP-27 seems to get omitted in the 2nd update. below is the info
>>> from update #1.
>>>
>>> - FLIP-27: Refactor Source Interface [20]
>>> - FLIP accepted. Implementation is in progress.
>>>
>>>
>>>
>>> On Fri, Nov 1, 2019 at 7:01 AM Gary Yao <ga...@ververica.com> wrote:
>>>
>>> > Hi community,
>>> >
>>> > Because we have approximately one month of development time left until
>>> the
>>> > targeted Flink 1.10 feature freeze, we thought now would be a good
>>> time to
>>> > give another progress update. Below we have included a list of the
>>> ongoing
>>> > efforts that have made progress since our last release progress update
>>> > [1]. As
>>> > always, if you are working on something that is not included here, feel
>>> > free
>>> > to use this thread to share your progress.
>>> >
>>> > - Support Java 11 [2]
>>> > - Implementation is in progress (18/21 subtasks resolved)
>>> >
>>> > - Table API improvements
>>> > - Full Data Type Support in Planner [3]
>>> > - Implementing (1/8 subtasks resolved)
>>> > - FLIP-66 Support Time Attribute in SQL DDL [4]
>>> > - Implementation is in progress (1/7 subtasks resolved).
>>> > - FLIP-70 Support Computed Column [5]
>>> > - FLIP voting [6]
>>> > - FLIP-63 Rework Table Partition Support [7]
>>> > - Implementation is in progress (3/15 subtasks resolved).
>>> > - FLIP-51 Rework of Expression Design [8]
>>> > - Implementation is in progress (2/12 subtasks resolved).
>>> > - FLIP-64 Support for Temporary Objects in Table Module [9]
>>> > - Implementation is in progress
>>> >
>>> > - Hive compatibility completion (DDL/UDF) to support full Hive
>>> integration
>>> > - FLIP-57 Rework FunctionCatalog [10]
>>> > - Implementation is in progress (6/9 subtasks resolved)
>>> > - FLIP-68 Extend Core Table System with Modular Plugins [11]
>>> > - Implementation is in progress (2/8 subtasks resolved)
>>> >
>>> > - Finer grained resource management
>>> > - FLIP-49: Unified Memory Configuration for TaskExecutors [12]
>>> > - Implementation is in progress (6/10 subtasks resolved)
>>> > - FLIP-53: Fine Grained Operator Resource Management [13]
>>> > - Implementation is in progress (1/9 subtasks resolved)
>>> >
>>> > - Finish scheduler re-architecture [14]
>>> > - Integration tests are being enabled for new scheduler
>>> >
>>> > - Executor/Client refactoring [15]
>>> > - FLIP-81: Executor-related new ConfigOptions [16]
>>> > - done
>>> > - FLIP-73: Introducing Executors for job submission [17]
>>> > - Implementation is in progress
>>> >
>>> > - FLIP-36 Support Interactive Programming [18]
>>> > - Is built on top of FLIP-67 [19], which has been accepted
>>> > - Implementation in progress
>>> >
>>> > - FLIP-58: Flink Python User-Defined Stateless Function for Table [20]
>>> > - Implementation is in progress (12/22 subtask resolved)
>>> > - FLIP-50: Spill-able Heap Keyed State Backend [21]
>>> > - Implementation is in progress (2/11 subtasks resolved)
>>> >
>>> > - RocksDB Backend Memory Control [22]
>>> > - FLIP for resource management on state backend will be opened soon
>>> > - Write Buffer Manager will be backported to FRocksDB due to
>>> > performance regression [23] in new RocksDB versions
>>> >
>>> > - Unaligned Checkpoints
>>> > - FLIP-76 [24] was published and received positive feedback
>>> > - Implementation is in progress
>>> >
>>> > - Separate framework and user class loader in per-job mode [25]
>>> > - First PR is almost done. Remaining PRs will be ready next week
>>> >
>>> > - Active Kubernetes Integration [26]
>>> > - Implementation is in progress (6/11 in review, 3/11 in progress,
>>> > 2/11 todo)
>>> >
>>> > - FLIP-39 Flink ML pipeline and ML libs [27]
>>> > - A few abstract ML classes have been merged (FLINK-13339,
>>> > FLINK-13513)
>>> > - Starting review of algorithms
>>> >
>>> > Again, the feature freeze is targeted to be at the end of November.
>>> Please
>>> > make sure that all important work threads can be completed until that
>>> date.
>>> > Feel free to use this thread to communicate any concerns about features
>>> > that
>>> > might not be finished until then. We will send another announcement
>>> later
>>> > in
>>> > the release cycle to make the date of the feature freeze official.
>>> >
>>> > Best,
>>> > Yu & Gary
>>> >
>>> > [1] https://s.apache.org/wc0dc
>>> > [2] https://issues.apache.org/jira/browse/FLINK-10725
>>> > [3] https://issues.apache.org/jira/browse/FLINK-14079
>>> > [4]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL
>>> > [5]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design
>>> > [6]
>>> >
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-70-Flink-SQL-Computed-Column-Design-td34385.html
>>> > [7]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
>>> > [8]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
>>> > [9]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module
>>> > [10]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog
>>> > [11]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
>>> > [12]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
>>> > [13]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
>>> > [14] https://issues.apache.org/jira/browse/FLINK-10429
>>> > [15]
>>> >
>>> https://lists.apache.org/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@%3Cdev.flink.apache.org%3E
>>> > [16]
>>> >
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524
>>> > [17]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission
>>> > [18]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
>>> > [19]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
>>> > [20]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>>> > [21]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
>>> > [22] https://issues.apache.org/jira/browse/FLINK-7289
>>> > [23] https://github.com/facebook/rocksdb/issues/5774
>>> > [24]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
>>> > [25] https://issues.apache.org/jira/browse/FLINK-13993
>>> > [26] https://issues.apache.org/jira/browse/FLINK-9953
>>> > [27]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
>>> >
>>>
>>
Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2
Posted by Thomas Weise <th...@apache.org>.
Hi Becket,
Thanks for the reply, it is good to know that there is activity on FLIP-27.
A while ago I was wondering if event time alignment is on the radar [1],
can you please clarify that?
There is a parallel discussion of adding it to the existing Kafka consumer
[2], could you please take a look and help figuring out the path forward?
Thanks,
Thomas
[1]
https://lists.apache.org/thread.html/c7336f259bbbb4c482de0580ecf2472a6b3e5a0fe2b7dd9258b159f2@%3Cdev.flink.apache.org%3E
[2] https://issues.apache.org/jira/browse/FLINK-12675
On Mon, Nov 4, 2019 at 5:48 AM Becket Qin <be...@gmail.com> wrote:
> Hi Steven and Thomas,
>
> Sorry about missing the update of FLIP-27.
>
> I am working on the implementation of FLIP-27 at this point. It is about
> 70% done. Right now I am integrating the source coordinator to the job
> master. Hopefully I can get the basics of Kafka connector work from end to
> end by this Sunday. Then I'll update FLIP-27 wiki page.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
> On Sat, Nov 2, 2019 at 8:18 AM Thomas Weise <th...@apache.org> wrote:
>
>> Is there any activity on FLIP-27 that would make it relevant for 1.10
>> release?
>>
>> Thanks Gary for the update, it provides excellent visibility on current
>> activity and what we can expect with the release.
>>
>>
>> On Fri, Nov 1, 2019 at 1:52 PM Steven Wu <st...@gmail.com> wrote:
>>
>>> Gary, FLIP-27 seems to get omitted in the 2nd update. below is the info
>>> from update #1.
>>>
>>> - FLIP-27: Refactor Source Interface [20]
>>> - FLIP accepted. Implementation is in progress.
>>>
>>>
>>>
>>> On Fri, Nov 1, 2019 at 7:01 AM Gary Yao <ga...@ververica.com> wrote:
>>>
>>> > Hi community,
>>> >
>>> > Because we have approximately one month of development time left until
>>> the
>>> > targeted Flink 1.10 feature freeze, we thought now would be a good
>>> time to
>>> > give another progress update. Below we have included a list of the
>>> ongoing
>>> > efforts that have made progress since our last release progress update
>>> > [1]. As
>>> > always, if you are working on something that is not included here, feel
>>> > free
>>> > to use this thread to share your progress.
>>> >
>>> > - Support Java 11 [2]
>>> > - Implementation is in progress (18/21 subtasks resolved)
>>> >
>>> > - Table API improvements
>>> > - Full Data Type Support in Planner [3]
>>> > - Implementing (1/8 subtasks resolved)
>>> > - FLIP-66 Support Time Attribute in SQL DDL [4]
>>> > - Implementation is in progress (1/7 subtasks resolved).
>>> > - FLIP-70 Support Computed Column [5]
>>> > - FLIP voting [6]
>>> > - FLIP-63 Rework Table Partition Support [7]
>>> > - Implementation is in progress (3/15 subtasks resolved).
>>> > - FLIP-51 Rework of Expression Design [8]
>>> > - Implementation is in progress (2/12 subtasks resolved).
>>> > - FLIP-64 Support for Temporary Objects in Table Module [9]
>>> > - Implementation is in progress
>>> >
>>> > - Hive compatibility completion (DDL/UDF) to support full Hive
>>> integration
>>> > - FLIP-57 Rework FunctionCatalog [10]
>>> > - Implementation is in progress (6/9 subtasks resolved)
>>> > - FLIP-68 Extend Core Table System with Modular Plugins [11]
>>> > - Implementation is in progress (2/8 subtasks resolved)
>>> >
>>> > - Finer grained resource management
>>> > - FLIP-49: Unified Memory Configuration for TaskExecutors [12]
>>> > - Implementation is in progress (6/10 subtasks resolved)
>>> > - FLIP-53: Fine Grained Operator Resource Management [13]
>>> > - Implementation is in progress (1/9 subtasks resolved)
>>> >
>>> > - Finish scheduler re-architecture [14]
>>> > - Integration tests are being enabled for new scheduler
>>> >
>>> > - Executor/Client refactoring [15]
>>> > - FLIP-81: Executor-related new ConfigOptions [16]
>>> > - done
>>> > - FLIP-73: Introducing Executors for job submission [17]
>>> > - Implementation is in progress
>>> >
>>> > - FLIP-36 Support Interactive Programming [18]
>>> > - Is built on top of FLIP-67 [19], which has been accepted
>>> > - Implementation in progress
>>> >
>>> > - FLIP-58: Flink Python User-Defined Stateless Function for Table [20]
>>> > - Implementation is in progress (12/22 subtask resolved)
>>> > - FLIP-50: Spill-able Heap Keyed State Backend [21]
>>> > - Implementation is in progress (2/11 subtasks resolved)
>>> >
>>> > - RocksDB Backend Memory Control [22]
>>> > - FLIP for resource management on state backend will be opened soon
>>> > - Write Buffer Manager will be backported to FRocksDB due to
>>> > performance regression [23] in new RocksDB versions
>>> >
>>> > - Unaligned Checkpoints
>>> > - FLIP-76 [24] was published and received positive feedback
>>> > - Implementation is in progress
>>> >
>>> > - Separate framework and user class loader in per-job mode [25]
>>> > - First PR is almost done. Remaining PRs will be ready next week
>>> >
>>> > - Active Kubernetes Integration [26]
>>> > - Implementation is in progress (6/11 in review, 3/11 in progress,
>>> > 2/11 todo)
>>> >
>>> > - FLIP-39 Flink ML pipeline and ML libs [27]
>>> > - A few abstract ML classes have been merged (FLINK-13339,
>>> > FLINK-13513)
>>> > - Starting review of algorithms
>>> >
>>> > Again, the feature freeze is targeted to be at the end of November.
>>> Please
>>> > make sure that all important work threads can be completed until that
>>> date.
>>> > Feel free to use this thread to communicate any concerns about features
>>> > that
>>> > might not be finished until then. We will send another announcement
>>> later
>>> > in
>>> > the release cycle to make the date of the feature freeze official.
>>> >
>>> > Best,
>>> > Yu & Gary
>>> >
>>> > [1] https://s.apache.org/wc0dc
>>> > [2] https://issues.apache.org/jira/browse/FLINK-10725
>>> > [3] https://issues.apache.org/jira/browse/FLINK-14079
>>> > [4]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL
>>> > [5]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design
>>> > [6]
>>> >
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-70-Flink-SQL-Computed-Column-Design-td34385.html
>>> > [7]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
>>> > [8]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
>>> > [9]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module
>>> > [10]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog
>>> > [11]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
>>> > [12]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
>>> > [13]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
>>> > [14] https://issues.apache.org/jira/browse/FLINK-10429
>>> > [15]
>>> >
>>> https://lists.apache.org/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@%3Cdev.flink.apache.org%3E
>>> > [16]
>>> >
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524
>>> > [17]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission
>>> > [18]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
>>> > [19]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
>>> > [20]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>>> > [21]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
>>> > [22] https://issues.apache.org/jira/browse/FLINK-7289
>>> > [23] https://github.com/facebook/rocksdb/issues/5774
>>> > [24]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
>>> > [25] https://issues.apache.org/jira/browse/FLINK-13993
>>> > [26] https://issues.apache.org/jira/browse/FLINK-9953
>>> > [27]
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
>>> >
>>>
>>
Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2
Posted by Becket Qin <be...@gmail.com>.
Hi Steven and Thomas,
Sorry about missing the update of FLIP-27.
I am working on the implementation of FLIP-27 at this point. It is about
70% done. Right now I am integrating the source coordinator to the job
master. Hopefully I can get the basics of Kafka connector work from end to
end by this Sunday. Then I'll update FLIP-27 wiki page.
Thanks,
Jiangjie (Becket) Qin
On Sat, Nov 2, 2019 at 8:18 AM Thomas Weise <th...@apache.org> wrote:
> Is there any activity on FLIP-27 that would make it relevant for 1.10
> release?
>
> Thanks Gary for the update, it provides excellent visibility on current
> activity and what we can expect with the release.
>
>
> On Fri, Nov 1, 2019 at 1:52 PM Steven Wu <st...@gmail.com> wrote:
>
>> Gary, FLIP-27 seems to get omitted in the 2nd update. below is the info
>> from update #1.
>>
>> - FLIP-27: Refactor Source Interface [20]
>> - FLIP accepted. Implementation is in progress.
>>
>>
>>
>> On Fri, Nov 1, 2019 at 7:01 AM Gary Yao <ga...@ververica.com> wrote:
>>
>> > Hi community,
>> >
>> > Because we have approximately one month of development time left until
>> the
>> > targeted Flink 1.10 feature freeze, we thought now would be a good time
>> to
>> > give another progress update. Below we have included a list of the
>> ongoing
>> > efforts that have made progress since our last release progress update
>> > [1]. As
>> > always, if you are working on something that is not included here, feel
>> > free
>> > to use this thread to share your progress.
>> >
>> > - Support Java 11 [2]
>> > - Implementation is in progress (18/21 subtasks resolved)
>> >
>> > - Table API improvements
>> > - Full Data Type Support in Planner [3]
>> > - Implementing (1/8 subtasks resolved)
>> > - FLIP-66 Support Time Attribute in SQL DDL [4]
>> > - Implementation is in progress (1/7 subtasks resolved).
>> > - FLIP-70 Support Computed Column [5]
>> > - FLIP voting [6]
>> > - FLIP-63 Rework Table Partition Support [7]
>> > - Implementation is in progress (3/15 subtasks resolved).
>> > - FLIP-51 Rework of Expression Design [8]
>> > - Implementation is in progress (2/12 subtasks resolved).
>> > - FLIP-64 Support for Temporary Objects in Table Module [9]
>> > - Implementation is in progress
>> >
>> > - Hive compatibility completion (DDL/UDF) to support full Hive
>> integration
>> > - FLIP-57 Rework FunctionCatalog [10]
>> > - Implementation is in progress (6/9 subtasks resolved)
>> > - FLIP-68 Extend Core Table System with Modular Plugins [11]
>> > - Implementation is in progress (2/8 subtasks resolved)
>> >
>> > - Finer grained resource management
>> > - FLIP-49: Unified Memory Configuration for TaskExecutors [12]
>> > - Implementation is in progress (6/10 subtasks resolved)
>> > - FLIP-53: Fine Grained Operator Resource Management [13]
>> > - Implementation is in progress (1/9 subtasks resolved)
>> >
>> > - Finish scheduler re-architecture [14]
>> > - Integration tests are being enabled for new scheduler
>> >
>> > - Executor/Client refactoring [15]
>> > - FLIP-81: Executor-related new ConfigOptions [16]
>> > - done
>> > - FLIP-73: Introducing Executors for job submission [17]
>> > - Implementation is in progress
>> >
>> > - FLIP-36 Support Interactive Programming [18]
>> > - Is built on top of FLIP-67 [19], which has been accepted
>> > - Implementation in progress
>> >
>> > - FLIP-58: Flink Python User-Defined Stateless Function for Table [20]
>> > - Implementation is in progress (12/22 subtask resolved)
>> > - FLIP-50: Spill-able Heap Keyed State Backend [21]
>> > - Implementation is in progress (2/11 subtasks resolved)
>> >
>> > - RocksDB Backend Memory Control [22]
>> > - FLIP for resource management on state backend will be opened soon
>> > - Write Buffer Manager will be backported to FRocksDB due to
>> > performance regression [23] in new RocksDB versions
>> >
>> > - Unaligned Checkpoints
>> > - FLIP-76 [24] was published and received positive feedback
>> > - Implementation is in progress
>> >
>> > - Separate framework and user class loader in per-job mode [25]
>> > - First PR is almost done. Remaining PRs will be ready next week
>> >
>> > - Active Kubernetes Integration [26]
>> > - Implementation is in progress (6/11 in review, 3/11 in progress,
>> > 2/11 todo)
>> >
>> > - FLIP-39 Flink ML pipeline and ML libs [27]
>> > - A few abstract ML classes have been merged (FLINK-13339,
>> > FLINK-13513)
>> > - Starting review of algorithms
>> >
>> > Again, the feature freeze is targeted to be at the end of November.
>> Please
>> > make sure that all important work threads can be completed until that
>> date.
>> > Feel free to use this thread to communicate any concerns about features
>> > that
>> > might not be finished until then. We will send another announcement
>> later
>> > in
>> > the release cycle to make the date of the feature freeze official.
>> >
>> > Best,
>> > Yu & Gary
>> >
>> > [1] https://s.apache.org/wc0dc
>> > [2] https://issues.apache.org/jira/browse/FLINK-10725
>> > [3] https://issues.apache.org/jira/browse/FLINK-14079
>> > [4]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL
>> > [5]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design
>> > [6]
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-70-Flink-SQL-Computed-Column-Design-td34385.html
>> > [7]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
>> > [8]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
>> > [9]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module
>> > [10]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog
>> > [11]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
>> > [12]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
>> > [13]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
>> > [14] https://issues.apache.org/jira/browse/FLINK-10429
>> > [15]
>> >
>> https://lists.apache.org/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@%3Cdev.flink.apache.org%3E
>> > [16]
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524
>> > [17]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission
>> > [18]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
>> > [19]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
>> > [20]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>> > [21]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
>> > [22] https://issues.apache.org/jira/browse/FLINK-7289
>> > [23] https://github.com/facebook/rocksdb/issues/5774
>> > [24]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
>> > [25] https://issues.apache.org/jira/browse/FLINK-13993
>> > [26] https://issues.apache.org/jira/browse/FLINK-9953
>> > [27]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
>> >
>>
>
Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2
Posted by Becket Qin <be...@gmail.com>.
Hi Steven and Thomas,
Sorry about missing the update of FLIP-27.
I am working on the implementation of FLIP-27 at this point. It is about
70% done. Right now I am integrating the source coordinator to the job
master. Hopefully I can get the basics of Kafka connector work from end to
end by this Sunday. Then I'll update FLIP-27 wiki page.
Thanks,
Jiangjie (Becket) Qin
On Sat, Nov 2, 2019 at 8:18 AM Thomas Weise <th...@apache.org> wrote:
> Is there any activity on FLIP-27 that would make it relevant for 1.10
> release?
>
> Thanks Gary for the update, it provides excellent visibility on current
> activity and what we can expect with the release.
>
>
> On Fri, Nov 1, 2019 at 1:52 PM Steven Wu <st...@gmail.com> wrote:
>
>> Gary, FLIP-27 seems to get omitted in the 2nd update. below is the info
>> from update #1.
>>
>> - FLIP-27: Refactor Source Interface [20]
>> - FLIP accepted. Implementation is in progress.
>>
>>
>>
>> On Fri, Nov 1, 2019 at 7:01 AM Gary Yao <ga...@ververica.com> wrote:
>>
>> > Hi community,
>> >
>> > Because we have approximately one month of development time left until
>> the
>> > targeted Flink 1.10 feature freeze, we thought now would be a good time
>> to
>> > give another progress update. Below we have included a list of the
>> ongoing
>> > efforts that have made progress since our last release progress update
>> > [1]. As
>> > always, if you are working on something that is not included here, feel
>> > free
>> > to use this thread to share your progress.
>> >
>> > - Support Java 11 [2]
>> > - Implementation is in progress (18/21 subtasks resolved)
>> >
>> > - Table API improvements
>> > - Full Data Type Support in Planner [3]
>> > - Implementing (1/8 subtasks resolved)
>> > - FLIP-66 Support Time Attribute in SQL DDL [4]
>> > - Implementation is in progress (1/7 subtasks resolved).
>> > - FLIP-70 Support Computed Column [5]
>> > - FLIP voting [6]
>> > - FLIP-63 Rework Table Partition Support [7]
>> > - Implementation is in progress (3/15 subtasks resolved).
>> > - FLIP-51 Rework of Expression Design [8]
>> > - Implementation is in progress (2/12 subtasks resolved).
>> > - FLIP-64 Support for Temporary Objects in Table Module [9]
>> > - Implementation is in progress
>> >
>> > - Hive compatibility completion (DDL/UDF) to support full Hive
>> integration
>> > - FLIP-57 Rework FunctionCatalog [10]
>> > - Implementation is in progress (6/9 subtasks resolved)
>> > - FLIP-68 Extend Core Table System with Modular Plugins [11]
>> > - Implementation is in progress (2/8 subtasks resolved)
>> >
>> > - Finer grained resource management
>> > - FLIP-49: Unified Memory Configuration for TaskExecutors [12]
>> > - Implementation is in progress (6/10 subtasks resolved)
>> > - FLIP-53: Fine Grained Operator Resource Management [13]
>> > - Implementation is in progress (1/9 subtasks resolved)
>> >
>> > - Finish scheduler re-architecture [14]
>> > - Integration tests are being enabled for new scheduler
>> >
>> > - Executor/Client refactoring [15]
>> > - FLIP-81: Executor-related new ConfigOptions [16]
>> > - done
>> > - FLIP-73: Introducing Executors for job submission [17]
>> > - Implementation is in progress
>> >
>> > - FLIP-36 Support Interactive Programming [18]
>> > - Is built on top of FLIP-67 [19], which has been accepted
>> > - Implementation in progress
>> >
>> > - FLIP-58: Flink Python User-Defined Stateless Function for Table [20]
>> > - Implementation is in progress (12/22 subtask resolved)
>> > - FLIP-50: Spill-able Heap Keyed State Backend [21]
>> > - Implementation is in progress (2/11 subtasks resolved)
>> >
>> > - RocksDB Backend Memory Control [22]
>> > - FLIP for resource management on state backend will be opened soon
>> > - Write Buffer Manager will be backported to FRocksDB due to
>> > performance regression [23] in new RocksDB versions
>> >
>> > - Unaligned Checkpoints
>> > - FLIP-76 [24] was published and received positive feedback
>> > - Implementation is in progress
>> >
>> > - Separate framework and user class loader in per-job mode [25]
>> > - First PR is almost done. Remaining PRs will be ready next week
>> >
>> > - Active Kubernetes Integration [26]
>> > - Implementation is in progress (6/11 in review, 3/11 in progress,
>> > 2/11 todo)
>> >
>> > - FLIP-39 Flink ML pipeline and ML libs [27]
>> > - A few abstract ML classes have been merged (FLINK-13339,
>> > FLINK-13513)
>> > - Starting review of algorithms
>> >
>> > Again, the feature freeze is targeted to be at the end of November.
>> Please
>> > make sure that all important work threads can be completed until that
>> date.
>> > Feel free to use this thread to communicate any concerns about features
>> > that
>> > might not be finished until then. We will send another announcement
>> later
>> > in
>> > the release cycle to make the date of the feature freeze official.
>> >
>> > Best,
>> > Yu & Gary
>> >
>> > [1] https://s.apache.org/wc0dc
>> > [2] https://issues.apache.org/jira/browse/FLINK-10725
>> > [3] https://issues.apache.org/jira/browse/FLINK-14079
>> > [4]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL
>> > [5]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design
>> > [6]
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-70-Flink-SQL-Computed-Column-Design-td34385.html
>> > [7]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
>> > [8]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
>> > [9]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module
>> > [10]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog
>> > [11]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
>> > [12]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
>> > [13]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
>> > [14] https://issues.apache.org/jira/browse/FLINK-10429
>> > [15]
>> >
>> https://lists.apache.org/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@%3Cdev.flink.apache.org%3E
>> > [16]
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524
>> > [17]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission
>> > [18]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
>> > [19]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
>> > [20]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>> > [21]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
>> > [22] https://issues.apache.org/jira/browse/FLINK-7289
>> > [23] https://github.com/facebook/rocksdb/issues/5774
>> > [24]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
>> > [25] https://issues.apache.org/jira/browse/FLINK-13993
>> > [26] https://issues.apache.org/jira/browse/FLINK-9953
>> > [27]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
>> >
>>
>
Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2
Posted by Thomas Weise <th...@apache.org>.
Is there any activity on FLIP-27 that would make it relevant for 1.10
release?
Thanks Gary for the update, it provides excellent visibility on current
activity and what we can expect with the release.
On Fri, Nov 1, 2019 at 1:52 PM Steven Wu <st...@gmail.com> wrote:
> Gary, FLIP-27 seems to get omitted in the 2nd update. below is the info
> from update #1.
>
> - FLIP-27: Refactor Source Interface [20]
> - FLIP accepted. Implementation is in progress.
>
>
>
> On Fri, Nov 1, 2019 at 7:01 AM Gary Yao <ga...@ververica.com> wrote:
>
> > Hi community,
> >
> > Because we have approximately one month of development time left until
> the
> > targeted Flink 1.10 feature freeze, we thought now would be a good time
> to
> > give another progress update. Below we have included a list of the
> ongoing
> > efforts that have made progress since our last release progress update
> > [1]. As
> > always, if you are working on something that is not included here, feel
> > free
> > to use this thread to share your progress.
> >
> > - Support Java 11 [2]
> > - Implementation is in progress (18/21 subtasks resolved)
> >
> > - Table API improvements
> > - Full Data Type Support in Planner [3]
> > - Implementing (1/8 subtasks resolved)
> > - FLIP-66 Support Time Attribute in SQL DDL [4]
> > - Implementation is in progress (1/7 subtasks resolved).
> > - FLIP-70 Support Computed Column [5]
> > - FLIP voting [6]
> > - FLIP-63 Rework Table Partition Support [7]
> > - Implementation is in progress (3/15 subtasks resolved).
> > - FLIP-51 Rework of Expression Design [8]
> > - Implementation is in progress (2/12 subtasks resolved).
> > - FLIP-64 Support for Temporary Objects in Table Module [9]
> > - Implementation is in progress
> >
> > - Hive compatibility completion (DDL/UDF) to support full Hive
> integration
> > - FLIP-57 Rework FunctionCatalog [10]
> > - Implementation is in progress (6/9 subtasks resolved)
> > - FLIP-68 Extend Core Table System with Modular Plugins [11]
> > - Implementation is in progress (2/8 subtasks resolved)
> >
> > - Finer grained resource management
> > - FLIP-49: Unified Memory Configuration for TaskExecutors [12]
> > - Implementation is in progress (6/10 subtasks resolved)
> > - FLIP-53: Fine Grained Operator Resource Management [13]
> > - Implementation is in progress (1/9 subtasks resolved)
> >
> > - Finish scheduler re-architecture [14]
> > - Integration tests are being enabled for new scheduler
> >
> > - Executor/Client refactoring [15]
> > - FLIP-81: Executor-related new ConfigOptions [16]
> > - done
> > - FLIP-73: Introducing Executors for job submission [17]
> > - Implementation is in progress
> >
> > - FLIP-36 Support Interactive Programming [18]
> > - Is built on top of FLIP-67 [19], which has been accepted
> > - Implementation in progress
> >
> > - FLIP-58: Flink Python User-Defined Stateless Function for Table [20]
> > - Implementation is in progress (12/22 subtask resolved)
> > - FLIP-50: Spill-able Heap Keyed State Backend [21]
> > - Implementation is in progress (2/11 subtasks resolved)
> >
> > - RocksDB Backend Memory Control [22]
> > - FLIP for resource management on state backend will be opened soon
> > - Write Buffer Manager will be backported to FRocksDB due to
> > performance regression [23] in new RocksDB versions
> >
> > - Unaligned Checkpoints
> > - FLIP-76 [24] was published and received positive feedback
> > - Implementation is in progress
> >
> > - Separate framework and user class loader in per-job mode [25]
> > - First PR is almost done. Remaining PRs will be ready next week
> >
> > - Active Kubernetes Integration [26]
> > - Implementation is in progress (6/11 in review, 3/11 in progress,
> > 2/11 todo)
> >
> > - FLIP-39 Flink ML pipeline and ML libs [27]
> > - A few abstract ML classes have been merged (FLINK-13339,
> > FLINK-13513)
> > - Starting review of algorithms
> >
> > Again, the feature freeze is targeted to be at the end of November.
> Please
> > make sure that all important work threads can be completed until that
> date.
> > Feel free to use this thread to communicate any concerns about features
> > that
> > might not be finished until then. We will send another announcement later
> > in
> > the release cycle to make the date of the feature freeze official.
> >
> > Best,
> > Yu & Gary
> >
> > [1] https://s.apache.org/wc0dc
> > [2] https://issues.apache.org/jira/browse/FLINK-10725
> > [3] https://issues.apache.org/jira/browse/FLINK-14079
> > [4]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL
> > [5]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design
> > [6]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-70-Flink-SQL-Computed-Column-Design-td34385.html
> > [7]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
> > [8]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
> > [9]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module
> > [10]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog
> > [11]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
> > [12]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
> > [13]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
> > [14] https://issues.apache.org/jira/browse/FLINK-10429
> > [15]
> >
> https://lists.apache.org/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@%3Cdev.flink.apache.org%3E
> > [16]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524
> > [17]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission
> > [18]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> > [19]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
> > [20]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> > [21]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
> > [22] https://issues.apache.org/jira/browse/FLINK-7289
> > [23] https://github.com/facebook/rocksdb/issues/5774
> > [24]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
> > [25] https://issues.apache.org/jira/browse/FLINK-13993
> > [26] https://issues.apache.org/jira/browse/FLINK-9953
> > [27]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
> >
>
Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2
Posted by Thomas Weise <th...@apache.org>.
Is there any activity on FLIP-27 that would make it relevant for 1.10
release?
Thanks Gary for the update, it provides excellent visibility on current
activity and what we can expect with the release.
On Fri, Nov 1, 2019 at 1:52 PM Steven Wu <st...@gmail.com> wrote:
> Gary, FLIP-27 seems to get omitted in the 2nd update. below is the info
> from update #1.
>
> - FLIP-27: Refactor Source Interface [20]
> - FLIP accepted. Implementation is in progress.
>
>
>
> On Fri, Nov 1, 2019 at 7:01 AM Gary Yao <ga...@ververica.com> wrote:
>
> > Hi community,
> >
> > Because we have approximately one month of development time left until
> the
> > targeted Flink 1.10 feature freeze, we thought now would be a good time
> to
> > give another progress update. Below we have included a list of the
> ongoing
> > efforts that have made progress since our last release progress update
> > [1]. As
> > always, if you are working on something that is not included here, feel
> > free
> > to use this thread to share your progress.
> >
> > - Support Java 11 [2]
> > - Implementation is in progress (18/21 subtasks resolved)
> >
> > - Table API improvements
> > - Full Data Type Support in Planner [3]
> > - Implementing (1/8 subtasks resolved)
> > - FLIP-66 Support Time Attribute in SQL DDL [4]
> > - Implementation is in progress (1/7 subtasks resolved).
> > - FLIP-70 Support Computed Column [5]
> > - FLIP voting [6]
> > - FLIP-63 Rework Table Partition Support [7]
> > - Implementation is in progress (3/15 subtasks resolved).
> > - FLIP-51 Rework of Expression Design [8]
> > - Implementation is in progress (2/12 subtasks resolved).
> > - FLIP-64 Support for Temporary Objects in Table Module [9]
> > - Implementation is in progress
> >
> > - Hive compatibility completion (DDL/UDF) to support full Hive
> integration
> > - FLIP-57 Rework FunctionCatalog [10]
> > - Implementation is in progress (6/9 subtasks resolved)
> > - FLIP-68 Extend Core Table System with Modular Plugins [11]
> > - Implementation is in progress (2/8 subtasks resolved)
> >
> > - Finer grained resource management
> > - FLIP-49: Unified Memory Configuration for TaskExecutors [12]
> > - Implementation is in progress (6/10 subtasks resolved)
> > - FLIP-53: Fine Grained Operator Resource Management [13]
> > - Implementation is in progress (1/9 subtasks resolved)
> >
> > - Finish scheduler re-architecture [14]
> > - Integration tests are being enabled for new scheduler
> >
> > - Executor/Client refactoring [15]
> > - FLIP-81: Executor-related new ConfigOptions [16]
> > - done
> > - FLIP-73: Introducing Executors for job submission [17]
> > - Implementation is in progress
> >
> > - FLIP-36 Support Interactive Programming [18]
> > - Is built on top of FLIP-67 [19], which has been accepted
> > - Implementation in progress
> >
> > - FLIP-58: Flink Python User-Defined Stateless Function for Table [20]
> > - Implementation is in progress (12/22 subtask resolved)
> > - FLIP-50: Spill-able Heap Keyed State Backend [21]
> > - Implementation is in progress (2/11 subtasks resolved)
> >
> > - RocksDB Backend Memory Control [22]
> > - FLIP for resource management on state backend will be opened soon
> > - Write Buffer Manager will be backported to FRocksDB due to
> > performance regression [23] in new RocksDB versions
> >
> > - Unaligned Checkpoints
> > - FLIP-76 [24] was published and received positive feedback
> > - Implementation is in progress
> >
> > - Separate framework and user class loader in per-job mode [25]
> > - First PR is almost done. Remaining PRs will be ready next week
> >
> > - Active Kubernetes Integration [26]
> > - Implementation is in progress (6/11 in review, 3/11 in progress,
> > 2/11 todo)
> >
> > - FLIP-39 Flink ML pipeline and ML libs [27]
> > - A few abstract ML classes have been merged (FLINK-13339,
> > FLINK-13513)
> > - Starting review of algorithms
> >
> > Again, the feature freeze is targeted to be at the end of November.
> Please
> > make sure that all important work threads can be completed until that
> date.
> > Feel free to use this thread to communicate any concerns about features
> > that
> > might not be finished until then. We will send another announcement later
> > in
> > the release cycle to make the date of the feature freeze official.
> >
> > Best,
> > Yu & Gary
> >
> > [1] https://s.apache.org/wc0dc
> > [2] https://issues.apache.org/jira/browse/FLINK-10725
> > [3] https://issues.apache.org/jira/browse/FLINK-14079
> > [4]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL
> > [5]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design
> > [6]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-70-Flink-SQL-Computed-Column-Design-td34385.html
> > [7]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
> > [8]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
> > [9]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module
> > [10]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog
> > [11]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
> > [12]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
> > [13]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
> > [14] https://issues.apache.org/jira/browse/FLINK-10429
> > [15]
> >
> https://lists.apache.org/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@%3Cdev.flink.apache.org%3E
> > [16]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524
> > [17]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission
> > [18]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> > [19]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
> > [20]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> > [21]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
> > [22] https://issues.apache.org/jira/browse/FLINK-7289
> > [23] https://github.com/facebook/rocksdb/issues/5774
> > [24]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
> > [25] https://issues.apache.org/jira/browse/FLINK-13993
> > [26] https://issues.apache.org/jira/browse/FLINK-9953
> > [27]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
> >
>
Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2
Posted by Steven Wu <st...@gmail.com>.
Gary, FLIP-27 seems to get omitted in the 2nd update. below is the info
from update #1.
- FLIP-27: Refactor Source Interface [20]
- FLIP accepted. Implementation is in progress.
On Fri, Nov 1, 2019 at 7:01 AM Gary Yao <ga...@ververica.com> wrote:
> Hi community,
>
> Because we have approximately one month of development time left until the
> targeted Flink 1.10 feature freeze, we thought now would be a good time to
> give another progress update. Below we have included a list of the ongoing
> efforts that have made progress since our last release progress update
> [1]. As
> always, if you are working on something that is not included here, feel
> free
> to use this thread to share your progress.
>
> - Support Java 11 [2]
> - Implementation is in progress (18/21 subtasks resolved)
>
> - Table API improvements
> - Full Data Type Support in Planner [3]
> - Implementing (1/8 subtasks resolved)
> - FLIP-66 Support Time Attribute in SQL DDL [4]
> - Implementation is in progress (1/7 subtasks resolved).
> - FLIP-70 Support Computed Column [5]
> - FLIP voting [6]
> - FLIP-63 Rework Table Partition Support [7]
> - Implementation is in progress (3/15 subtasks resolved).
> - FLIP-51 Rework of Expression Design [8]
> - Implementation is in progress (2/12 subtasks resolved).
> - FLIP-64 Support for Temporary Objects in Table Module [9]
> - Implementation is in progress
>
> - Hive compatibility completion (DDL/UDF) to support full Hive integration
> - FLIP-57 Rework FunctionCatalog [10]
> - Implementation is in progress (6/9 subtasks resolved)
> - FLIP-68 Extend Core Table System with Modular Plugins [11]
> - Implementation is in progress (2/8 subtasks resolved)
>
> - Finer grained resource management
> - FLIP-49: Unified Memory Configuration for TaskExecutors [12]
> - Implementation is in progress (6/10 subtasks resolved)
> - FLIP-53: Fine Grained Operator Resource Management [13]
> - Implementation is in progress (1/9 subtasks resolved)
>
> - Finish scheduler re-architecture [14]
> - Integration tests are being enabled for new scheduler
>
> - Executor/Client refactoring [15]
> - FLIP-81: Executor-related new ConfigOptions [16]
> - done
> - FLIP-73: Introducing Executors for job submission [17]
> - Implementation is in progress
>
> - FLIP-36 Support Interactive Programming [18]
> - Is built on top of FLIP-67 [19], which has been accepted
> - Implementation in progress
>
> - FLIP-58: Flink Python User-Defined Stateless Function for Table [20]
> - Implementation is in progress (12/22 subtask resolved)
> - FLIP-50: Spill-able Heap Keyed State Backend [21]
> - Implementation is in progress (2/11 subtasks resolved)
>
> - RocksDB Backend Memory Control [22]
> - FLIP for resource management on state backend will be opened soon
> - Write Buffer Manager will be backported to FRocksDB due to
> performance regression [23] in new RocksDB versions
>
> - Unaligned Checkpoints
> - FLIP-76 [24] was published and received positive feedback
> - Implementation is in progress
>
> - Separate framework and user class loader in per-job mode [25]
> - First PR is almost done. Remaining PRs will be ready next week
>
> - Active Kubernetes Integration [26]
> - Implementation is in progress (6/11 in review, 3/11 in progress,
> 2/11 todo)
>
> - FLIP-39 Flink ML pipeline and ML libs [27]
> - A few abstract ML classes have been merged (FLINK-13339,
> FLINK-13513)
> - Starting review of algorithms
>
> Again, the feature freeze is targeted to be at the end of November. Please
> make sure that all important work threads can be completed until that date.
> Feel free to use this thread to communicate any concerns about features
> that
> might not be finished until then. We will send another announcement later
> in
> the release cycle to make the date of the feature freeze official.
>
> Best,
> Yu & Gary
>
> [1] https://s.apache.org/wc0dc
> [2] https://issues.apache.org/jira/browse/FLINK-10725
> [3] https://issues.apache.org/jira/browse/FLINK-14079
> [4]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL
> [5]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design
> [6]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-70-Flink-SQL-Computed-Column-Design-td34385.html
> [7]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
> [8]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
> [9]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module
> [10]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog
> [11]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
> [12]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
> [13]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
> [14] https://issues.apache.org/jira/browse/FLINK-10429
> [15]
> https://lists.apache.org/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@%3Cdev.flink.apache.org%3E
> [16]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524
> [17]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission
> [18]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> [19]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
> [20]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> [21]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
> [22] https://issues.apache.org/jira/browse/FLINK-7289
> [23] https://github.com/facebook/rocksdb/issues/5774
> [24]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
> [25] https://issues.apache.org/jira/browse/FLINK-13993
> [26] https://issues.apache.org/jira/browse/FLINK-9953
> [27]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
>
Re: [ANNOUNCE] Progress of Apache Flink 1.10 #2
Posted by Steven Wu <st...@gmail.com>.
Gary, FLIP-27 seems to get omitted in the 2nd update. below is the info
from update #1.
- FLIP-27: Refactor Source Interface [20]
- FLIP accepted. Implementation is in progress.
On Fri, Nov 1, 2019 at 7:01 AM Gary Yao <ga...@ververica.com> wrote:
> Hi community,
>
> Because we have approximately one month of development time left until the
> targeted Flink 1.10 feature freeze, we thought now would be a good time to
> give another progress update. Below we have included a list of the ongoing
> efforts that have made progress since our last release progress update
> [1]. As
> always, if you are working on something that is not included here, feel
> free
> to use this thread to share your progress.
>
> - Support Java 11 [2]
> - Implementation is in progress (18/21 subtasks resolved)
>
> - Table API improvements
> - Full Data Type Support in Planner [3]
> - Implementing (1/8 subtasks resolved)
> - FLIP-66 Support Time Attribute in SQL DDL [4]
> - Implementation is in progress (1/7 subtasks resolved).
> - FLIP-70 Support Computed Column [5]
> - FLIP voting [6]
> - FLIP-63 Rework Table Partition Support [7]
> - Implementation is in progress (3/15 subtasks resolved).
> - FLIP-51 Rework of Expression Design [8]
> - Implementation is in progress (2/12 subtasks resolved).
> - FLIP-64 Support for Temporary Objects in Table Module [9]
> - Implementation is in progress
>
> - Hive compatibility completion (DDL/UDF) to support full Hive integration
> - FLIP-57 Rework FunctionCatalog [10]
> - Implementation is in progress (6/9 subtasks resolved)
> - FLIP-68 Extend Core Table System with Modular Plugins [11]
> - Implementation is in progress (2/8 subtasks resolved)
>
> - Finer grained resource management
> - FLIP-49: Unified Memory Configuration for TaskExecutors [12]
> - Implementation is in progress (6/10 subtasks resolved)
> - FLIP-53: Fine Grained Operator Resource Management [13]
> - Implementation is in progress (1/9 subtasks resolved)
>
> - Finish scheduler re-architecture [14]
> - Integration tests are being enabled for new scheduler
>
> - Executor/Client refactoring [15]
> - FLIP-81: Executor-related new ConfigOptions [16]
> - done
> - FLIP-73: Introducing Executors for job submission [17]
> - Implementation is in progress
>
> - FLIP-36 Support Interactive Programming [18]
> - Is built on top of FLIP-67 [19], which has been accepted
> - Implementation in progress
>
> - FLIP-58: Flink Python User-Defined Stateless Function for Table [20]
> - Implementation is in progress (12/22 subtask resolved)
> - FLIP-50: Spill-able Heap Keyed State Backend [21]
> - Implementation is in progress (2/11 subtasks resolved)
>
> - RocksDB Backend Memory Control [22]
> - FLIP for resource management on state backend will be opened soon
> - Write Buffer Manager will be backported to FRocksDB due to
> performance regression [23] in new RocksDB versions
>
> - Unaligned Checkpoints
> - FLIP-76 [24] was published and received positive feedback
> - Implementation is in progress
>
> - Separate framework and user class loader in per-job mode [25]
> - First PR is almost done. Remaining PRs will be ready next week
>
> - Active Kubernetes Integration [26]
> - Implementation is in progress (6/11 in review, 3/11 in progress,
> 2/11 todo)
>
> - FLIP-39 Flink ML pipeline and ML libs [27]
> - A few abstract ML classes have been merged (FLINK-13339,
> FLINK-13513)
> - Starting review of algorithms
>
> Again, the feature freeze is targeted to be at the end of November. Please
> make sure that all important work threads can be completed until that date.
> Feel free to use this thread to communicate any concerns about features
> that
> might not be finished until then. We will send another announcement later
> in
> the release cycle to make the date of the feature freeze official.
>
> Best,
> Yu & Gary
>
> [1] https://s.apache.org/wc0dc
> [2] https://issues.apache.org/jira/browse/FLINK-10725
> [3] https://issues.apache.org/jira/browse/FLINK-14079
> [4]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL
> [5]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design
> [6]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-70-Flink-SQL-Computed-Column-Design-td34385.html
> [7]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
> [8]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
> [9]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module
> [10]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog
> [11]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
> [12]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
> [13]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
> [14] https://issues.apache.org/jira/browse/FLINK-10429
> [15]
> https://lists.apache.org/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@%3Cdev.flink.apache.org%3E
> [16]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524
> [17]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission
> [18]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> [19]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
> [20]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> [21]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
> [22] https://issues.apache.org/jira/browse/FLINK-7289
> [23] https://github.com/facebook/rocksdb/issues/5774
> [24]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
> [25] https://issues.apache.org/jira/browse/FLINK-13993
> [26] https://issues.apache.org/jira/browse/FLINK-9953
> [27]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
>