You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Wencong Liu <li...@163.com> on 2023/09/01 04:00:01 UTC

[DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

Hi devs,

I would like to start a discussion on FLIP-357: Deprecate Iteration API of DataStream [1].

Currently, the Iteration API of DataStream is incomplete. For instance, it lacks support
for iteration in sync mode and exactly once semantics. Additionally, it does not offer the
ability to set iteration termination conditions. As a result, it's hard for developers to
build an iteration pipeline by DataStream in the practical applications such as machine learning.

FLIP-176: Unified Iteration to Support Algorithms [2] has introduced a unified iteration library
in the Flink ML repository. This library addresses all the issues present in the Iteration API of
DataStream and could provide solution for all the iteration use-cases. However, maintaining two
separate implementations of iteration in both the Flink repository and the Flink ML repository
would introduce unnecessary complexity and make it difficult to maintain the Iteration API.

As such I propose deprecating the Iteration API of DataStream and removing it completely in the next
major version. In the future, if other modules in the Flink repository require the use of the
Iteration API, we can consider extracting all Iteration implementations from the Flink ML repository
into an independent module.

Looking forward to your feedback.


[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%3A+Deprecate+Iteration+API+of+DataStream
[2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=184615300

Best regards,

Wencong Liu

Re: [DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

Posted by Stefan Richter <sr...@confluent.io.INVALID>.
+1

Thanks,
Stefan


> On 5. Sep 2023, at 10:45, David Morávek <dm...@apache.org> wrote:
> 
> +1 since there is an alternative, more complete implementation available
> 
> Best,
> D.
> 
> On Sat, Sep 2, 2023 at 12:07 AM David Anderson <danderson@apache.org <ma...@apache.org>> wrote:
> 
>> +1
>> 
>> Keeping the legacy implementation in place is confusing and encourages
>> adoption of something that really shouldn't be used.
>> 
>> Thanks for driving this,
>> David
>> 
>> On Fri, Sep 1, 2023 at 8:45 AM Jing Ge <ji...@ververica.com.invalid> wrote:
>>> 
>>> Hi Wencong,
>>> 
>>> Thanks for your clarification! +1
>>> 
>>> Best regards,
>>> Jing
>>> 
>>> On Fri, Sep 1, 2023 at 12:36 PM Wencong Liu <li...@163.com>
>> wrote:
>>> 
>>>> Hi Jing,
>>>> 
>>>> 
>>>> Thanks for your reply!
>>>> 
>>>> 
>>>>> Or the "independent module extraction" mentioned in the FLIP does
>> mean an
>>>> independent module in Flink?
>>>> 
>>>> 
>>>> Yes. If there are submodules in Flink repository needs the iteration
>>>> (currently not),
>>>> we could consider extracting them to a new submodule of Flink.
>>>> 
>>>> 
>>>>> users will have to add one more dependency of Flink ML. If iteration
>> is
>>>> the
>>>> only feature they need, it will look a little bit weird.
>>>> 
>>>> 
>>>> If users only need to execute iteration jobs, they can simply remove
>> the
>>>> Flink
>>>> dependency and add the necessary dependencies related to Flink ML.
>>>> However,
>>>> they can still utilize the DataStream API as it is also a dependency of
>>>> Flink ML.
>>>> 
>>>> 
>>>> Keeping an iteration submodule in Flink repository and make Flink ML
>>>> depends it
>>>> is also another solution. But the current implementation of Iteration
>> in
>>>> DataStream
>>>> should be removed definitely due to its Incompleteness.
>>>> 
>>>> 
>>>> The placement of the Iteration API in the repository is a topic that
>> has
>>>> multiple
>>>> potential solutions. WDYT?
>>>> 
>>>> 
>>>> Best,
>>>> Wencong
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> At 2023-09-01 17:59:34, "Jing Ge" <ji...@ververica.com.INVALID> wrote:
>>>>> Hi Wencong,
>>>>> 
>>>>> Thanks for the proposal!
>>>>> 
>>>>> "The Iteration API in DataStream is planned be deprecated in Flink
>> 1.19
>>>> and
>>>>> then finally removed in Flink 2.0. For the users that rely on the
>>>> Iteration
>>>>> API in DataStream, they will have to migrate to Flink ML."
>>>>> - Does it make sense to migrate the iteration module into Flink
>> directly?
>>>>> Or the "independent module extraction" mentioned in the FLIP does
>> mean an
>>>>> independent module in Flink? Since the iteration will be removed in
>> Flink,
>>>>> users will have to add one more dependency of Flink ML. If iteration
>> is
>>>> the
>>>>> only feature they need, it will look a little bit weird.
>>>>> 
>>>>> 
>>>>> Best regards,
>>>>> Jing
>>>>> 
>>>>> On Fri, Sep 1, 2023 at 11:05 AM weijie guo <guoweijiereswqa@gmail.com
>>> 
>>>>> wrote:
>>>>> 
>>>>>> Thanks, +1 for this.
>>>>>> 
>>>>>> Best regards,
>>>>>> 
>>>>>> Weijie
>>>>>> 
>>>>>> 
>>>>>> Yangze Guo <ka...@gmail.com> 于2023年9月1日周五 14:29写道:
>>>>>> 
>>>>>>> +1
>>>>>>> 
>>>>>>> Thanks for driving this.
>>>>>>> 
>>>>>>> Best,
>>>>>>> Yangze Guo
>>>>>>> 
>>>>>>> On Fri, Sep 1, 2023 at 2:00 PM Xintong Song <
>> tonysong820@gmail.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>> +1
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> 
>>>>>>>> Xintong
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Sep 1, 2023 at 1:11 PM Dong Lin <li...@gmail.com>
>>>> wrote:
>>>>>>>> 
>>>>>>>>> Thanks Wencong for initiating the discussion.
>>>>>>>>> 
>>>>>>>>> +1 for the proposal.
>>>>>>>>> 
>>>>>>>>> On Fri, Sep 1, 2023 at 12:00 PM Wencong Liu <
>> liuwenclever@163.com
>>>>> 
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi devs,
>>>>>>>>>> 
>>>>>>>>>> I would like to start a discussion on FLIP-357: Deprecate
>>>> Iteration
>>>>>>> API
>>>>>>>>> of
>>>>>>>>>> DataStream [1].
>>>>>>>>>> 
>>>>>>>>>> Currently, the Iteration API of DataStream is incomplete.
>> For
>>>>>>> instance,
>>>>>>>>> it
>>>>>>>>>> lacks support
>>>>>>>>>> for iteration in sync mode and exactly once semantics.
>>>>>> Additionally,
>>>>>>> it
>>>>>>>>>> does not offer the
>>>>>>>>>> ability to set iteration termination conditions. As a
>> result,
>>>> it's
>>>>>>> hard
>>>>>>>>>> for developers to
>>>>>>>>>> build an iteration pipeline by DataStream in the practical
>>>>>>> applications
>>>>>>>>>> such as machine learning.
>>>>>>>>>> 
>>>>>>>>>> FLIP-176: Unified Iteration to Support Algorithms [2] has
>>>>>> introduced
>>>>>>> a
>>>>>>>>>> unified iteration library
>>>>>>>>>> in the Flink ML repository. This library addresses all the
>>>> issues
>>>>>>> present
>>>>>>>>>> in the Iteration API of
>>>>>>>>>> DataStream and could provide solution for all the iteration
>>>>>>> use-cases.
>>>>>>>>>> However, maintaining two
>>>>>>>>>> separate implementations of iteration in both the Flink
>>>> repository
>>>>>>> and
>>>>>>>>> the
>>>>>>>>>> Flink ML repository
>>>>>>>>>> would introduce unnecessary complexity and make it
>> difficult to
>>>>>>> maintain
>>>>>>>>>> the Iteration API.
>>>>>>>>>> 
>>>>>>>>>> As such I propose deprecating the Iteration API of
>> DataStream
>>>> and
>>>>>>>>> removing
>>>>>>>>>> it completely in the next
>>>>>>>>>> major version. In the future, if other modules in the Flink
>>>>>>> repository
>>>>>>>>>> require the use of the
>>>>>>>>>> Iteration API, we can consider extracting all Iteration
>>>>>>> implementations
>>>>>>>>>> from the Flink ML repository
>>>>>>>>>> into an independent module.
>>>>>>>>>> 
>>>>>>>>>> Looking forward to your feedback.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> [1]
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%253A%2BDeprecate%2BIteration%2BAPI%2Bof%2BDataStream&source=gmail-imap&ust=1694508403000000&usg=AOvVaw1cQxgpHJ7psRVNbB8EIOj_
>>>>>>>>>> [2]
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>> https://www.google.com/url?q=https://cwiki.apache.org/confluence/pages/viewpage.action?pageId%3D184615300&source=gmail-imap&ust=1694508403000000&usg=AOvVaw1dNs3y_W5lYqRjauWaOEME
>>>>>>>>>> 
>>>>>>>>>> Best regards,
>>>>>>>>>> 
>>>>>>>>>> Wencong Liu


Re: Re: [DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

Posted by David Morávek <dm...@apache.org>.
+1 since there is an alternative, more complete implementation available

Best,
D.

On Sat, Sep 2, 2023 at 12:07 AM David Anderson <da...@apache.org> wrote:

> +1
>
> Keeping the legacy implementation in place is confusing and encourages
> adoption of something that really shouldn't be used.
>
> Thanks for driving this,
> David
>
> On Fri, Sep 1, 2023 at 8:45 AM Jing Ge <ji...@ververica.com.invalid> wrote:
> >
> > Hi Wencong,
> >
> > Thanks for your clarification! +1
> >
> > Best regards,
> > Jing
> >
> > On Fri, Sep 1, 2023 at 12:36 PM Wencong Liu <li...@163.com>
> wrote:
> >
> > > Hi Jing,
> > >
> > >
> > > Thanks for your reply!
> > >
> > >
> > > > Or the "independent module extraction" mentioned in the FLIP does
> mean an
> > > independent module in Flink?
> > >
> > >
> > > Yes. If there are submodules in Flink repository needs the iteration
> > > (currently not),
> > > we could consider extracting them to a new submodule of Flink.
> > >
> > >
> > > > users will have to add one more dependency of Flink ML. If iteration
> is
> > > the
> > > only feature they need, it will look a little bit weird.
> > >
> > >
> > > If users only need to execute iteration jobs, they can simply remove
> the
> > > Flink
> > > dependency and add the necessary dependencies related to Flink ML.
> > > However,
> > > they can still utilize the DataStream API as it is also a dependency of
> > > Flink ML.
> > >
> > >
> > > Keeping an iteration submodule in Flink repository and make Flink ML
> > > depends it
> > > is also another solution. But the current implementation of Iteration
> in
> > > DataStream
> > > should be removed definitely due to its Incompleteness.
> > >
> > >
> > > The placement of the Iteration API in the repository is a topic that
> has
> > > multiple
> > > potential solutions. WDYT?
> > >
> > >
> > > Best,
> > > Wencong
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > At 2023-09-01 17:59:34, "Jing Ge" <ji...@ververica.com.INVALID> wrote:
> > > >Hi Wencong,
> > > >
> > > >Thanks for the proposal!
> > > >
> > > >"The Iteration API in DataStream is planned be deprecated in Flink
> 1.19
> > > and
> > > >then finally removed in Flink 2.0. For the users that rely on the
> > > Iteration
> > > >API in DataStream, they will have to migrate to Flink ML."
> > > >- Does it make sense to migrate the iteration module into Flink
> directly?
> > > >Or the "independent module extraction" mentioned in the FLIP does
> mean an
> > > >independent module in Flink? Since the iteration will be removed in
> Flink,
> > > >users will have to add one more dependency of Flink ML. If iteration
> is
> > > the
> > > >only feature they need, it will look a little bit weird.
> > > >
> > > >
> > > >Best regards,
> > > >Jing
> > > >
> > > >On Fri, Sep 1, 2023 at 11:05 AM weijie guo <guoweijiereswqa@gmail.com
> >
> > > >wrote:
> > > >
> > > >> Thanks, +1 for this.
> > > >>
> > > >> Best regards,
> > > >>
> > > >> Weijie
> > > >>
> > > >>
> > > >> Yangze Guo <ka...@gmail.com> 于2023年9月1日周五 14:29写道:
> > > >>
> > > >> > +1
> > > >> >
> > > >> > Thanks for driving this.
> > > >> >
> > > >> > Best,
> > > >> > Yangze Guo
> > > >> >
> > > >> > On Fri, Sep 1, 2023 at 2:00 PM Xintong Song <
> tonysong820@gmail.com>
> > > >> wrote:
> > > >> > >
> > > >> > > +1
> > > >> > >
> > > >> > > Best,
> > > >> > >
> > > >> > > Xintong
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Fri, Sep 1, 2023 at 1:11 PM Dong Lin <li...@gmail.com>
> > > wrote:
> > > >> > >
> > > >> > > > Thanks Wencong for initiating the discussion.
> > > >> > > >
> > > >> > > > +1 for the proposal.
> > > >> > > >
> > > >> > > > On Fri, Sep 1, 2023 at 12:00 PM Wencong Liu <
> liuwenclever@163.com
> > > >
> > > >> > wrote:
> > > >> > > >
> > > >> > > > > Hi devs,
> > > >> > > > >
> > > >> > > > > I would like to start a discussion on FLIP-357: Deprecate
> > > Iteration
> > > >> > API
> > > >> > > > of
> > > >> > > > > DataStream [1].
> > > >> > > > >
> > > >> > > > > Currently, the Iteration API of DataStream is incomplete.
> For
> > > >> > instance,
> > > >> > > > it
> > > >> > > > > lacks support
> > > >> > > > > for iteration in sync mode and exactly once semantics.
> > > >> Additionally,
> > > >> > it
> > > >> > > > > does not offer the
> > > >> > > > > ability to set iteration termination conditions. As a
> result,
> > > it's
> > > >> > hard
> > > >> > > > > for developers to
> > > >> > > > > build an iteration pipeline by DataStream in the practical
> > > >> > applications
> > > >> > > > > such as machine learning.
> > > >> > > > >
> > > >> > > > > FLIP-176: Unified Iteration to Support Algorithms [2] has
> > > >> introduced
> > > >> > a
> > > >> > > > > unified iteration library
> > > >> > > > > in the Flink ML repository. This library addresses all the
> > > issues
> > > >> > present
> > > >> > > > > in the Iteration API of
> > > >> > > > > DataStream and could provide solution for all the iteration
> > > >> > use-cases.
> > > >> > > > > However, maintaining two
> > > >> > > > > separate implementations of iteration in both the Flink
> > > repository
> > > >> > and
> > > >> > > > the
> > > >> > > > > Flink ML repository
> > > >> > > > > would introduce unnecessary complexity and make it
> difficult to
> > > >> > maintain
> > > >> > > > > the Iteration API.
> > > >> > > > >
> > > >> > > > > As such I propose deprecating the Iteration API of
> DataStream
> > > and
> > > >> > > > removing
> > > >> > > > > it completely in the next
> > > >> > > > > major version. In the future, if other modules in the Flink
> > > >> > repository
> > > >> > > > > require the use of the
> > > >> > > > > Iteration API, we can consider extracting all Iteration
> > > >> > implementations
> > > >> > > > > from the Flink ML repository
> > > >> > > > > into an independent module.
> > > >> > > > >
> > > >> > > > > Looking forward to your feedback.
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > [1]
> > > >> > > > >
> > > >> > > >
> > > >> >
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%3A+Deprecate+Iteration+API+of+DataStream
> > > >> > > > > [2]
> > > >> > > > >
> > > >> > > >
> > > >> >
> > > >>
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=184615300
> > > >> > > > >
> > > >> > > > > Best regards,
> > > >> > > > >
> > > >> > > > > Wencong Liu
> > > >> > > >
> > > >> >
> > > >>
> > >
>

Re: Re: [DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

Posted by David Anderson <da...@apache.org>.
+1

Keeping the legacy implementation in place is confusing and encourages
adoption of something that really shouldn't be used.

Thanks for driving this,
David

On Fri, Sep 1, 2023 at 8:45 AM Jing Ge <ji...@ververica.com.invalid> wrote:
>
> Hi Wencong,
>
> Thanks for your clarification! +1
>
> Best regards,
> Jing
>
> On Fri, Sep 1, 2023 at 12:36 PM Wencong Liu <li...@163.com> wrote:
>
> > Hi Jing,
> >
> >
> > Thanks for your reply!
> >
> >
> > > Or the "independent module extraction" mentioned in the FLIP does mean an
> > independent module in Flink?
> >
> >
> > Yes. If there are submodules in Flink repository needs the iteration
> > (currently not),
> > we could consider extracting them to a new submodule of Flink.
> >
> >
> > > users will have to add one more dependency of Flink ML. If iteration is
> > the
> > only feature they need, it will look a little bit weird.
> >
> >
> > If users only need to execute iteration jobs, they can simply remove the
> > Flink
> > dependency and add the necessary dependencies related to Flink ML.
> > However,
> > they can still utilize the DataStream API as it is also a dependency of
> > Flink ML.
> >
> >
> > Keeping an iteration submodule in Flink repository and make Flink ML
> > depends it
> > is also another solution. But the current implementation of Iteration in
> > DataStream
> > should be removed definitely due to its Incompleteness.
> >
> >
> > The placement of the Iteration API in the repository is a topic that has
> > multiple
> > potential solutions. WDYT?
> >
> >
> > Best,
> > Wencong
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > At 2023-09-01 17:59:34, "Jing Ge" <ji...@ververica.com.INVALID> wrote:
> > >Hi Wencong,
> > >
> > >Thanks for the proposal!
> > >
> > >"The Iteration API in DataStream is planned be deprecated in Flink 1.19
> > and
> > >then finally removed in Flink 2.0. For the users that rely on the
> > Iteration
> > >API in DataStream, they will have to migrate to Flink ML."
> > >- Does it make sense to migrate the iteration module into Flink directly?
> > >Or the "independent module extraction" mentioned in the FLIP does mean an
> > >independent module in Flink? Since the iteration will be removed in Flink,
> > >users will have to add one more dependency of Flink ML. If iteration is
> > the
> > >only feature they need, it will look a little bit weird.
> > >
> > >
> > >Best regards,
> > >Jing
> > >
> > >On Fri, Sep 1, 2023 at 11:05 AM weijie guo <gu...@gmail.com>
> > >wrote:
> > >
> > >> Thanks, +1 for this.
> > >>
> > >> Best regards,
> > >>
> > >> Weijie
> > >>
> > >>
> > >> Yangze Guo <ka...@gmail.com> 于2023年9月1日周五 14:29写道:
> > >>
> > >> > +1
> > >> >
> > >> > Thanks for driving this.
> > >> >
> > >> > Best,
> > >> > Yangze Guo
> > >> >
> > >> > On Fri, Sep 1, 2023 at 2:00 PM Xintong Song <to...@gmail.com>
> > >> wrote:
> > >> > >
> > >> > > +1
> > >> > >
> > >> > > Best,
> > >> > >
> > >> > > Xintong
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Fri, Sep 1, 2023 at 1:11 PM Dong Lin <li...@gmail.com>
> > wrote:
> > >> > >
> > >> > > > Thanks Wencong for initiating the discussion.
> > >> > > >
> > >> > > > +1 for the proposal.
> > >> > > >
> > >> > > > On Fri, Sep 1, 2023 at 12:00 PM Wencong Liu <liuwenclever@163.com
> > >
> > >> > wrote:
> > >> > > >
> > >> > > > > Hi devs,
> > >> > > > >
> > >> > > > > I would like to start a discussion on FLIP-357: Deprecate
> > Iteration
> > >> > API
> > >> > > > of
> > >> > > > > DataStream [1].
> > >> > > > >
> > >> > > > > Currently, the Iteration API of DataStream is incomplete. For
> > >> > instance,
> > >> > > > it
> > >> > > > > lacks support
> > >> > > > > for iteration in sync mode and exactly once semantics.
> > >> Additionally,
> > >> > it
> > >> > > > > does not offer the
> > >> > > > > ability to set iteration termination conditions. As a result,
> > it's
> > >> > hard
> > >> > > > > for developers to
> > >> > > > > build an iteration pipeline by DataStream in the practical
> > >> > applications
> > >> > > > > such as machine learning.
> > >> > > > >
> > >> > > > > FLIP-176: Unified Iteration to Support Algorithms [2] has
> > >> introduced
> > >> > a
> > >> > > > > unified iteration library
> > >> > > > > in the Flink ML repository. This library addresses all the
> > issues
> > >> > present
> > >> > > > > in the Iteration API of
> > >> > > > > DataStream and could provide solution for all the iteration
> > >> > use-cases.
> > >> > > > > However, maintaining two
> > >> > > > > separate implementations of iteration in both the Flink
> > repository
> > >> > and
> > >> > > > the
> > >> > > > > Flink ML repository
> > >> > > > > would introduce unnecessary complexity and make it difficult to
> > >> > maintain
> > >> > > > > the Iteration API.
> > >> > > > >
> > >> > > > > As such I propose deprecating the Iteration API of DataStream
> > and
> > >> > > > removing
> > >> > > > > it completely in the next
> > >> > > > > major version. In the future, if other modules in the Flink
> > >> > repository
> > >> > > > > require the use of the
> > >> > > > > Iteration API, we can consider extracting all Iteration
> > >> > implementations
> > >> > > > > from the Flink ML repository
> > >> > > > > into an independent module.
> > >> > > > >
> > >> > > > > Looking forward to your feedback.
> > >> > > > >
> > >> > > > >
> > >> > > > > [1]
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%3A+Deprecate+Iteration+API+of+DataStream
> > >> > > > > [2]
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=184615300
> > >> > > > >
> > >> > > > > Best regards,
> > >> > > > >
> > >> > > > > Wencong Liu
> > >> > > >
> > >> >
> > >>
> >

Re: Re: [DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Wencong,

Thanks for your clarification! +1

Best regards,
Jing

On Fri, Sep 1, 2023 at 12:36 PM Wencong Liu <li...@163.com> wrote:

> Hi Jing,
>
>
> Thanks for your reply!
>
>
> > Or the "independent module extraction" mentioned in the FLIP does mean an
> independent module in Flink?
>
>
> Yes. If there are submodules in Flink repository needs the iteration
> (currently not),
> we could consider extracting them to a new submodule of Flink.
>
>
> > users will have to add one more dependency of Flink ML. If iteration is
> the
> only feature they need, it will look a little bit weird.
>
>
> If users only need to execute iteration jobs, they can simply remove the
> Flink
> dependency and add the necessary dependencies related to Flink ML.
> However,
> they can still utilize the DataStream API as it is also a dependency of
> Flink ML.
>
>
> Keeping an iteration submodule in Flink repository and make Flink ML
> depends it
> is also another solution. But the current implementation of Iteration in
> DataStream
> should be removed definitely due to its Incompleteness.
>
>
> The placement of the Iteration API in the repository is a topic that has
> multiple
> potential solutions. WDYT?
>
>
> Best,
> Wencong
>
>
>
>
>
>
>
>
>
>
>
> At 2023-09-01 17:59:34, "Jing Ge" <ji...@ververica.com.INVALID> wrote:
> >Hi Wencong,
> >
> >Thanks for the proposal!
> >
> >"The Iteration API in DataStream is planned be deprecated in Flink 1.19
> and
> >then finally removed in Flink 2.0. For the users that rely on the
> Iteration
> >API in DataStream, they will have to migrate to Flink ML."
> >- Does it make sense to migrate the iteration module into Flink directly?
> >Or the "independent module extraction" mentioned in the FLIP does mean an
> >independent module in Flink? Since the iteration will be removed in Flink,
> >users will have to add one more dependency of Flink ML. If iteration is
> the
> >only feature they need, it will look a little bit weird.
> >
> >
> >Best regards,
> >Jing
> >
> >On Fri, Sep 1, 2023 at 11:05 AM weijie guo <gu...@gmail.com>
> >wrote:
> >
> >> Thanks, +1 for this.
> >>
> >> Best regards,
> >>
> >> Weijie
> >>
> >>
> >> Yangze Guo <ka...@gmail.com> 于2023年9月1日周五 14:29写道:
> >>
> >> > +1
> >> >
> >> > Thanks for driving this.
> >> >
> >> > Best,
> >> > Yangze Guo
> >> >
> >> > On Fri, Sep 1, 2023 at 2:00 PM Xintong Song <to...@gmail.com>
> >> wrote:
> >> > >
> >> > > +1
> >> > >
> >> > > Best,
> >> > >
> >> > > Xintong
> >> > >
> >> > >
> >> > >
> >> > > On Fri, Sep 1, 2023 at 1:11 PM Dong Lin <li...@gmail.com>
> wrote:
> >> > >
> >> > > > Thanks Wencong for initiating the discussion.
> >> > > >
> >> > > > +1 for the proposal.
> >> > > >
> >> > > > On Fri, Sep 1, 2023 at 12:00 PM Wencong Liu <liuwenclever@163.com
> >
> >> > wrote:
> >> > > >
> >> > > > > Hi devs,
> >> > > > >
> >> > > > > I would like to start a discussion on FLIP-357: Deprecate
> Iteration
> >> > API
> >> > > > of
> >> > > > > DataStream [1].
> >> > > > >
> >> > > > > Currently, the Iteration API of DataStream is incomplete. For
> >> > instance,
> >> > > > it
> >> > > > > lacks support
> >> > > > > for iteration in sync mode and exactly once semantics.
> >> Additionally,
> >> > it
> >> > > > > does not offer the
> >> > > > > ability to set iteration termination conditions. As a result,
> it's
> >> > hard
> >> > > > > for developers to
> >> > > > > build an iteration pipeline by DataStream in the practical
> >> > applications
> >> > > > > such as machine learning.
> >> > > > >
> >> > > > > FLIP-176: Unified Iteration to Support Algorithms [2] has
> >> introduced
> >> > a
> >> > > > > unified iteration library
> >> > > > > in the Flink ML repository. This library addresses all the
> issues
> >> > present
> >> > > > > in the Iteration API of
> >> > > > > DataStream and could provide solution for all the iteration
> >> > use-cases.
> >> > > > > However, maintaining two
> >> > > > > separate implementations of iteration in both the Flink
> repository
> >> > and
> >> > > > the
> >> > > > > Flink ML repository
> >> > > > > would introduce unnecessary complexity and make it difficult to
> >> > maintain
> >> > > > > the Iteration API.
> >> > > > >
> >> > > > > As such I propose deprecating the Iteration API of DataStream
> and
> >> > > > removing
> >> > > > > it completely in the next
> >> > > > > major version. In the future, if other modules in the Flink
> >> > repository
> >> > > > > require the use of the
> >> > > > > Iteration API, we can consider extracting all Iteration
> >> > implementations
> >> > > > > from the Flink ML repository
> >> > > > > into an independent module.
> >> > > > >
> >> > > > > Looking forward to your feedback.
> >> > > > >
> >> > > > >
> >> > > > > [1]
> >> > > > >
> >> > > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%3A+Deprecate+Iteration+API+of+DataStream
> >> > > > > [2]
> >> > > > >
> >> > > >
> >> >
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=184615300
> >> > > > >
> >> > > > > Best regards,
> >> > > > >
> >> > > > > Wencong Liu
> >> > > >
> >> >
> >>
>

Re:Re: [DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

Posted by Wencong Liu <li...@163.com>.
Hi Jing,


Thanks for your reply!


> Or the "independent module extraction" mentioned in the FLIP does mean an
independent module in Flink?


Yes. If there are submodules in Flink repository needs the iteration (currently not),
we could consider extracting them to a new submodule of Flink.


> users will have to add one more dependency of Flink ML. If iteration is the
only feature they need, it will look a little bit weird.


If users only need to execute iteration jobs, they can simply remove the Flink 
dependency and add the necessary dependencies related to Flink ML. However, 
they can still utilize the DataStream API as it is also a dependency of Flink ML.


Keeping an iteration submodule in Flink repository and make Flink ML depends it
is also another solution. But the current implementation of Iteration in DataStream
should be removed definitely due to its Incompleteness.


The placement of the Iteration API in the repository is a topic that has multiple 
potential solutions. WDYT?


Best,
Wencong











At 2023-09-01 17:59:34, "Jing Ge" <ji...@ververica.com.INVALID> wrote:
>Hi Wencong,
>
>Thanks for the proposal!
>
>"The Iteration API in DataStream is planned be deprecated in Flink 1.19 and
>then finally removed in Flink 2.0. For the users that rely on the Iteration
>API in DataStream, they will have to migrate to Flink ML."
>- Does it make sense to migrate the iteration module into Flink directly?
>Or the "independent module extraction" mentioned in the FLIP does mean an
>independent module in Flink? Since the iteration will be removed in Flink,
>users will have to add one more dependency of Flink ML. If iteration is the
>only feature they need, it will look a little bit weird.
>
>
>Best regards,
>Jing
>
>On Fri, Sep 1, 2023 at 11:05 AM weijie guo <gu...@gmail.com>
>wrote:
>
>> Thanks, +1 for this.
>>
>> Best regards,
>>
>> Weijie
>>
>>
>> Yangze Guo <ka...@gmail.com> 于2023年9月1日周五 14:29写道:
>>
>> > +1
>> >
>> > Thanks for driving this.
>> >
>> > Best,
>> > Yangze Guo
>> >
>> > On Fri, Sep 1, 2023 at 2:00 PM Xintong Song <to...@gmail.com>
>> wrote:
>> > >
>> > > +1
>> > >
>> > > Best,
>> > >
>> > > Xintong
>> > >
>> > >
>> > >
>> > > On Fri, Sep 1, 2023 at 1:11 PM Dong Lin <li...@gmail.com> wrote:
>> > >
>> > > > Thanks Wencong for initiating the discussion.
>> > > >
>> > > > +1 for the proposal.
>> > > >
>> > > > On Fri, Sep 1, 2023 at 12:00 PM Wencong Liu <li...@163.com>
>> > wrote:
>> > > >
>> > > > > Hi devs,
>> > > > >
>> > > > > I would like to start a discussion on FLIP-357: Deprecate Iteration
>> > API
>> > > > of
>> > > > > DataStream [1].
>> > > > >
>> > > > > Currently, the Iteration API of DataStream is incomplete. For
>> > instance,
>> > > > it
>> > > > > lacks support
>> > > > > for iteration in sync mode and exactly once semantics.
>> Additionally,
>> > it
>> > > > > does not offer the
>> > > > > ability to set iteration termination conditions. As a result, it's
>> > hard
>> > > > > for developers to
>> > > > > build an iteration pipeline by DataStream in the practical
>> > applications
>> > > > > such as machine learning.
>> > > > >
>> > > > > FLIP-176: Unified Iteration to Support Algorithms [2] has
>> introduced
>> > a
>> > > > > unified iteration library
>> > > > > in the Flink ML repository. This library addresses all the issues
>> > present
>> > > > > in the Iteration API of
>> > > > > DataStream and could provide solution for all the iteration
>> > use-cases.
>> > > > > However, maintaining two
>> > > > > separate implementations of iteration in both the Flink repository
>> > and
>> > > > the
>> > > > > Flink ML repository
>> > > > > would introduce unnecessary complexity and make it difficult to
>> > maintain
>> > > > > the Iteration API.
>> > > > >
>> > > > > As such I propose deprecating the Iteration API of DataStream and
>> > > > removing
>> > > > > it completely in the next
>> > > > > major version. In the future, if other modules in the Flink
>> > repository
>> > > > > require the use of the
>> > > > > Iteration API, we can consider extracting all Iteration
>> > implementations
>> > > > > from the Flink ML repository
>> > > > > into an independent module.
>> > > > >
>> > > > > Looking forward to your feedback.
>> > > > >
>> > > > >
>> > > > > [1]
>> > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%3A+Deprecate+Iteration+API+of+DataStream
>> > > > > [2]
>> > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=184615300
>> > > > >
>> > > > > Best regards,
>> > > > >
>> > > > > Wencong Liu
>> > > >
>> >
>>

Re: [DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Wencong,

Thanks for the proposal!

"The Iteration API in DataStream is planned be deprecated in Flink 1.19 and
then finally removed in Flink 2.0. For the users that rely on the Iteration
API in DataStream, they will have to migrate to Flink ML."
- Does it make sense to migrate the iteration module into Flink directly?
Or the "independent module extraction" mentioned in the FLIP does mean an
independent module in Flink? Since the iteration will be removed in Flink,
users will have to add one more dependency of Flink ML. If iteration is the
only feature they need, it will look a little bit weird.


Best regards,
Jing

On Fri, Sep 1, 2023 at 11:05 AM weijie guo <gu...@gmail.com>
wrote:

> Thanks, +1 for this.
>
> Best regards,
>
> Weijie
>
>
> Yangze Guo <ka...@gmail.com> 于2023年9月1日周五 14:29写道:
>
> > +1
> >
> > Thanks for driving this.
> >
> > Best,
> > Yangze Guo
> >
> > On Fri, Sep 1, 2023 at 2:00 PM Xintong Song <to...@gmail.com>
> wrote:
> > >
> > > +1
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Fri, Sep 1, 2023 at 1:11 PM Dong Lin <li...@gmail.com> wrote:
> > >
> > > > Thanks Wencong for initiating the discussion.
> > > >
> > > > +1 for the proposal.
> > > >
> > > > On Fri, Sep 1, 2023 at 12:00 PM Wencong Liu <li...@163.com>
> > wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > I would like to start a discussion on FLIP-357: Deprecate Iteration
> > API
> > > > of
> > > > > DataStream [1].
> > > > >
> > > > > Currently, the Iteration API of DataStream is incomplete. For
> > instance,
> > > > it
> > > > > lacks support
> > > > > for iteration in sync mode and exactly once semantics.
> Additionally,
> > it
> > > > > does not offer the
> > > > > ability to set iteration termination conditions. As a result, it's
> > hard
> > > > > for developers to
> > > > > build an iteration pipeline by DataStream in the practical
> > applications
> > > > > such as machine learning.
> > > > >
> > > > > FLIP-176: Unified Iteration to Support Algorithms [2] has
> introduced
> > a
> > > > > unified iteration library
> > > > > in the Flink ML repository. This library addresses all the issues
> > present
> > > > > in the Iteration API of
> > > > > DataStream and could provide solution for all the iteration
> > use-cases.
> > > > > However, maintaining two
> > > > > separate implementations of iteration in both the Flink repository
> > and
> > > > the
> > > > > Flink ML repository
> > > > > would introduce unnecessary complexity and make it difficult to
> > maintain
> > > > > the Iteration API.
> > > > >
> > > > > As such I propose deprecating the Iteration API of DataStream and
> > > > removing
> > > > > it completely in the next
> > > > > major version. In the future, if other modules in the Flink
> > repository
> > > > > require the use of the
> > > > > Iteration API, we can consider extracting all Iteration
> > implementations
> > > > > from the Flink ML repository
> > > > > into an independent module.
> > > > >
> > > > > Looking forward to your feedback.
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%3A+Deprecate+Iteration+API+of+DataStream
> > > > > [2]
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=184615300
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Wencong Liu
> > > >
> >
>

Re: [DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

Posted by weijie guo <gu...@gmail.com>.
Thanks, +1 for this.

Best regards,

Weijie


Yangze Guo <ka...@gmail.com> 于2023年9月1日周五 14:29写道:

> +1
>
> Thanks for driving this.
>
> Best,
> Yangze Guo
>
> On Fri, Sep 1, 2023 at 2:00 PM Xintong Song <to...@gmail.com> wrote:
> >
> > +1
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Fri, Sep 1, 2023 at 1:11 PM Dong Lin <li...@gmail.com> wrote:
> >
> > > Thanks Wencong for initiating the discussion.
> > >
> > > +1 for the proposal.
> > >
> > > On Fri, Sep 1, 2023 at 12:00 PM Wencong Liu <li...@163.com>
> wrote:
> > >
> > > > Hi devs,
> > > >
> > > > I would like to start a discussion on FLIP-357: Deprecate Iteration
> API
> > > of
> > > > DataStream [1].
> > > >
> > > > Currently, the Iteration API of DataStream is incomplete. For
> instance,
> > > it
> > > > lacks support
> > > > for iteration in sync mode and exactly once semantics. Additionally,
> it
> > > > does not offer the
> > > > ability to set iteration termination conditions. As a result, it's
> hard
> > > > for developers to
> > > > build an iteration pipeline by DataStream in the practical
> applications
> > > > such as machine learning.
> > > >
> > > > FLIP-176: Unified Iteration to Support Algorithms [2] has introduced
> a
> > > > unified iteration library
> > > > in the Flink ML repository. This library addresses all the issues
> present
> > > > in the Iteration API of
> > > > DataStream and could provide solution for all the iteration
> use-cases.
> > > > However, maintaining two
> > > > separate implementations of iteration in both the Flink repository
> and
> > > the
> > > > Flink ML repository
> > > > would introduce unnecessary complexity and make it difficult to
> maintain
> > > > the Iteration API.
> > > >
> > > > As such I propose deprecating the Iteration API of DataStream and
> > > removing
> > > > it completely in the next
> > > > major version. In the future, if other modules in the Flink
> repository
> > > > require the use of the
> > > > Iteration API, we can consider extracting all Iteration
> implementations
> > > > from the Flink ML repository
> > > > into an independent module.
> > > >
> > > > Looking forward to your feedback.
> > > >
> > > >
> > > > [1]
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%3A+Deprecate+Iteration+API+of+DataStream
> > > > [2]
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=184615300
> > > >
> > > > Best regards,
> > > >
> > > > Wencong Liu
> > >
>

Re: [DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

Posted by Yangze Guo <ka...@gmail.com>.
+1

Thanks for driving this.

Best,
Yangze Guo

On Fri, Sep 1, 2023 at 2:00 PM Xintong Song <to...@gmail.com> wrote:
>
> +1
>
> Best,
>
> Xintong
>
>
>
> On Fri, Sep 1, 2023 at 1:11 PM Dong Lin <li...@gmail.com> wrote:
>
> > Thanks Wencong for initiating the discussion.
> >
> > +1 for the proposal.
> >
> > On Fri, Sep 1, 2023 at 12:00 PM Wencong Liu <li...@163.com> wrote:
> >
> > > Hi devs,
> > >
> > > I would like to start a discussion on FLIP-357: Deprecate Iteration API
> > of
> > > DataStream [1].
> > >
> > > Currently, the Iteration API of DataStream is incomplete. For instance,
> > it
> > > lacks support
> > > for iteration in sync mode and exactly once semantics. Additionally, it
> > > does not offer the
> > > ability to set iteration termination conditions. As a result, it's hard
> > > for developers to
> > > build an iteration pipeline by DataStream in the practical applications
> > > such as machine learning.
> > >
> > > FLIP-176: Unified Iteration to Support Algorithms [2] has introduced a
> > > unified iteration library
> > > in the Flink ML repository. This library addresses all the issues present
> > > in the Iteration API of
> > > DataStream and could provide solution for all the iteration use-cases.
> > > However, maintaining two
> > > separate implementations of iteration in both the Flink repository and
> > the
> > > Flink ML repository
> > > would introduce unnecessary complexity and make it difficult to maintain
> > > the Iteration API.
> > >
> > > As such I propose deprecating the Iteration API of DataStream and
> > removing
> > > it completely in the next
> > > major version. In the future, if other modules in the Flink repository
> > > require the use of the
> > > Iteration API, we can consider extracting all Iteration implementations
> > > from the Flink ML repository
> > > into an independent module.
> > >
> > > Looking forward to your feedback.
> > >
> > >
> > > [1]
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%3A+Deprecate+Iteration+API+of+DataStream
> > > [2]
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=184615300
> > >
> > > Best regards,
> > >
> > > Wencong Liu
> >

Re: [DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

Posted by Xintong Song <to...@gmail.com>.
+1

Best,

Xintong



On Fri, Sep 1, 2023 at 1:11 PM Dong Lin <li...@gmail.com> wrote:

> Thanks Wencong for initiating the discussion.
>
> +1 for the proposal.
>
> On Fri, Sep 1, 2023 at 12:00 PM Wencong Liu <li...@163.com> wrote:
>
> > Hi devs,
> >
> > I would like to start a discussion on FLIP-357: Deprecate Iteration API
> of
> > DataStream [1].
> >
> > Currently, the Iteration API of DataStream is incomplete. For instance,
> it
> > lacks support
> > for iteration in sync mode and exactly once semantics. Additionally, it
> > does not offer the
> > ability to set iteration termination conditions. As a result, it's hard
> > for developers to
> > build an iteration pipeline by DataStream in the practical applications
> > such as machine learning.
> >
> > FLIP-176: Unified Iteration to Support Algorithms [2] has introduced a
> > unified iteration library
> > in the Flink ML repository. This library addresses all the issues present
> > in the Iteration API of
> > DataStream and could provide solution for all the iteration use-cases.
> > However, maintaining two
> > separate implementations of iteration in both the Flink repository and
> the
> > Flink ML repository
> > would introduce unnecessary complexity and make it difficult to maintain
> > the Iteration API.
> >
> > As such I propose deprecating the Iteration API of DataStream and
> removing
> > it completely in the next
> > major version. In the future, if other modules in the Flink repository
> > require the use of the
> > Iteration API, we can consider extracting all Iteration implementations
> > from the Flink ML repository
> > into an independent module.
> >
> > Looking forward to your feedback.
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%3A+Deprecate+Iteration+API+of+DataStream
> > [2]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=184615300
> >
> > Best regards,
> >
> > Wencong Liu
>

Re: [DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

Posted by Dong Lin <li...@gmail.com>.
Thanks Wencong for initiating the discussion.

+1 for the proposal.

On Fri, Sep 1, 2023 at 12:00 PM Wencong Liu <li...@163.com> wrote:

> Hi devs,
>
> I would like to start a discussion on FLIP-357: Deprecate Iteration API of
> DataStream [1].
>
> Currently, the Iteration API of DataStream is incomplete. For instance, it
> lacks support
> for iteration in sync mode and exactly once semantics. Additionally, it
> does not offer the
> ability to set iteration termination conditions. As a result, it's hard
> for developers to
> build an iteration pipeline by DataStream in the practical applications
> such as machine learning.
>
> FLIP-176: Unified Iteration to Support Algorithms [2] has introduced a
> unified iteration library
> in the Flink ML repository. This library addresses all the issues present
> in the Iteration API of
> DataStream and could provide solution for all the iteration use-cases.
> However, maintaining two
> separate implementations of iteration in both the Flink repository and the
> Flink ML repository
> would introduce unnecessary complexity and make it difficult to maintain
> the Iteration API.
>
> As such I propose deprecating the Iteration API of DataStream and removing
> it completely in the next
> major version. In the future, if other modules in the Flink repository
> require the use of the
> Iteration API, we can consider extracting all Iteration implementations
> from the Flink ML repository
> into an independent module.
>
> Looking forward to your feedback.
>
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%3A+Deprecate+Iteration+API+of+DataStream
> [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=184615300
>
> Best regards,
>
> Wencong Liu