You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Zhipeng Zhang (Jira)" <ji...@apache.org> on 2023/02/23 04:24:00 UTC

[jira] [Updated] (FLINK-31173) Fix several bugs in flink-ml-iteration module

     [ https://issues.apache.org/jira/browse/FLINK-31173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zhipeng Zhang updated FLINK-31173:
----------------------------------
    Description: 
In flink-ml-iteration, there are several bugs as follows:

 
 # TailOperator should have one input operator. We have added a Tail operator to increment the epoch watermark at each iteration. We have made an assumption that each Tail operator have only one input and did not align the epoch watermarks from different inputs. This assumption might not be true if the input is an `union`.
 # ReplayOperator should replay the records when it receives max epochwatermark. Currently ReplayOperator does not replay the records when it receives the max epochwatermark. However, it is inconsistent with the HeadOperator. As HeadOperator will always forward the record to downstreams.  
 # ProxyOperatorStateBackend does not correctly initialize the state descriptor.

  was:
In flink-ml-iteration, we have added a Tail operator to increment the epoch watermark at each iteration. We have made an assumption that each Tail operator have only one input and did not align the epoch watermarks from different inputs. This assumption might not be true if the input is an `union`.

 

I propose to add an explicit check to prevent from having multiple input for the TailOperator. If a the input does contain multiple inputs, then users can add a map operator after union.

        Summary: Fix several bugs in flink-ml-iteration module  (was: TailOperator should only have one input)

> Fix several bugs in flink-ml-iteration module
> ---------------------------------------------
>
>                 Key: FLINK-31173
>                 URL: https://issues.apache.org/jira/browse/FLINK-31173
>             Project: Flink
>          Issue Type: Bug
>          Components: Library / Machine Learning
>    Affects Versions: ml-2.0.0, ml-2.1.0, ml-2.2.0
>            Reporter: Zhipeng Zhang
>            Priority: Major
>
> In flink-ml-iteration, there are several bugs as follows:
>  
>  # TailOperator should have one input operator. We have added a Tail operator to increment the epoch watermark at each iteration. We have made an assumption that each Tail operator have only one input and did not align the epoch watermarks from different inputs. This assumption might not be true if the input is an `union`.
>  # ReplayOperator should replay the records when it receives max epochwatermark. Currently ReplayOperator does not replay the records when it receives the max epochwatermark. However, it is inconsistent with the HeadOperator. As HeadOperator will always forward the record to downstreams.  
>  # ProxyOperatorStateBackend does not correctly initialize the state descriptor.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)