[jira] [Commented] (FLINK-28242) CDC source with meta columns may cause error result on downstream stateful operators

["lincoln lee (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-28242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566230#comment-17566230 ] 

lincoln lee commented on FLINK-28242:
-------------------------------------

This can be unified as one case of non-deterministic update which will be solved by FLINK-27849, so mark this as duplicated.

> CDC source with meta columns may cause error result on downstream stateful operators
> ------------------------------------------------------------------------------------
>
>                 Key: FLINK-28242
>                 URL: https://issues.apache.org/jira/browse/FLINK-28242
>             Project: Flink
>          Issue Type: Bug
>          Components: Table SQL / Runtime
>    Affects Versions: 1.15.0
>            Reporter: lincoln lee
>            Priority: Major
>
> The intermediate result of current test case temporalJoinITCase#testEventTimeMultiTemporalJoin is wrong:
> {code}
> +I,    5,RMB,40,2020-08-16T00:03,null,null,null,null
> +I,    2,US Dollar,1,2020-08-15T00:02,102,2020-08-15T00:00:02,102,2020-08-15T00:00:02
> +I,    3,RMB,40,2020-08-15T00:03,702,2020-08-15T00:00:04,702,2020-08-15T00:00:04
> -U,   2,US Dollar,1,2020-08-16T00:03,106,2020-08-16T00:02,106,2020-08-16T00:02
> ...
> {code}
> because the "-U,   2,US Dollar,1,2020-08-16T00:03..." has a different 'order_time' column against "+I,    2,US Dollar,1,2020-08-15T00:02...", and after join there's no upsert key, so downstream operator can only do retract by the complete row, and will fail at this case.
> The root cause is when cdc source carries meta data column (e.g., operation time in binlog or operation type, which will make the delete|update_before message not exactly the same as the previous version), and after some operations like join (not on the primary key of cdc source, the output will have no upsert key anymore), then downstream operator can not do retract correctly.
> This is obscure to users, but we should think of a way to at least report the error to users (during compiling), or other solution eliminate the problem completely.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)