[jira] [Commented] (FLINK-23426) Support changelog processing in batch mode

["Timo Walther (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-23426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383244#comment-17383244 ] 

Timo Walther commented on FLINK-23426:
--------------------------------------

[~jark] [~ykt836] what have been your thoughts on this topic?

> Support changelog processing in batch mode
> ------------------------------------------
>
>                 Key: FLINK-23426
>                 URL: https://issues.apache.org/jira/browse/FLINK-23426
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Table SQL / API
>            Reporter: Timo Walther
>            Priority: Major
>
> The DataStream API can execute arbitrary DataStream programs when running in batch mode. However, this is not the case for the Table API batch mode. E.g. a source with non-insert only changes is not supported and updates/deletes cannot be emitted.
> In theory, we could make this work by running the "stream mode" of the planner (CDC transformations) on top of the "batch mode" of DataStream API (specialized state backend, sorted inputs). It is up for discussion if and how we expose such functionality.
> If we don't allow enabling incremental updates, we can also add a special batch operator that materializes the incoming changes for a batch pipeline. However, it would require "complete" CDC logs (i.e. no missing UPDATE_AFTER).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)