[jira] [Commented] (FLINK-18825) Support to process CDC message in batch mode

["Nicholas Jiang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17191278#comment-17191278 ] 

Nicholas Jiang commented on FLINK-18825:
----------------------------------------

[~jark], if no anyone try to work this issue, I would like to add MaterializeOperator above mentioned for supporting to process CDC message in batch mode. If possible, could you please assign to me?

> Support to process CDC message in batch mode
> --------------------------------------------
>
>                 Key: FLINK-18825
>                 URL: https://issues.apache.org/jira/browse/FLINK-18825
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table SQL / Ecosystem
>            Reporter: Jark Wu
>            Priority: Major
>
> Currently, processing CDC is only supported for streaming mode. However, there is also cases to process CDC data in batch mode. For example, load, materialize and compress CDC data into Hive using parquet format. 
> Interpreting changelog in batch mode can be supported by adding a MaterializeOperator after the CDC source operator. The MaterializeOperator  acts like a HashAggregate and SortAggregate based on whether all the fields are fixed-length. It will consume all the input RowDatas and replace/remove rows depending on the flag in the header. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)