[jira] [Commented] (FLINK-18826) Support to emit and encode upsert messages to Kafka

["Shengkai Fang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-18826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185716#comment-17185716 ] 

Shengkai Fang commented on FLINK-18826:
---------------------------------------

hi, all. I think it's a good idea to add an option such as {{sink.update-mode}} in Kafka connector to indicate whether to support upsert write. It will drop all delete messages and retain all insert message automatically if this option turns on.

> Support to emit and encode upsert messages to Kafka
> ---------------------------------------------------
>
>                 Key: FLINK-18826
>                 URL: https://issues.apache.org/jira/browse/FLINK-18826
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table SQL / Ecosystem
>            Reporter: Jark Wu
>            Assignee: Shengkai Fang
>            Priority: Major
>
> Currently, it is not possible to emit the results of a group by query into Kafka. We can provide a format (e.g. {{upsert-json}}??) to encode changelogs to make it possible to write into Kafka. Another idea is provide an {{update-mode=upsert}} property, in this way, we can support upsert behavior for all the append-only formats (avro, avro-confluent, json, csv) without changing the implementation of the formats. The update-mode came from kafka connector [1] before 1.10, but was dropped after that. 
> [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/connect.html#update-modes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)