You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "David Radley (Jira)" <ji...@apache.org> on 2025/02/19 14:36:00 UTC
[jira] [Comment Edited] (FLINK-37327) Debezium Avro Format: Add FormatOption to Optionally Skip emitting UPDATE_BEFORE Rows
[ https://issues.apache.org/jira/browse/FLINK-37327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928441#comment-17928441 ]
David Radley edited comment on FLINK-37327 at 2/19/25 2:35 PM:
---------------------------------------------------------------
[~klam-shop]Please could you give more detail as to how the downstream sinks can handle it.
From [https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/table/formats/debezium/ |https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/table/formats/debezium/] it say "However, currently Flink can’t combine UPDATE_BEFORE and UPDATE_AFTER into a single UPDATE message. Therefore, Flink encodes UPDATE_BEFORE and UDPATE_AFTER as DELETE and INSERT Debezium messages." I assume if this new mode is on then we do not get a DELETE but we only get an INSERT, so you are saying the downstream sinks can handle an INSERT when previously it got a DELETE first.
Also I wonder how downstream operators would be effected by this mode.
was (Author: davidrad):
Please could you give more detail as to how the downstream sinks can handle it.
From [https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/table/formats/debezium/ |https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/table/formats/debezium/] it say "However, currently Flink can’t combine UPDATE_BEFORE and UPDATE_AFTER into a single UPDATE message. Therefore, Flink encodes UPDATE_BEFORE and UDPATE_AFTER as DELETE and INSERT Debezium messages." I assume if this new mode is on then we do not get a DELETE but we only get an INSERT, so you are saying the downstream sinks can handle an INSERT when previously it got a DELETE first.
Also I wonder how downstream operators would be effected by this mode.
> Debezium Avro Format: Add FormatOption to Optionally Skip emitting UPDATE_BEFORE Rows
> -------------------------------------------------------------------------------------
>
> Key: FLINK-37327
> URL: https://issues.apache.org/jira/browse/FLINK-37327
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Affects Versions: 1.20.1
> Reporter: Kevin Lam
> Priority: Minor
> Labels: pull-request-available
>
> Add a Format Option to the Debezium Format to optionally skip emitting the UPDATE_BEFORE Rows when deserializing a Debezium message with op='u'.
> This is helpful for Flink SQL applications that want to operate in UPSERT (ChangelogMode=[I,UA,D]) mode and save on processing the UPDATE_BEFORE Rows since the downstream sinks can handle it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)