You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "klam-shop (via GitHub)" <gi...@apache.org> on 2025/02/14 20:43:24 UTC
[PR] [FLINK-37327] [Formats (JSON, Avro, Parquet, ORC, SequenceFile)] Debezium Avro Format: Add FormatOption to Optionally Skip emitting UPDATE_BEFORE Rows [flink]
klam-shop opened a new pull request, #26159:
URL: https://github.com/apache/flink/pull/26159
…
<!--
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
Add a Format Option to the Debezium Format to optionally skip emitting the UPDATE_BEFORE Rows when deserializing a Debezium message with op='u'.
This is helpful for Flink SQL applications that want to operate in UPSERT (ChangelogMode=[I,UA,D]) mode and save on processing the UPDATE_BEFORE Rows since the downstream sinks can handle it.
Note no changes are required for the encoder since Flink [encodes UPDATE_BEFORE and UPDATE_AFTER as DELETE and INSERT Debezium messages.](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/table/formats/debezium/)
## Brief change log
- Add `ENABLE_UPSERT_MODE` ConfigOption to `DebeziumAvroFormatFactory`
- Update `DebeziumAvroDeserializationSchema` to handle the new option properly and skip emitting UPDATE_BEFORE when the option is enabled
- Add test coverage to `DebeziumAvroSerDeSchemaTest`
## Verifying this change
This change added tests and can be verified as follows:
- Added `testUpdateDataDeserializationWithUpsertMode` that deserializes an update and checks no -U is emitted when upsert mode is enabled.
- Added `testSeDeSchemaWithUpsertMode` to test the format factory flag functionality, and changelog mode is correctly set.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): /no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37327] [Formats (JSON, Avro, Parquet, ORC, SequenceFile)] Debezium Avro Format: Add FormatOption to Optionally Skip emitting UPDATE_BEFORE Rows [flink]
Posted by "TiansuYu (via GitHub)" <gi...@apache.org>.
TiansuYu commented on PR #26159:
URL: https://github.com/apache/flink/pull/26159#issuecomment-5407444707
Hi @klam-shop, this contribution is very useful for our use case. Is there anything I can do to help this getting merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37327] [Formats (JSON, Avro, Parquet, ORC, SequenceFile)] Debezium Avro Format: Add FormatOption to Optionally Skip emitting UPDATE_BEFORE Rows [flink]
Posted by "klam-shop (via GitHub)" <gi...@apache.org>.
klam-shop commented on PR #26159:
URL: https://github.com/apache/flink/pull/26159#issuecomment-2672553872
@AHeise any chance you have bandwidth to review this PR, or can help finding a committer to review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37327] [Formats (JSON, Avro, Parquet, ORC, SequenceFile)] Debezium Avro Format: Add FormatOption to Optionally Skip emitting UPDATE_BEFORE Rows [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26159:
URL: https://github.com/apache/flink/pull/26159#issuecomment-2993968627
This PR has been closed since it has not had any activity in 120 days.
If you feel like this was a mistake, or you would like to continue working on it,
please feel free to re-open the PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37327] [Formats (JSON, Avro, Parquet, ORC, SequenceFile)] Debezium Avro Format: Add FormatOption to Optionally Skip emitting UPDATE_BEFORE Rows [flink]
Posted by "klam-shop (via GitHub)" <gi...@apache.org>.
klam-shop commented on PR #26159:
URL: https://github.com/apache/flink/pull/26159#issuecomment-2671730447
Thanks for your review @davidradl ! Updated s/Rows/rows/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37327] [Formats (JSON, Avro, Parquet, ORC, SequenceFile)] Debezium Avro Format: Add FormatOption to Optionally Skip emitting UPDATE_BEFORE Rows [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26159:
URL: https://github.com/apache/flink/pull/26159#issuecomment-2900060181
This PR is being marked as stale since it has not had any activity in the last 90 days.
If you would like to keep this PR alive, please leave a comment asking for a review.
If the PR has merge conflicts, update it with the latest from the base branch.
If you are having difficulty finding a reviewer, please reach out to the
community, contact details can be found here: https://flink.apache.org/what-is-flink/community/
If this PR is no longer valid or desired, please feel free to close it.
If no activity occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37327] [Formats (JSON, Avro, Parquet, ORC, SequenceFile)] Debezium Avro Format: Add FormatOption to Optionally Skip emitting UPDATE_BEFORE Rows [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26159:
URL: https://github.com/apache/flink/pull/26159#issuecomment-2660255145
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7538c1b60c950d00d272b4dbcf6ee923336dca7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7538c1b60c950d00d272b4dbcf6ee923336dca7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7538c1b60c950d00d272b4dbcf6ee923336dca7e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37327] [Formats (JSON, Avro, Parquet, ORC, SequenceFile)] Debezium Avro Format: Add FormatOption to Optionally Skip emitting UPDATE_BEFORE Rows [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26159:
URL: https://github.com/apache/flink/pull/26159#discussion_r1963417984
##########
docs/content.zh/docs/connectors/table/formats/debezium.md:
##########
@@ -320,6 +320,13 @@ Flink 提供了 `debezium-avro-confluent` 和 `debezium-json` 两种 format 来
<td>String</td>
<td>Bearer auth token for Schema Registry</td>
</tr>
+ <tr>
+ <td><h5>debezium-avro-confluent.upsert-mode</h5></td>
+ <td>optional</td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Boolean</td>
+ <td>In upsert mode the format will not produce UPDATE_BEFORE Rows from Debezium op='u' change events when deserializing.</td>
Review Comment:
nit: Rows ->rows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37327] [Formats (JSON, Avro, Parquet, ORC, SequenceFile)] Debezium Avro Format: Add FormatOption to Optionally Skip emitting UPDATE_BEFORE Rows [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #26159: [FLINK-37327] [Formats (JSON, Avro, Parquet, ORC, SequenceFile)] Debezium Avro Format: Add FormatOption to Optionally Skip emitting UPDATE_BEFORE Rows
URL: https://github.com/apache/flink/pull/26159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37327] [Formats (JSON, Avro, Parquet, ORC, SequenceFile)] Debezium Avro Format: Add FormatOption to Optionally Skip emitting UPDATE_BEFORE Rows [flink]
Posted by "klam-shop (via GitHub)" <gi...@apache.org>.
klam-shop commented on PR #26159:
URL: https://github.com/apache/flink/pull/26159#issuecomment-5411297938
Hi @TiansuYu, this PR needs a review from a Flink committer. It's also a bit older so probably needs a rebase. I'm not currently working on Flink, so feel free to take over the PR if you like
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org