You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "junmuz (via GitHub)" <gi...@apache.org> on 2026/09/01 12:26:06 UTC

[PR] [core] Support changelog-producer.ignore-update-before and changelog-producer.ignore-delete options [paimon]

junmuz opened a new pull request, #9531:
URL: https://github.com/apache/paimon/pull/9531

   
   Add two new configurations for lookup and full-compaction changelog producers to filter out UPDATE_BEFORE (-U) and DELETE (-D) records from changelog files.
   
   ### Purpose
   
   When using lookup or full-compaction changelog producers, the generated changelog files contain complete CDC records including UPDATE_BEFORE (-U) and DELETE (-D) entries. However, many downstream consumers don't need these retraction records:
     - Upsert sinks (e.g. databases with primary keys) overwrite by key and don't need -U records to correctly apply updates. This is quite important if you are migrating away from Kafka. The existing `ignore-update-before` option is only applicable for data writes.
     - `change log-producer.ignore-delete` was added to keep it in sync with `ignore-delete` pattern.
     - Storage-constrained environments where changelog files contribute significant overhead — filtering unnecessary row kinds reduces changelog file size and I/O.
   
   These options give users fine-grained control over what gets included in the changelog output.
   
   Note: These options should not be used when downstream consumers rely on retraction semantics (e.g. Flink retract-based aggregations or joins), as missing -U or -D records will produce incorrect results.
   
   ### Tests
   - Existing ChangelogMergeTreeRewriterTest and LookupMergeTreeCompactRewriterTest pass with updated constructor signatures


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Support changelog-producer.ignore-update-before and changelog-producer.ignore-delete options [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9531:
URL: https://github.com/apache/paimon/pull/9531


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Support changelog-producer.ignore-update-before and changelog-producer.ignore-delete options [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9531:
URL: https://github.com/apache/paimon/pull/9531#issuecomment-5505552269

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org