[I] [Feature] Support an update-if-changed merge engine for primary-key tables [fluss]

["litiliu (via GitHub)" <gi...@apache.org>]

litiliu opened a new issue, #4343:
URL: https://github.com/apache/fluss/issues/4343

   ### Search before asking
   
   - [x] I searched in the issues and found nothing similar.
   
   ### Motivation
   
   For a primary-key table using the default write semantics, an upsert produces a new changelog record even when every logical field has the same value as the currently stored row.
   
   This can cause unnecessary KV writes and changelog amplification when upstream data is replayed, retried, or backfilled. It also means that value-identical records replicated between two primary-key tables can continue producing new changelog records instead of becoming a no-op when they return to a table that already contains the same value.
   
   The existing merge engines do not provide the required semantics:
   
   - `FIRST_ROW` ignores every subsequent row for an existing primary key, including legitimate updates.
   - `VERSIONED` requires a version column and accepts rows whose version is equal to the stored version.
   - `AGGREGATION` applies field-level aggregation rather than last-row upsert semantics.
   
   ### Proposed solution
   
   Introduce an update-if-changed merge engine for primary-key tables. The final name could be `DEDUPLICATE` or `UPDATE_IF_CHANGED`.
   
   Its behavior should be:
   
   | Stored row | Incoming operation | Result |
   | --- | --- | --- |
   | Absent | Insert/upsert | Store the incoming row and emit an insert changelog |
   | Present | Every logical field is equal | Keep the stored row and emit no changelog |
   | Present | At least one logical field differs | Store the incoming row and emit the normal update changelog |
   | Present | Delete | Delete the row and emit a delete changelog |
   | Absent | Delete | No-op and emit no changelog |
   
   Equality should be based on logical field values, not raw serialized bytes. In particular:
   
   - Null values should compare using SQL/logical equality suitable for change detection.
   - Rows written with different schema versions should be aligned to a common logical schema before comparison.
   - For partial updates, the incoming update should first be merged into a complete candidate row and then compared with the stored row.
   - Binary values should be compared by content.
   
   The current KV write path already skips an update when a row merger returns the existing `oldValue`. The new merge engine can therefore return `oldValue` when the complete candidate row is logically equal and return the candidate row otherwise.
   
   This engine requires reading the stored value before deciding whether a full-row upsert is unchanged. It therefore cannot use the current WAL full-row fast path that skips old-value lookup. The old-value lookup batching proposed in #4063 can help reduce this cost.
   
   ### Scope and semantics
   
   This is value-based duplicate suppression, not version ordering:
   
   - Replaying a row that is identical to the current stored value is a no-op.
   - Replaying a historical row whose fields differ from the current value is still accepted.
   - The feature does not replace exactly-once delivery or version-based conflict resolution.
   - Existing primary-key table and merge-engine behavior should remain unchanged unless this engine is explicitly configured.
   
   This extends the merge-engine use cases described in #212 without changing the semantics of the existing engines.
   
   ### Anything else?
   
   A primary use case is reducing changelog and downstream lake-storage amplification caused by repeated full-row upserts.
   
   ### Willingness to contribute
   
   - [x] I am willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Support an update-if-changed merge engine for primary-key tables [fluss]

["platinumhamburg (via GitHub)" <gi...@apache.org>]

platinumhamburg closed issue #4343: [Feature] Support an update-if-changed merge engine for primary-key tables
URL: https://github.com/apache/fluss/issues/4343


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org