You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 05:46:23 UTC

[I] Support Write-on-compaction mode when query cdc on MOR tables [hudi]

hudi-bot opened a new issue, #15384:
URL: https://github.com/apache/hudi/issues/15384

   For the case that query cdc on MOR tables, the initial implementation use the `Write-on-indexing`  way to extract the cdc data by merging the base file and log files in-flight.
   
   This ticket wants to support the `Write-on-compaction` way to get the cdc data just by reading the persisted cdc files which are written at the compaction operation.
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-4705
   - Type: New Feature
   
   
   ---
   
   
   ## Comments
   
   24/May/23 06:38;lizhiqiang;Does this Jira have plans for the next step?
   
   [~biyan900116@gmail.com] ;;;
   
   ---
   
   07/Jun/24 16:41;xushiyan;[~lizhiqiang] [~biyan900116@gmail.com] to clarify, CDC for spark works on MOR, just that the implementation is using write-on-indexing strategy (ref: [https://github.com/apache/hudi/blob/master/rfc/rfc-51/rfc-51.md#persisting-cdc-in-mor-write-on-indexing-vs-write-on-compaction)]
   
    
   
   We want to unify the implementation as write-on-compaction, which allows flink writer to work too. (write-on-indexing strategy does not work for flink as explained in the RFC);;;
   
   ---
   
   14/Nov/24 02:35;wangchao316;hi, flink write cdc for MOR , which is not support ?;;;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support Write-on-compaction mode when query cdc on MOR tables [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15384:
URL: https://github.com/apache/hudi/issues/15384#issuecomment-4795359476

   This appears to have been implemented by [HUDI-7911](https://issues.apache.org/jira/browse/HUDI-7911) / PR [#11490](https://github.com/apache/hudi/pull/11490) (commit `87f9fdaee9`, 'Enable cdc log for MOR table'), which adds the write-on-compaction CDC path for MOR tables.
   
   Key pieces in current `master`:
   - `HoodieMergeHandleFactory.getMergeHandleClassesCompaction()` selects `HoodieMergeHandleWithChangeLog` during compaction when CDC is enabled and the writer yields pure log files (gated by `HoodieWriteConfig.isYieldingPureLogForMor()` — true for `BUCKET` and `FLINK_STATE` indexes).
   - `CdcInputFormat.DataLogFileIterator` (Flink) consumes the persisted CDC files for the new `LOG_FILE` CDC inference case, which is the read-side counterpart that this ticket asked for.
   - The unified design is documented in `rfc/rfc-51/rfc-51.md` ('Persisting CDC in MOR: Write-on-indexing vs Write-on-compaction'): write-on-indexing for Spark non-bucket inserts, write-on-compaction for everything else (Flink + Bucket-index Spark).
   
   If this matches what you were tracking, this issue / HUDI-4705 can likely be closed. If there's a remaining gap (e.g., a specific writer/index combination still going through the on-the-fly path), it would help to call it out specifically.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support Write-on-compaction mode when query cdc on MOR tables [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15384: Support Write-on-compaction mode when query cdc on MOR tables
URL: https://github.com/apache/hudi/issues/15384


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org