You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 08:39:32 UTC
[I] Implement mor skip merge for new hudi parquet file format [hudi]
hudi-bot opened a new issue, #16150:
URL: https://github.com/apache/hudi/issues/16150
Skip merge needs to be implemented
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6637
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-6568
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Implement mor skip merge for new hudi parquet file format [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16150: Implement mor skip merge for new hudi parquet file format
URL: https://github.com/apache/hudi/issues/16150
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Implement mor skip merge for new hudi parquet file format [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16150:
URL: https://github.com/apache/hudi/issues/16150#issuecomment-4795544669
Triage: this looks fixed on current `master`. The new file-group-reader-based Parquet file format (`HoodieFileGroupReaderBasedFileFormat`) now supports `skip_merge` end-to-end:
- `HoodieReaderConfig.REALTIME_SKIP_MERGE = "skip_merge"` is an accepted value of `MERGE_TYPE` (`hudi-common/.../HoodieReaderConfig.java`).
- `DefaultFileGroupRecordBufferLoader.getRecordBuffer` detects `isSkipMerge` and instantiates `UnmergedFileGroupRecordBuffer`, which preserves base+log records without merging (`hudi-common/.../buffer/DefaultFileGroupRecordBufferLoader.java`, `UnmergedFileGroupRecordBuffer.java`).
- `HoodieFileGroupReader` (line ~114) disables record-position merging when skip-merge is on.
- The Spark side flows `MERGE_TYPE=skip_merge` via `DataSourceReadOptions.REALTIME_SKIP_MERGE_OPT_VAL`/`HoodieReaderConfig.MERGE_TYPE`, and tests in `TestMORDataSource.scala` (e.g. lines 314, 367, 548, 637) and `TestHoodieFileGroupReaderBase.java` (line ~1021) cover this path.
If there's a specific scenario where skip merge still misbehaves with the new format, please share a repro — otherwise this can likely be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org