You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 03:36:16 UTC
[I] Able to merge with base file for incremental pulling on MOR table [hudi]
hudi-bot opened a new issue, #14678:
URL: https://github.com/apache/hudi/issues/14678
With the implementation of HUDI-920, if the commits include uncompacted log files, hudi will read those log files. But for some use cases, the log files only include partial data, so the users want to merge with the base file first then return the query results. We need to provide an option for the user to choose whether to merge with the base file.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-1515
- Type: New Feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Able to merge with base file for incremental pulling on MOR table [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed issue #14678: Able to merge with base file for incremental pulling on MOR table
URL: https://github.com/apache/hudi/issues/14678
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Able to merge with base file for incremental pulling on MOR table [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #14678:
URL: https://github.com/apache/hudi/issues/14678#issuecomment-5116136636
This issue was reviewed as part of the JIRA-migrated backlog triage.
**Findings:** this is available. Incremental queries take an explicit format option - `hoodie.datasource.query.incremental.format` (`hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:70-78`, `sinceVersion` **0.13.0**):
- `latest_state` (default) - returns the latest record values, i.e. log records merged with the base file, which is the behaviour requested here
- `cdc` - returns change-data records only
On MOR the merge itself is done by `HoodieFileGroupReader` driven by `IncrementalQueryAnalyzer` (`hudi-common/src/main/java/org/apache/hudi/common/table/read/`), so a partial log block no longer yields partial rows under `latest_state`.
Closing as fixed. Please reopen if you need merge-with-base semantics in a context this option doesn't cover.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org