You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "fhan688 (via GitHub)" <gi...@apache.org> on 2026/09/14 08:51:06 UTC
[PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
fhan688 opened a new pull request, #19948:
URL: https://github.com/apache/hudi/pull/19948
### Describe the issue this Pull Request addresses
Part of #19286.
For pre-8 tables using a layout V1 timeline, Spark incremental queries can use the 1.x reader with completion-time query boundaries. The query analyzer converts these boundaries into a requested-time `InstantRange`.
When a MOR incremental query falls back to a full-table scan, the latest file slice can contain log records beyond the requested end instant. The Spark MOR file-group reader paths did not receive the analyzed `InstantRange`, so an out-of-range update could participate in record merging and mask the latest in-range version of the same key.
### Summary and Changelog
This PR propagates the incremental query's `InstantRange` to the MOR file-group reader before record merging.
- Exposes the analyzed `InstantRange` from `MergeOnReadIncrementalRelationV2`.
- Propagates the range through the default HadoopFsRelation/file-format path.
- Propagates the same range through the legacy `HoodieMergeOnReadRDDV2` path.
- Passes the range into the Spark reader context so base and log records are filtered before merging.
- Keeps snapshot, V1 incremental, and other unaffected paths unchanged by defaulting to an empty range.
- Adds regression coverage for V6 MOR tables read with the V8 incremental reader, including archived instants, full-table-scan fallback, and an out-of-range log update.
- Verifies both the default HadoopFsRelation path and the legacy RDD path.
### Impact
Fixes incremental read correctness for MOR pre-8 tables when the V8 reader falls back to a full-table scan.
There are no storage format, configuration, or public API changes. Other read paths continue to use an empty `InstantRange` and retain their existing behavior.
### Risk Level
Low.
The change is limited to MOR incremental reads with an analyzed instant range. Regression tests cover both Spark file-group reader entry paths and confirm that a physically present log update after the query end instant is excluded before merging.
Verification:
- `TestIncrementalReadWithFileGroupReader`: 9 tests passed, 0 failures, 0 errors.
- Relevant modules compile successfully with 0 scalastyle errors.
### Documentation Update
None. This is an internal correctness fix with no new user-facing configuration or API.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 closed pull request #19948: fix(spark): propagate incremental instant range to MOR file group reader
URL: https://github.com/apache/hudi/pull/19948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5775223716
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090",
"triggerID" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"triggerType" : "PUSH"
}, {
"hash" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17212",
"triggerID" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"triggerType" : "PUSH"
}, {
"hash" : "a495ebb4309215ebaaa8cf2a711376bd9dc7d391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17213",
"triggerID" : "a495ebb4309215ebaaa8cf2a711376bd9dc7d391",
"triggerType" : "PUSH"
}, {
"hash" : "d0cbab78be803f712b7b708f6ba47a56d8bcd1ec",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17232",
"triggerID" : "d0cbab78be803f712b7b708f6ba47a56d8bcd1ec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d0cbab78be803f712b7b708f6ba47a56d8bcd1ec Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17232)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on code in PR #19948:
URL: https://github.com/apache/hudi/pull/19948#discussion_r4056406710
##########
hudi-common/src/main/java/org/apache/hudi/core/read/BaseHoodieTableFileIndex.java:
##########
@@ -429,6 +430,9 @@ protected List<PartitionPath> listPartitionPaths(List<String> relativePartitionP
}
private boolean isBeforeTimelineStarts() {
+ if (START_COMMIT_EARLIEST.equalsIgnoreCase(incrementalQueryStartTime.get())) {
Review Comment:
> `isBeforeTimelineStarts` means the start insant might be archived, the semantics does not equal to reading from earliest.
Good point. earliest is a sentinel rather than an instant before the active timeline. I’ll keep the special handling but move it out of `isBeforeTimelineStarts`, so the method retains its original semantics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5661805496
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090",
"triggerID" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5b8ae8573d4680549ee1adea78199759e4c7e047 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5773883255
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090",
"triggerID" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"triggerType" : "PUSH"
}, {
"hash" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17212",
"triggerID" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"triggerType" : "PUSH"
}, {
"hash" : "a495ebb4309215ebaaa8cf2a711376bd9dc7d391",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17213",
"triggerID" : "a495ebb4309215ebaaa8cf2a711376bd9dc7d391",
"triggerType" : "PUSH"
}, {
"hash" : "d0cbab78be803f712b7b708f6ba47a56d8bcd1ec",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17232",
"triggerID" : "d0cbab78be803f712b7b708f6ba47a56d8bcd1ec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a495ebb4309215ebaaa8cf2a711376bd9dc7d391 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17213)
* d0cbab78be803f712b7b708f6ba47a56d8bcd1ec Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17232)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5662829201
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090",
"triggerID" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5b8ae8573d4680549ee1adea78199759e4c7e047 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5747172373
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090",
"triggerID" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"triggerType" : "PUSH"
}, {
"hash" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17212",
"triggerID" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5b8ae8573d4680549ee1adea78199759e4c7e047 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090)
* d64fb301bc673b3d2339d1fb8fb012d7d07401fd Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17212)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5773806834
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090",
"triggerID" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"triggerType" : "PUSH"
}, {
"hash" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17212",
"triggerID" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"triggerType" : "PUSH"
}, {
"hash" : "a495ebb4309215ebaaa8cf2a711376bd9dc7d391",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17213",
"triggerID" : "a495ebb4309215ebaaa8cf2a711376bd9dc7d391",
"triggerType" : "PUSH"
}, {
"hash" : "d0cbab78be803f712b7b708f6ba47a56d8bcd1ec",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d0cbab78be803f712b7b708f6ba47a56d8bcd1ec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a495ebb4309215ebaaa8cf2a711376bd9dc7d391 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17213)
* d0cbab78be803f712b7b708f6ba47a56d8bcd1ec UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5747115369
cc @cshuo can you enforce this PR to fix the timetravel query issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19948:
URL: https://github.com/apache/hudi/pull/19948#discussion_r4056085063
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV2.scala:
##########
@@ -85,6 +87,7 @@ case class MergeOnReadIncrementalRelationV2(override val sqlContext: SQLContext,
mergeType = mergeType,
fileSplits = fileSplits,
includedInstantTimeSet = Option(includedCommits.map(_.requestedTime).toSet),
+ instantRangeOpt = queryContext.getInstantRange,
Review Comment:
🤖 This passes the range on the non-full-scan path too, and on this legacy RDD path `requiredFilters` is `Seq.empty`, so before this change base rows and pre-range log blocks were merged first and only then post-filtered by `includedInstantTimeSet`. With EVENT_TIME_ORDERING (or a partial-update payload), an in-range log record with a lower ordering value than the out-of-range base row used to be suppressed by the merge but will now be emitted on its own — is that the intended incremental semantics here, or should the range be limited to the `fullTableScan` case on this path?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV2.scala:
##########
@@ -45,6 +46,7 @@ import scala.collection.immutable
trait MergeOnReadIncrementalRelation {
def listFileSplits(partitionFilters: Seq[Expression], dataFilters: Seq[Expression]): Map[InternalRow, Seq[FileSlice]]
def getRequiredFilters: Seq[Filter]
+ def getInstantRange: HOption[InstantRange] = HOption.empty()
Review Comment:
🤖 nit: giving `getInstantRange` a default `HOption.empty()` in the trait means any other implementation (e.g. V1) silently opts out of range filtering. Have you considered leaving it abstract like `getRequiredFilters`, so each relation makes the choice explicit?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-common/src/main/java/org/apache/hudi/core/read/BaseHoodieTableFileIndex.java:
##########
@@ -429,6 +430,9 @@ protected List<PartitionPath> listPartitionPaths(List<String> relativePartitionP
}
private boolean isBeforeTimelineStarts() {
+ if (START_COMMIT_EARLIEST.equalsIgnoreCase(incrementalQueryStartTime.get())) {
Review Comment:
🤖 nit: a one-line comment on why the earliest marker short-circuits here (it's not a real timestamp, so the completion/requested-time comparisons below would be meaningless) would help — could also be pulled into a small `isStartCommitEarliest(...)` helper next to `START_COMMIT_EARLIEST` in `IncrementalQueryAnalyzer` if that check is repeated elsewhere.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5662038840
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19948?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 79.90%. Comparing base ([`648996f`](https://app.codecov.io/gh/apache/hudi/commit/648996f002c3c9e24a1e2bf53f97fe03e8bc4c81?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5b8ae85`](https://app.codecov.io/gh/apache/hudi/commit/5b8ae8573d4680549ee1adea78199759e4c7e047?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19948 +/- ##
============================================
- Coverage 80.20% 79.90% -0.31%
- Complexity 34661 34901 +240
============================================
Files 2546 2546
Lines 142452 143806 +1354
Branches 17330 17575 +245
============================================
+ Hits 114260 114901 +641
- Misses 20281 20943 +662
- Partials 7911 7962 +51
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/19948/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19948/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.52% <ø> (-0.32%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19948/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.44% <ø> (-0.84%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19948/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.57% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19948/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.27% <ø> (-0.01%)` | :arrow_down: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19948/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.17% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19948/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `69.99% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19948/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.02% <ø> (-0.80%)` | :arrow_down: |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19948/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.99% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19948/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.61% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19948/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.34% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19948/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `80.99% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19948/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19948/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19948/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.90% <ø> (-0.03%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19948?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/BaseSparkInternalRowReaderContext.java](https://app.codecov.io/gh/apache/hudi/pull/19948?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FBaseSparkInternalRowReaderContext.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0Jhc2VTcGFya0ludGVybmFsUm93UmVhZGVyQ29udGV4dC5qYXZh) | `88.88% <ø> (-0.59%)` | :arrow_down: |
| [...hudi/SparkFileFormatInternalRowReaderContext.scala](https://app.codecov.io/gh/apache/hudi/pull/19948?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FSparkFileFormatInternalRowReaderContext.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL1NwYXJrRmlsZUZvcm1hdEludGVybmFsUm93UmVhZGVyQ29udGV4dC5zY2FsYQ==) | `74.53% <ø> (-0.35%)` | :arrow_down: |
| [...rg/apache/hudi/HoodieHadoopFsRelationFactory.scala](https://app.codecov.io/gh/apache/hudi/pull/19948?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FHoodieHadoopFsRelationFactory.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9Ib29kaWVIYWRvb3BGc1JlbGF0aW9uRmFjdG9yeS5zY2FsYQ==) | `82.64% <ø> (-0.08%)` | :arrow_down: |
| [...scala/org/apache/hudi/HoodieMergeOnReadRDDV2.scala](https://app.codecov.io/gh/apache/hudi/pull/19948?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FHoodieMergeOnReadRDDV2.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9Ib29kaWVNZXJnZU9uUmVhZFJERFYyLnNjYWxh) | `79.28% <ø> (-0.25%)` | :arrow_down: |
| [...apache/hudi/MergeOnReadIncrementalRelationV2.scala](https://app.codecov.io/gh/apache/hudi/pull/19948?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FMergeOnReadIncrementalRelationV2.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9NZXJnZU9uUmVhZEluY3JlbWVudGFsUmVsYXRpb25WMi5zY2FsYQ==) | `86.40% <ø> (ø)` | |
| [...parquet/HoodieFileGroupReaderBasedFileFormat.scala](https://app.codecov.io/gh/apache/hudi/pull/19948?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fexecution%2Fdatasources%2Fparquet%2FHoodieFileGroupReaderBasedFileFormat.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL2V4ZWN1dGlvbi9kYXRhc291cmNlcy9wYXJxdWV0L0hvb2RpZUZpbGVHcm91cFJlYWRlckJhc2VkRmlsZUZvcm1hdC5zY2FsYQ==) | `84.13% <ø> (-0.11%)` | :arrow_down: |
... and [19 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19948/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5747694277
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090",
"triggerID" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"triggerType" : "PUSH"
}, {
"hash" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17212",
"triggerID" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d64fb301bc673b3d2339d1fb8fb012d7d07401fd Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17212)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19948:
URL: https://github.com/apache/hudi/pull/19948#discussion_r4007080260
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -69,22 +70,33 @@ import scala.collection.JavaConverters._
* schema cannot be recovered from a HoodieSchema round-trip (#18739 sub-task 4).
* Kept Spark-side so the engine-neutral schema model stays free of Spark 4.1
* variant concepts.
+ * @param instantRangeOpt optional requested-time range applied to base and log records before merging
*/
class SparkFileFormatInternalRowReaderContext(baseFileReader: SparkColumnarFileReader,
filters: Seq[Filter],
requiredFilters: Seq[Filter],
storageConfiguration: StorageConfiguration[_],
tableConfig: HoodieTableConfig,
- sparkRequiredSchema: Option[StructType] = None)
- extends BaseSparkInternalRowReaderContext(storageConfiguration, tableConfig, SparkFileFormatInternalRecordContext.apply(tableConfig)) {
+ sparkRequiredSchema: Option[StructType] = None,
+ instantRangeOpt: HOption[InstantRange] = HOption.empty())
+ extends BaseSparkInternalRowReaderContext(storageConfiguration, tableConfig, instantRangeOpt,
+ SparkFileFormatInternalRecordContext.apply(tableConfig)) {
// Java-friendly auxiliary constructor (Scala default args don't generate matching Java overloads).
def this(baseFileReader: SparkColumnarFileReader,
filters: Seq[Filter],
requiredFilters: Seq[Filter],
storageConfiguration: StorageConfiguration[_],
tableConfig: HoodieTableConfig) =
- this(baseFileReader, filters, requiredFilters, storageConfiguration, tableConfig, None)
+ this(baseFileReader, filters, requiredFilters, storageConfiguration, tableConfig, None, HOption.empty())
+
+ def this(baseFileReader: SparkColumnarFileReader,
Review Comment:
🤖 nit: we now have three Java-friendly auxiliary constructors here differing only by which trailing optional params are included. Might be worth a comment above them noting they exist purely to work around Scala default args not generating Java overloads, so future readers don't try to collapse them.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on code in PR #19948:
URL: https://github.com/apache/hudi/pull/19948#discussion_r4043484459
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -55,7 +56,14 @@ public abstract class BaseSparkInternalRowReaderContext extends HoodieReaderCont
protected BaseSparkInternalRowReaderContext(StorageConfiguration<?> storageConfig,
HoodieTableConfig tableConfig,
BaseSparkInternalRecordContext recordContext) {
- super(storageConfig, tableConfig, Option.empty(), Option.empty(), recordContext);
+ this(storageConfig, tableConfig, Option.empty(), recordContext);
+ }
+
+ protected BaseSparkInternalRowReaderContext(StorageConfiguration<?> storageConfig,
Review Comment:
> can we try to extend the existing the constructors instead of adding new ones?
Thanks, makes sense for the protected base constructor. I’ll extend it with `instantRangeOpt` and update the internal subclass/test call sites. I’ll keep the existing Java-friendly constructor on `SparkFileFormatInternalRowReaderContext`, since Java callers rely on that overload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5748559205
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090",
"triggerID" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"triggerType" : "PUSH"
}, {
"hash" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17212",
"triggerID" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"triggerType" : "PUSH"
}, {
"hash" : "a495ebb4309215ebaaa8cf2a711376bd9dc7d391",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17213",
"triggerID" : "a495ebb4309215ebaaa8cf2a711376bd9dc7d391",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d64fb301bc673b3d2339d1fb8fb012d7d07401fd Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17212)
* a495ebb4309215ebaaa8cf2a711376bd9dc7d391 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17213)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5749033890
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090",
"triggerID" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"triggerType" : "PUSH"
}, {
"hash" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17212",
"triggerID" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"triggerType" : "PUSH"
}, {
"hash" : "a495ebb4309215ebaaa8cf2a711376bd9dc7d391",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17213",
"triggerID" : "a495ebb4309215ebaaa8cf2a711376bd9dc7d391",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a495ebb4309215ebaaa8cf2a711376bd9dc7d391 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17213)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19948:
URL: https://github.com/apache/hudi/pull/19948#discussion_r4069990012
##########
hudi-common/src/main/java/org/apache/hudi/core/read/BaseHoodieTableFileIndex.java:
##########
@@ -402,7 +403,10 @@ protected List<PartitionPath> listPartitionPaths(List<String> relativePartitionP
List<String> matchedPartitionPaths;
try {
if (isPartitionedTable()) {
- if (queryType == HoodieTableQueryType.INCREMENTAL && incrementalQueryStartTime.isPresent() && !isBeforeTimelineStarts()) {
+ if (queryType == HoodieTableQueryType.INCREMENTAL
Review Comment:
🤖 nit: this predicate now has four clauses; it might be worth pulling it into a small helper like `canListPartitionsFromIncrementalTimeline()` so the intent ("we have a concrete start instant that is still on the active timeline") reads at a glance.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5747147336
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090",
"triggerID" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"triggerType" : "PUSH"
}, {
"hash" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5b8ae8573d4680549ee1adea78199759e4c7e047 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090)
* d64fb301bc673b3d2339d1fb8fb012d7d07401fd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19948:
URL: https://github.com/apache/hudi/pull/19948#discussion_r4055847399
##########
hudi-common/src/main/java/org/apache/hudi/core/read/BaseHoodieTableFileIndex.java:
##########
@@ -429,6 +430,9 @@ protected List<PartitionPath> listPartitionPaths(List<String> relativePartitionP
}
private boolean isBeforeTimelineStarts() {
+ if (START_COMMIT_EARLIEST.equalsIgnoreCase(incrementalQueryStartTime.get())) {
Review Comment:
`isBeforeTimelineStarts` means the start insant might be archived, the semantics does not equal to reading from earliest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5748532836
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17090",
"triggerID" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"triggerType" : "PUSH"
}, {
"hash" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17212",
"triggerID" : "d64fb301bc673b3d2339d1fb8fb012d7d07401fd",
"triggerType" : "PUSH"
}, {
"hash" : "a495ebb4309215ebaaa8cf2a711376bd9dc7d391",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a495ebb4309215ebaaa8cf2a711376bd9dc7d391",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d64fb301bc673b3d2339d1fb8fb012d7d07401fd Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17212)
* a495ebb4309215ebaaa8cf2a711376bd9dc7d391 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5694455019
please help review, thanks! @yihua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19948:
URL: https://github.com/apache/hudi/pull/19948#discussion_r4043168057
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -55,7 +56,14 @@ public abstract class BaseSparkInternalRowReaderContext extends HoodieReaderCont
protected BaseSparkInternalRowReaderContext(StorageConfiguration<?> storageConfig,
HoodieTableConfig tableConfig,
BaseSparkInternalRecordContext recordContext) {
- super(storageConfig, tableConfig, Option.empty(), Option.empty(), recordContext);
+ this(storageConfig, tableConfig, Option.empty(), recordContext);
+ }
+
+ protected BaseSparkInternalRowReaderContext(StorageConfiguration<?> storageConfig,
Review Comment:
can we try to extend the existing the constructors instead of adding new ones?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19948:
URL: https://github.com/apache/hudi/pull/19948#issuecomment-5661537834
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5b8ae8573d4680549ee1adea78199759e4c7e047",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5b8ae8573d4680549ee1adea78199759e4c7e047 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19948:
URL: https://github.com/apache/hudi/pull/19948#discussion_r4068361527
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV2.scala:
##########
@@ -150,6 +153,13 @@ case class MergeOnReadIncrementalRelationV2(override val sqlContext: SQLContext,
}
}
+ /**
+ * Returns the requested-time range selected by the completion-time query analysis. The file-group
+ * reader needs this in addition to Spark's required filters so that out-of-range log records do
+ * not participate in record merging and mask an earlier in-range version of the same key.
+ */
+ override def getInstantRange: HOption[InstantRange] = queryContext.getInstantRange
Review Comment:
Could we scope the new instant-range propagation to full-table-scan fallback? The reported bug occurs when fallback selects the latest file slices, allowing out-of-range log blocks to participate in merging. This would keep the fix focused on that case and preserve the existing behavior of normal incremental reads.
In `MergeOnReadIncrementalRelationV2`:
```scala
override def getInstantRange: HOption[InstantRange] =
if (fullTableScan) queryContext.getInstantRange
else HOption.empty()
```
Then use the same accessor in `composeRDD`:
```scala
instantRangeOpt = getInstantRange,
```
The HadoopFsRelation factory already calls this accessor, so both reader paths would follow the same condition. We can keep `incrementalSpanRecordFilters` unchanged and retain the fallback regression coverage for both paths and table versions 6/8, alongside the normal incremental-read tests.
This is a scope suggestion for the fallback fix, rather than a claim that file selection guarantees every log block is in range on all normal-read paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #19948:
URL: https://github.com/apache/hudi/pull/19948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): propagate incremental instant range to MOR file group reader [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #19948:
URL: https://github.com/apache/hudi/pull/19948#discussion_r4036689007
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestIncrementalReadWithFileGroupReader.scala:
##########
@@ -123,6 +125,57 @@ class TestIncrementalReadWithFileGroupReader extends SparkClientFunctionalTestHa
assertIncrementalRange(readVersion, instants, 6, 6, Set.empty)
}
+ @ParameterizedTest(name = "V2 MOR full-table scan applies instant range, legacy RDD = {0}")
+ @ValueSource(booleans = Array(false, true))
+ def testV2MorFullTableScanAppliesInstantRange(useLegacyRdd: Boolean): Unit = {
+ val archivalOptions = Map(
+ HoodieArchivalConfig.MIN_COMMITS_TO_KEEP.key -> "2",
+ HoodieArchivalConfig.MAX_COMMITS_TO_KEEP.key -> "5",
+ HoodieCleanConfig.CLEANER_COMMITS_RETAINED.key -> "1",
+ HoodieCleanConfig.AUTO_CLEAN.key -> "false",
+ HoodieMetadataConfig.ENABLE.key -> "false",
+ HoodieWriteConfig.AUTO_UPGRADE_VERSION.key -> "false")
+
+ batches.zipWithIndex.foreach { case ((data, operation), i) =>
+ write(data, "MERGE_ON_READ", 6, operation,
Review Comment:
**Cover full-table-scan fallback on V8 tables**
Could we parameterize this regression test over table versions 6 and 8? The range propagation applies to all V2 MOR incremental reads, while this fallback test only exercises a V6 table. A V8 case using completion-time query boundaries would also verify the conversion to the requested-time range passed to the reader.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestIncrementalReadWithFileGroupReader.scala:
##########
@@ -123,6 +125,57 @@ class TestIncrementalReadWithFileGroupReader extends SparkClientFunctionalTestHa
assertIncrementalRange(readVersion, instants, 6, 6, Set.empty)
}
+ @ParameterizedTest(name = "V2 MOR full-table scan applies instant range, legacy RDD = {0}")
+ @ValueSource(booleans = Array(false, true))
+ def testV2MorFullTableScanAppliesInstantRange(useLegacyRdd: Boolean): Unit = {
+ val archivalOptions = Map(
+ HoodieArchivalConfig.MIN_COMMITS_TO_KEEP.key -> "2",
+ HoodieArchivalConfig.MAX_COMMITS_TO_KEEP.key -> "5",
+ HoodieCleanConfig.CLEANER_COMMITS_RETAINED.key -> "1",
+ HoodieCleanConfig.AUTO_CLEAN.key -> "false",
+ HoodieMetadataConfig.ENABLE.key -> "false",
+ HoodieWriteConfig.AUTO_UPGRADE_VERSION.key -> "false")
+
+ batches.zipWithIndex.foreach { case ((data, operation), i) =>
+ write(data, "MERGE_ON_READ", 6, operation,
+ if (i == 0) SaveMode.Overwrite else SaveMode.Append, archivalOptions)
+ }
+
+ val metaClient = HoodieTableMetaClient.builder()
+ .setConf(storageConf().newInstance()).setBasePath(basePath()).build()
+ assertEquals(6, metaClient.getTableConfig.getTableVersion.versionCode())
+ val archivedInstants = metaClient.getArchivedTimeline.getCommitsTimeline.filterCompletedInstants
+ .getInstants.asScala.toList
+ val activeInstants = metaClient.getActiveTimeline.getCommitsTimeline.filterCompletedInstants
+ .getInstants.asScala.toList
+ val instants = (archivedInstants ++ activeInstants)
+ .map(instant => instant.requestedTime -> instant).toMap.values.toList.sortBy(_.requestedTime)
+
+ assertTrue(archivedInstants.nonEmpty, "The query must include archived instants to force a full-table scan")
+ assertEquals(6, instants.size)
+ assertTrue(instants.last.requestedTime.compareTo(instants(4).requestedTime) > 0,
+ "c6 must be outside the requested range")
+ val (_, logFiles) = listDataFiles()
+ assertEquals(3, logFiles.size, "The out-of-range c6 log must exist physically")
+
+ val readOptions = Map(
+ DataSourceReadOptions.QUERY_TYPE.key -> DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL,
+ DataSourceReadOptions.START_COMMIT.key -> "000",
Review Comment:
**Exercise a bounded start and the `earliest` range variant**
This case only exercises an `EXACT_MATCH` range starting from `"000"`. Consider adding `(c3, c5]` to verify exclusion of older base/log records, and `earliest → c5` to exercise the `CLOSED_CLOSED` range with a null lower boundary. The former should return only `k1=5`, `k2=4`, `k3=5`, and `k4=5`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org