You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "danny0405 (via GitHub)" <gi...@apache.org> on 2026/08/28 14:48:18 UTC
[PR] fix: preserve committed logs after a failed leading write [hudi]
danny0405 opened a new pull request, #19786:
URL: https://github.com/apache/hudi/pull/19786
### Describe the issue this Pull Request addresses
With MOR tables using the bucket index and non-blocking concurrency control, a failed first write can leave an uncommitted log file as the earliest physical file in a file group. The file group is then anchored at that uncommitted instant, causing the committed slice APIs to discard the entire slice, including later committed logs.
Closes #19774
### Summary and Changelog
This change constructs log-only file groups around the earliest completed log so that a failed leading log cannot hide later committed data.
- Add log files to `HoodieFileGroup` as a batch sorted by delta-commit time.
- When a file group has no existing base or compaction slice, establish the earliest completed log instant as the initial slice base.
- Route earlier pending logs through the existing pending-log rule while leaving `isFileSliceCommitted` unchanged.
- Update filesystem-view construction to use the batch API.
- Add model and filesystem-view regressions covering out-of-order input and the MOR/NBCC sequence with an inflight first log followed by two completed logs.
- No code was copied from another project.
### Impact
Snapshot readers retain later committed logs when the earliest physical log in a file group belongs to a failed, uncommitted write. Existing base-file and pending-compaction slices remain authoritative, pending-only groups retain their existing behavior, and V8+ file-level filtering continues to remove the uncommitted log itself.
The internal file-group construction API now batches log files through `HoodieFileGroup#addLogFiles`. There are no storage-format, configuration, or expected performance changes.
### Risk Level
medium
This changes file-slice construction in a shared filesystem-view path. The risk is mitigated by the existing pre-V8 and V8+ slicing coverage plus the new model and filesystem-view regressions. Validation completed with:
- `mvn -pl hudi-hadoop-common -am -DskipShade -DskipITs -Dcheckstyle.skip -Drat.skip -Dtest=TestHoodieFileGroup -Dsurefire.failIfNoSpecifiedTests=false test`
- `mvn -pl hudi-hadoop-common -am -DskipShade -DskipITs -Dcheckstyle.skip -Drat.skip -Dtest=TestHoodieTableFileSystemView -Dsurefire.failIfNoSpecifiedTests=false test`
Together these ran 59 tests with no failures or errors.
### Documentation Update
none. This is a correctness fix with no new feature, configuration, default, or storage-format change.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: preserve committed logs after a failed leading write [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed pull request #19786: fix: preserve committed logs after a failed leading write
URL: https://github.com/apache/hudi/pull/19786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: preserve committed logs after a failed leading write [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19786:
URL: https://github.com/apache/hudi/pull/19786#issuecomment-5454134012
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19786?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 70.46%. Comparing base ([`816bd96`](https://app.codecov.io/gh/apache/hudi/commit/816bd96a583c29aafa18982ab8d30187eb48490a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`0ccbd4c`](https://app.codecov.io/gh/apache/hudi/commit/0ccbd4c83eef58b053de5736843d1d9f6b5b86d6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19786 +/- ##
============================================
- Coverage 70.46% 70.46% -0.01%
- Complexity 30513 30721 +208
============================================
Files 2540 2540
Lines 141413 142591 +1178
Branches 17123 17186 +63
============================================
+ Hits 99646 100475 +829
- Misses 34504 34835 +331
- Partials 7263 7281 +18
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/19786/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19786/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.17% <100.00%> (+0.02%)` | :arrow_up: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19786/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.35% <ø> (-0.27%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19786/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.65% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19786/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.51% <ø> (ø)` | |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19786/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.51% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19786/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.06% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19786/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `66.83% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19786/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.54% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19786/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.24% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19786/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.57% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19786/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.81% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19786/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19786/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19786/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.04% <100.00%> (+0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19786?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [.../org/apache/hudi/common/model/HoodieFileGroup.java](https://app.codecov.io/gh/apache/hudi/pull/19786?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fmodel%2FHoodieFileGroup.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUZpbGVHcm91cC5qYXZh) | `100.00% <100.00%> (ø)` | |
| [...common/table/view/AbstractTableFileSystemView.java](https://app.codecov.io/gh/apache/hudi/pull/19786?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fview%2FAbstractTableFileSystemView.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvQWJzdHJhY3RUYWJsZUZpbGVTeXN0ZW1WaWV3LmphdmE=) | `88.11% <100.00%> (-0.02%)` | :arrow_down: |
... and [5 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19786/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org