You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/10/09 07:39:49 UTC
[PR] fix(writer): do not drop MOR records when the operation field is enabled [hudi]
yihua opened a new pull request, #20261:
URL: https://github.com/apache/hudi/pull/20261
### Describe the issue this Pull Request addresses
closes #20260
part of #20257
With `hoodie.allow.operation.metadata.field=true`, MOR upserts and deletes from writers that do not set a record operation (Spark datasource, Spark SQL, Java client) NPE on every record in `HoodieAppendHandle.populateMetadataFields` on table versions 6, 8 and 10. The failure is swallowed under the default `hoodie.write.ignore.failed=true`, and `close()` drops the all-failed write status, so the deltacommit succeeds without the records. The NPE dates from #3285 (0.9.0), and the dropped status from #2300 (0.7.0).
### Summary and Changelog
MOR writes with the operation field enabled now land in the log files, and record failures in an append handle reach the caller.
- `populateMetadataFields` derives `U`/`I` from the record location when the record has no operation. Log compaction keeps the stored value.
- `writeRecord` sends a delete to the delete block unless it carries both an operation and a row.
- `close()` returns the write status of a handle that failed without appending a block, and sets `totalWriteErrors` on the stats so table services fail on record errors.
- Write stats without a path are skipped in `CommitUtils.buildMetadata` and `HoodieTablePreCommitFileSystemView`, and `CoalescingPartitioner` maps a null key to partition 0.
- Tests: `TestMORDataSource` (table versions 6, 8 and 10, upsert, delete, inline compaction; red on master), plus unit tests in `TestHoodieNativeLogAppendHandle`, `TestCommitUtils`, `TestHoodieTablePreCommitFileSystemView` and `TestCoalescingPartitioner`.
### Impact
Fixes silent data loss for MOR writes with the operation field enabled from non-Flink writers. Log compaction with record failures now fails instead of committing without those records.
### Risk Level
medium: writes that used to lose records silently now report the failures, so Spark datasource, SQL and Streamer writes with failed records fail or roll back unless configured to commit despite errors. Spark and Java log compaction now fails on any record failure instead of committing without those records (Flink log compaction still commits, as its error handling is a no-op). Write stats without a path are dropped when commit metadata is assembled and skipped in pre-commit validation.
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(writer): do not drop MOR records when the operation field is enabled [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20261:
URL: https://github.com/apache/hudi/pull/20261#issuecomment-6077505771
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20261?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `50.00000%` with `11 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 72.88%. Comparing base ([`308f78e`](https://app.codecov.io/gh/apache/hudi/commit/308f78e71deabf7a9eec4b0a2ea26b1360658030?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`4549cf5`](https://app.codecov.io/gh/apache/hudi/commit/4549cf5d1101b47b69f641a313a46e19c0c90861?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20261?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://app.codecov.io/gh/apache/hudi/pull/20261?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2FHoodieAppendHandle.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | 37.50% | [5 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20261?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...table/view/HoodieTablePreCommitFileSystemView.java](https://app.codecov.io/gh/apache/hudi/pull/20261?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fview%2FHoodieTablePreCommitFileSystemView.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvSG9vZGllVGFibGVQcmVDb21taXRGaWxlU3lzdGVtVmlldy5qYXZh) | 0.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20261?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (308f78e) and HEAD (4549cf5). Click for more details.
>
> <details><summary>HEAD has 3 uploads less than BASE</summary>
>
>| Flag | BASE (308f78e) | HEAD (4549cf5) |
>|------|------|------|
>|spark-java-tests|3|1|
>|spark-scala-tests|2|1|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20261 +/- ##
============================================
- Coverage 80.62% 72.88% -7.75%
+ Complexity 35055 31796 -3259
============================================
Files 2552 2552
Lines 143400 143416 +16
Branches 17454 17462 +8
============================================
- Hits 115615 104522 -11093
- Misses 19867 31409 +11542
+ Partials 7918 7485 -433
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/20261/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20261/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.74% <66.66%> (-2.43%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20261/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.16% <47.36%> (-6.66%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20261/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.96% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20261/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.54% <ø> (-35.51%)` | :arrow_down: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20261/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.19% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20261/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.80% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20261/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `67.98% <ø> (-3.14%)` | :arrow_down: |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20261/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.16% <ø> (-0.03%)` | :arrow_down: |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20261/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `80.91% <ø> (-0.52%)` | :arrow_down: |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20261/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.53% <ø> (-4.88%)` | :arrow_down: |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20261/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20261/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/20261/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [flink-integration-tests](https://app.codecov.io/gh/apache/hudi/pull/20261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.52% <47.61%> (-0.01%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.95% <28.57%> (-0.03%)` | :arrow_down: |
| [integration-tests](https://app.codecov.io/gh/apache/hudi/pull/20261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `13.45% <22.72%> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/20261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.69% <40.90%> (-0.01%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/20261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.20% <27.27%> (-20.48%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/20261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `35.02% <27.27%> (-12.50%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/20261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `36.89% <27.27%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20261?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...rg/apache/hudi/io/HoodieInlineLogAppendHandle.java](https://app.codecov.io/gh/apache/hudi/pull/20261?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2FHoodieInlineLogAppendHandle.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUlubGluZUxvZ0FwcGVuZEhhbmRsZS5qYXZh) | `81.88% <100.00%> (-0.73%)` | :arrow_down: |
| [...rg/apache/hudi/io/HoodieNativeLogAppendHandle.java](https://app.codecov.io/gh/apache/hudi/pull/20261?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2FHoodieNativeLogAppendHandle.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU5hdGl2ZUxvZ0FwcGVuZEhhbmRsZS5qYXZh) | `96.11% <100.00%> (ø)` | |
| [.../org/apache/hudi/client/CoalescingPartitioner.java](https://app.codecov.io/gh/apache/hudi/pull/20261?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fclient%2FCoalescingPartitioner.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0NvYWxlc2NpbmdQYXJ0aXRpb25lci5qYXZh) | `100.00% <100.00%> (ø)` | |
| [.../java/org/apache/hudi/common/util/CommitUtils.java](https://app.codecov.io/gh/apache/hudi/pull/20261?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Futil%2FCommitUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvQ29tbWl0VXRpbHMuamF2YQ==) | `84.05% <100.00%> (-1.02%)` | :arrow_down: |
| [...table/view/HoodieTablePreCommitFileSystemView.java](https://app.codecov.io/gh/apache/hudi/pull/20261?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fview%2FHoodieTablePreCommitFileSystemView.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvSG9vZGllVGFibGVQcmVDb21taXRGaWxlU3lzdGVtVmlldy5qYXZh) | `88.88% <0.00%> (-11.12%)` | :arrow_down: |
| [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://app.codecov.io/gh/apache/hudi/pull/20261?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2FHoodieAppendHandle.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `75.17% <37.50%> (-6.92%)` | :arrow_down: |
... and [553 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/20261/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org