You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rahil-c (via GitHub)" <gi...@apache.org> on 2026/10/05 22:34:36 UTC
[PR] fix(metadata): don't fail record index updates when the only completed commit has no schema [hudi]
rahil-c opened a new pull request, #20205:
URL: https://github.com/apache/hudi/pull/20205
### Describe the issue this Pull Request addresses
closes #20204
When the only completed commit of a table is an empty commit written with a null writer schema (for example a Hudi Streamer sync with no input), `HoodieTableMetadataUtil#tryResolveSchemaForTable` throws `HoodieSchemaNotFoundException`, so every later record index update fails with `Failed to generate RLI records for metadata table` and the table can no longer be written.
### Summary and Changelog
`tryResolveSchemaForTable` now returns `Option.empty()` when the table has completed commits but no schema can be resolved, the same result it already gives for a table with no completed commit. Its callers already handle the empty case, and other failures to read the schema are still rethrown. Record index generation now fails fast if it has to read log files without a schema, so deletes in log files can never be dropped from the index silently.
- `HoodieTableMetadataUtil#tryResolveSchemaForTable`: return empty on `HoodieSchemaNotFoundException`.
- `BaseRecordIndexer#convertMetadataToRecordIndexRecords`: throw when no schema is resolved and the commit wrote log files.
- `TestMetadataUtilRLIandSIRecordGeneration`: on a MOR table with the record index, commits an empty upsert with the null schema, then an insert, an update and a delete, and checks the record index per key. It runs with streaming metadata writes on and off; with them off it fails without the fix on the `HoodieSchemaNotFoundException` above.
- `TestRecordIndexer`: covers the log file check.
### Impact
A table whose first commit is empty and schemaless can be written again. No API or config change.
### Risk Level
low. Only the schema-not-found case changes, and it now behaves like the existing no-commit case.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(metadata): don't fail record index updates when the only completed commit has no schema [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20205:
URL: https://github.com/apache/hudi/pull/20205#issuecomment-6004846047
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20205?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `4 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 77.85%. Comparing base ([`2fc6486`](https://app.codecov.io/gh/apache/hudi/commit/2fc6486f9b298f40a232c062da3bb461de52caa3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`4a48619`](https://app.codecov.io/gh/apache/hudi/commit/4a486190ad8cdf45a2cfb01dfebad3f8d0c8ead8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20205?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../hudi/metadata/index/record/BaseRecordIndexer.java](https://app.codecov.io/gh/apache/hudi/pull/20205?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2Findex%2Frecord%2FBaseRecordIndexer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL2luZGV4L3JlY29yZC9CYXNlUmVjb3JkSW5kZXhlci5qYXZh) | 0.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20205?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../apache/hudi/metadata/HoodieTableMetadataUtil.java](https://app.codecov.io/gh/apache/hudi/pull/20205?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieTableMetadataUtil.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllVGFibGVNZXRhZGF0YVV0aWwuamF2YQ==) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20205?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20205 +/- ##
============================================
- Coverage 77.86% 77.85% -0.01%
- Complexity 33777 33778 +1
============================================
Files 2550 2550
Lines 143117 143121 +4
Branches 17399 17399
============================================
- Hits 111431 111429 -2
- Misses 23597 23601 +4
- Partials 8089 8091 +2
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/20205/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20205/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.50% <0.00%> (-0.01%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20205/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.09% <0.00%> (-0.02%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20205/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.96% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20205/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `66.93% <ø> (ø)` | |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20205/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.18% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20205/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.78% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20205/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `68.29% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20205/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.23% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20205/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.26% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20205/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.50% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20205/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20205/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/20205/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20205/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.97% <0.00%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20205?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [.../hudi/metadata/index/record/BaseRecordIndexer.java](https://app.codecov.io/gh/apache/hudi/pull/20205?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2Findex%2Frecord%2FBaseRecordIndexer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL2luZGV4L3JlY29yZC9CYXNlUmVjb3JkSW5kZXhlci5qYXZh) | `72.53% <0.00%> (-0.52%)` | :arrow_down: |
| [.../apache/hudi/metadata/HoodieTableMetadataUtil.java](https://app.codecov.io/gh/apache/hudi/pull/20205?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieTableMetadataUtil.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllVGFibGVNZXRhZGF0YVV0aWwuamF2YQ==) | `89.35% <0.00%> (-0.18%)` | :arrow_down: |
... and [1 file with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/20205/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(metadata): don't fail record index updates when the only completed commit has no schema [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #20205: fix(metadata): don't fail record index updates when the only completed commit has no schema
URL: https://github.com/apache/hudi/pull/20205
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(metadata): don't fail record index updates when the only completed commit has no schema [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #20205:
URL: https://github.com/apache/hudi/pull/20205#issuecomment-6701606020
Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(metadata): don't fail record index updates when the only completed commit has no schema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20205:
URL: https://github.com/apache/hudi/pull/20205#issuecomment-6008327239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a486190ad8cdf45a2cfb01dfebad3f8d0c8ead8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17468",
"triggerID" : "4a486190ad8cdf45a2cfb01dfebad3f8d0c8ead8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a486190ad8cdf45a2cfb01dfebad3f8d0c8ead8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17468)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(metadata): don't fail record index updates when the only completed commit has no schema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20205:
URL: https://github.com/apache/hudi/pull/20205#issuecomment-6004985624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a486190ad8cdf45a2cfb01dfebad3f8d0c8ead8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17468",
"triggerID" : "4a486190ad8cdf45a2cfb01dfebad3f8d0c8ead8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a486190ad8cdf45a2cfb01dfebad3f8d0c8ead8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17468)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(metadata): don't fail record index updates when the only completed commit has no schema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20205:
URL: https://github.com/apache/hudi/pull/20205#issuecomment-6004879463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a486190ad8cdf45a2cfb01dfebad3f8d0c8ead8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a486190ad8cdf45a2cfb01dfebad3f8d0c8ead8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a486190ad8cdf45a2cfb01dfebad3f8d0c8ead8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org