You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2026/10/15 12:32:54 UTC
[PR] feat(client): add an opt-in to skip inline metadata table services [hudi]
codope opened a new pull request, #20350:
URL: https://github.com/apache/hudi/pull/20350
### Describe the issue this Pull Request addresses
Data table writers run clean, compaction and archival on the metadata table inline: Spark and Java when a write starts (under the table lock), Flink when the coordinator starts and on each metadata table commit. Some jobs must not maintain the metadata table, for example a partition delete run by a table service platform. `hoodie.table.services.enabled=false` does not stop this.
### Summary and Changelog
- New advanced config `hoodie.metadata.inline.table.services.enabled` (default true). When false, writers skip inline metadata table services.
- The check is added in `SparkRDDWriteClient`, `HoodieJavaWriteClient`, `HoodieFlinkTableServiceClient.initMetadataTable` and `FlinkHoodieBackedTableMetadataWriter.commitInternal`.
- This doesn't reuse `hoodie.table.services.enabled` because writers that keep that flag off rely on these paths to compact the metadata table, and data table archival cannot move past the latest metadata compaction.
- Tests: new `TestSparkDeletePartitionTableServices` and `TestStreamWriteOperatorCoordinator#testSkipInlineMetadataTableServices`. Each checks that no metadata clean, compaction or archival happens with the setting off. Removing any of the checks fails a test.
### Impact
None by default. With the setting off, the metadata table is maintained only by other writers or an external table service.
### Risk Level
low
### Documentation Update
Config doc string only.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(client): add an opt-in to skip inline metadata table services [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #20350:
URL: https://github.com/apache/hudi/pull/20350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(client): add an opt-in to skip inline metadata table services [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20350:
URL: https://github.com/apache/hudi/pull/20350#issuecomment-6192442792
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20350?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `72.72727%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.73%. Comparing base ([`75fb5d5`](https://app.codecov.io/gh/apache/hudi/commit/75fb5d5ecd922ed873bcd6bb31382b6630f36e67?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`92f66cf`](https://app.codecov.io/gh/apache/hudi/commit/92f66cf52fff7092d5aaaa558703795027635543?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20350?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...pache/hudi/common/config/HoodieMetadataConfig.java](https://app.codecov.io/gh/apache/hudi/pull/20350?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fconfig%2FHoodieMetadataConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2NvbmZpZy9Ib29kaWVNZXRhZGF0YUNvbmZpZy5qYXZh) | 77.77% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20350?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../org/apache/hudi/client/HoodieJavaWriteClient.java](https://app.codecov.io/gh/apache/hudi/pull/20350?src=pr&el=tree&filepath=hudi-client%2Fhudi-java-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fclient%2FHoodieJavaWriteClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jbGllbnQvSG9vZGllSmF2YVdyaXRlQ2xpZW50LmphdmE=) | 50.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20350?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20350 +/- ##
============================================
- Coverage 80.73% 80.73% -0.01%
+ Complexity 35167 35166 -1
============================================
Files 2553 2553
Lines 143580 143587 +7
Branches 17486 17486
============================================
+ Hits 115925 115929 +4
- Misses 19696 19698 +2
- Partials 7959 7960 +1
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/20350/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20350/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.18% <77.77%> (-0.01%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20350/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.97% <50.00%> (-0.01%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20350/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `86.07% <ø> (-0.01%)` | :arrow_down: |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20350/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.20% <ø> (ø)` | |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20350/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.13% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20350/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.80% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20350/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `71.17% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20350/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.58% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20350/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.52% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20350/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.41% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20350/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20350/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.90% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/20350/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20350/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.95% <72.72%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20350?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/client/HoodieFlinkTableServiceClient.java](https://app.codecov.io/gh/apache/hudi/pull/20350?src=pr&el=tree&filepath=hudi-client%2Fhudi-flink-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fclient%2FHoodieFlinkTableServiceClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZUZsaW5rVGFibGVTZXJ2aWNlQ2xpZW50LmphdmE=) | `72.94% <ø> (-0.32%)` | :arrow_down: |
| [...metadata/FlinkHoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/20350?src=pr&el=tree&filepath=hudi-client%2Fhudi-flink-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FFlinkHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvRmxpbmtIb29kaWVCYWNrZWRUYWJsZU1ldGFkYXRhV3JpdGVyLmphdmE=) | `85.36% <ø> (-0.35%)` | :arrow_down: |
| [...va/org/apache/hudi/client/SparkRDDWriteClient.java](https://app.codecov.io/gh/apache/hudi/pull/20350?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fclient%2FSparkRDDWriteClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrUkREV3JpdGVDbGllbnQuamF2YQ==) | `91.27% <ø> (-0.06%)` | :arrow_down: |
| [.../org/apache/hudi/client/HoodieJavaWriteClient.java](https://app.codecov.io/gh/apache/hudi/pull/20350?src=pr&el=tree&filepath=hudi-client%2Fhudi-java-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fclient%2FHoodieJavaWriteClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jbGllbnQvSG9vZGllSmF2YVdyaXRlQ2xpZW50LmphdmE=) | `83.33% <50.00%> (-0.94%)` | :arrow_down: |
| [...pache/hudi/common/config/HoodieMetadataConfig.java](https://app.codecov.io/gh/apache/hudi/pull/20350?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fconfig%2FHoodieMetadataConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2NvbmZpZy9Ib29kaWVNZXRhZGF0YUNvbmZpZy5qYXZh) | `89.60% <77.77%> (-0.15%)` | :arrow_down: |
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(client): add an opt-in to skip inline metadata table services [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #20350:
URL: https://github.com/apache/hudi/pull/20350#discussion_r4283030152
##########
hudi-client/hudi-java-client/src/main/java/org/apache/hudi/client/HoodieJavaWriteClient.java:
##########
@@ -248,7 +248,9 @@ private void initializeMetadataTable(Option<String> inFlightInstantTimestamp, Ho
if (writer.hasPartitionsStateChanged()) {
metaClient.reloadTableConfig();
}
- writer.performTableServices(inFlightInstantTimestamp);
+ if (config.getMetadataConfig().isInlineTableServicesEnabled()) {
Review Comment:
🤖 non-blocking: It looks like nothing tests this Java client check, since the new tests only cover Spark and Flink, so removing it wouldn't fail anything; could you add a small case next to `TestJavaHoodieBackedMetadata` that mirrors `TestSparkDeletePartitionTableServices`?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -100,6 +100,15 @@ public final class HoodieMetadataConfig extends HoodieConfig {
+ "instead of executing them inline. This prevents the current writer from executing compaction/logcompaction "
+ "on the metadata table, allowing a separate async pipeline to handle them.");
+ public static final ConfigProperty<Boolean> INLINE_TABLE_SERVICES_ENABLED = ConfigProperty
+ .key(METADATA_PREFIX + ".inline.table.services.enabled")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("1.3.0")
+ .withDocumentation("If false, a data table writer does not run clean, compaction or archival on the metadata table "
+ + "inline. For jobs that must not maintain the metadata table, such as a partition delete, "
Review Comment:
🤖 nit: could you reword the second sentence of this doc? "For jobs that must not maintain the metadata table ... and that leave it to ..." reads as a fragment, e.g. "Useful for jobs such as partition delete that should not maintain the metadata table and leave that to regular writers or a table service platform."
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org