You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "prashantwason (via GitHub)" <gi...@apache.org> on 2026/02/11 22:27:03 UTC
[PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
prashantwason opened a new pull request, #18181:
URL: https://github.com/apache/hudi/pull/18181
### Describe the issue this Pull Request addresses
Closes #18180
When the write config does not have correct enable flags for MDT (Metadata Table) partitions, they are automatically deleted. This does not work well for production datasets with multiple writers where accidental config changes can cause large MDT indexes to be deleted.
### Summary and Changelog
Added a new configuration `hoodie.metadata.auto.delete.partitions` that controls whether metadata table partitions can be automatically deleted when the corresponding config flags are disabled in the write config.
**Changes:**
- Added `hoodie.metadata.auto.delete.partitions` config in `HoodieMetadataConfig.java` (default: `true`)
- Added `isAutoDeleteMdtPartitionsEnabled()` getter in `HoodieWriteConfig.java`
- Modified `maybeDeleteMetadataTable()` in `HoodieTable.java` to check the config before deleting
- Modified `deleteMetadataIndexIfNecessary()` in `HoodieTable.java` to check the config before deleting
- Updated `HoodieSparkTable.java` to check `isMetadataTableAvailable()` in addition to `isMetadataTableEnabled()` for metadata writer initialization
- Replaced `storage.exists()` check with `metadataWriter.isInitialized()` for better reliability
### Impact
- New config: `hoodie.metadata.auto.delete.partitions` (default: `true`)
- No breaking changes - default behavior is unchanged (auto-delete enabled)
- Production users can set to `false` to prevent accidental deletion of metadata indexes
### Risk Level
low - This is an additive change with a feature flag. Default behavior remains unchanged.
### Documentation Update
Config description is included in the code. The new config `hoodie.metadata.auto.delete.partitions`:
- Default: `true`
- When enabled (default), metadata table partitions that are disabled in the write config will be automatically deleted
- When disabled, users must explicitly drop metadata partitions using hudi-cli or DROP INDEX command
- Recommended to disable for production datasets with multiple writers
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18181:
URL: https://github.com/apache/hudi/pull/18181#discussion_r2796270870
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -1121,6 +1121,9 @@ protected Option<HoodieTableMetadataWriter> getMetadataWriter(
* Deletes the metadata table if the writer disables metadata table with hoodie.metadata.enable=false
*/
public void maybeDeleteMetadataTable() {
Review Comment:
nit: the Javadoc above still only mentions `hoodie.metadata.enable=false` as the trigger. Could you update it to mention the new `auto.delete.partitions` config as well, so future readers understand why the method might be a no-op?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java:
##########
@@ -114,14 +110,9 @@ protected Option<HoodieTableMetadataWriter> getMetadataWriter(
HoodieTableMetadataWriter metadataWriter = streamingWrites
? SparkMetadataWriterFactory.createWithStreamingWrites(getContext().getStorageConf(), config, failedWritesCleaningPolicy, getContext(), Option.of(triggeringInstantTimestamp))
: SparkMetadataWriterFactory.create(getContext().getStorageConf(), config, failedWritesCleaningPolicy, getContext(), Option.of(triggeringInstantTimestamp), metaClient.getTableConfig());
- try {
- if (isMetadataTableExists || metaClient.getStorage().exists(
- HoodieTableMetadata.getMetadataTableBasePath(metaClient.getBasePath()))) {
- isMetadataTableExists = true;
- return Option.of(metadataWriter);
- }
- } catch (IOException e) {
- throw new HoodieMetadataException("Checking existence of metadata table failed", e);
+ if (isMetadataTableExists || metadataWriter.isInitialized()) {
Review Comment:
I understand the changes avoid the deletion of the metadata table if the feature flag is turned off. Have you checked if the MDT partitions can go outdated if the metadata config does not have the MDT partitions enabled, but the MDT table has the corresponding partitions and table config still marks them as available to read?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java:
##########
@@ -101,7 +97,7 @@ protected Option<HoodieTableMetadataWriter> getMetadataWriter(
if (isMetadataTable()) {
return Option.empty();
}
- if (config.isMetadataTableEnabled()) {
+ if (getMetaClient().getTableConfig().isMetadataTableAvailable() || config.isMetadataTableEnabled()) {
Review Comment:
With the `isMetadataTableAvailable()` condition added, when `config.isMetadataTableEnabled()` is false but the MDT exists on disk (available in table config), we'll now create a metadata writer and return it. Previously this path would have fallen to the `else` branch and called `maybeDeleteMetadataTable()`. Is the intent that the `isAutoDeleteMdtPartitionsEnabled` config guard in `maybeDeleteMetadataTable()` alone handles this, or is this `isMetadataTableAvailable()` change specifically meant to keep the writer alive when auto-delete is disabled? It might be worth adding a comment clarifying the intent.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java:
##########
@@ -114,14 +110,9 @@ protected Option<HoodieTableMetadataWriter> getMetadataWriter(
HoodieTableMetadataWriter metadataWriter = streamingWrites
? SparkMetadataWriterFactory.createWithStreamingWrites(getContext().getStorageConf(), config, failedWritesCleaningPolicy, getContext(), Option.of(triggeringInstantTimestamp))
: SparkMetadataWriterFactory.create(getContext().getStorageConf(), config, failedWritesCleaningPolicy, getContext(), Option.of(triggeringInstantTimestamp), metaClient.getTableConfig());
- try {
- if (isMetadataTableExists || metaClient.getStorage().exists(
- HoodieTableMetadata.getMetadataTableBasePath(metaClient.getBasePath()))) {
- isMetadataTableExists = true;
- return Option.of(metadataWriter);
- }
- } catch (IOException e) {
- throw new HoodieMetadataException("Checking existence of metadata table failed", e);
+ if (isMetadataTableExists || metadataWriter.isInitialized()) {
Review Comment:
Let's also add a test case around this?
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -509,6 +509,18 @@ public final class HoodieMetadataConfig extends HoodieConfig {
+ "The index name either starts with or matches exactly can be one of the following: "
+ StringUtils.join(Arrays.stream(MetadataPartitionType.values()).map(MetadataPartitionType::getPartitionPath).collect(Collectors.toList()), ", "));
+ // Config to enable/disable automatic deletion of MDT partitions
Review Comment:
nit: redundant comment. The config docs is self-explained.
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -509,6 +509,18 @@ public final class HoodieMetadataConfig extends HoodieConfig {
+ "The index name either starts with or matches exactly can be one of the following: "
+ StringUtils.join(Arrays.stream(MetadataPartitionType.values()).map(MetadataPartitionType::getPartitionPath).collect(Collectors.toList()), ", "));
+ // Config to enable/disable automatic deletion of MDT partitions
+ public static final ConfigProperty<Boolean> AUTO_DELETE_PARTITIONS = ConfigProperty
+ .key(METADATA_PREFIX + ".auto.delete.partitions")
+ .defaultValue(true)
Review Comment:
Let's add a test with this config disabled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18181:
URL: https://github.com/apache/hudi/pull/18181#issuecomment-3887794353
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11952",
"triggerID" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"triggerType" : "PUSH"
}, {
"hash" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab6556646cae6540bb0057c0d7d5bb8a3a917f69 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11952)
* 0e6352e9e9b06ea35696d23dc382070568d8ae90 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18181:
URL: https://github.com/apache/hudi/pull/18181#issuecomment-3948050336
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11952",
"triggerID" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"triggerType" : "PUSH"
}, {
"hash" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11959",
"triggerID" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"triggerType" : "PUSH"
}, {
"hash" : "1fa54c5ec3a53bd670538522aad69264a4e2aaa2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12239",
"triggerID" : "1fa54c5ec3a53bd670538522aad69264a4e2aaa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fa54c5ec3a53bd670538522aad69264a4e2aaa2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12239)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #18181:
URL: https://github.com/apache/hudi/pull/18181#discussion_r2943005267
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/table/TestHoodieTable.java:
##########
@@ -136,4 +138,54 @@ void testRollbackInflightInstant() throws IOException {
// Validate that function scheduleRollback is called.
assertEquals(1, ((TestBaseHoodieTable) hoodieTable).getCountOfScheduleRollbackFunctionCalls());
}
+
+ @Test
+ void testMaybeDeleteMetadataTableSkipsWhenAutoDeleteDisabled() throws IOException {
Review Comment:
and also validate that col stats did receive a log file as well (from HoodieCommitMetadata).
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/table/TestHoodieTable.java:
##########
@@ -136,4 +138,54 @@ void testRollbackInflightInstant() throws IOException {
// Validate that function scheduleRollback is called.
assertEquals(1, ((TestBaseHoodieTable) hoodieTable).getCountOfScheduleRollbackFunctionCalls());
}
+
+ @Test
+ void testMaybeDeleteMetadataTableSkipsWhenAutoDeleteDisabled() throws IOException {
Review Comment:
can we add one end to end functional test to validate that, after initializing, if mdt config had disable col stats, but w/ "auto delete partitions' config as false. we validate that, col stats is not deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18181:
URL: https://github.com/apache/hudi/pull/18181#discussion_r2910865241
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -509,6 +509,18 @@ public final class HoodieMetadataConfig extends HoodieConfig {
+ "The index name either starts with or matches exactly can be one of the following: "
+ StringUtils.join(Arrays.stream(MetadataPartitionType.values()).map(MetadataPartitionType::getPartitionPath).collect(Collectors.toList()), ", "));
+ // Config to enable/disable automatic deletion of MDT partitions
+ public static final ConfigProperty<Boolean> AUTO_DELETE_PARTITIONS = ConfigProperty
+ .key(METADATA_PREFIX + ".auto.delete.partitions")
+ .defaultValue(true)
Review Comment:
Added two tests in `TestHoodieTable.java` in the latest commit:
1. `testMaybeDeleteMetadataTableSkipsWhenAutoDeleteDisabled` - verifies MDT is not deleted when `auto.delete.partitions=false`
2. `testDeleteMetadataIndexIfNecessarySkipsWhenAutoDeleteDisabled` - verifies index deletion is skipped when `auto.delete.partitions=false`
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java:
##########
@@ -114,14 +110,9 @@ protected Option<HoodieTableMetadataWriter> getMetadataWriter(
HoodieTableMetadataWriter metadataWriter = streamingWrites
? SparkMetadataWriterFactory.createWithStreamingWrites(getContext().getStorageConf(), config, failedWritesCleaningPolicy, getContext(), Option.of(triggeringInstantTimestamp))
: SparkMetadataWriterFactory.create(getContext().getStorageConf(), config, failedWritesCleaningPolicy, getContext(), Option.of(triggeringInstantTimestamp), metaClient.getTableConfig());
- try {
- if (isMetadataTableExists || metaClient.getStorage().exists(
- HoodieTableMetadata.getMetadataTableBasePath(metaClient.getBasePath()))) {
- isMetadataTableExists = true;
- return Option.of(metadataWriter);
- }
- } catch (IOException e) {
- throw new HoodieMetadataException("Checking existence of metadata table failed", e);
+ if (isMetadataTableExists || metadataWriter.isInitialized()) {
Review Comment:
The writer creation path in `HoodieSparkTable.getMetadataWriter()` requires Spark context for a proper integration test. The two unit tests added in `TestHoodieTable.java` cover the auto-delete guard behavior at the `HoodieTable` level.
For the outdated partitions concern — as explained above, the metadata writer uses `dataMetaClient.getTableConfig().getMetadataPartitions()` as the source of truth for updates, so partitions won't go stale as long as the writer is created. The `isMetadataTableAvailable()` condition ensures the writer is created whenever MDT exists on disk.
If you'd like a Spark integration test specifically for the writer creation path, I can add one — let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18181:
URL: https://github.com/apache/hudi/pull/18181#issuecomment-3888084756
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11952",
"triggerID" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"triggerType" : "PUSH"
}, {
"hash" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11959",
"triggerID" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0e6352e9e9b06ea35696d23dc382070568d8ae90 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11959)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason merged PR #18181:
URL: https://github.com/apache/hudi/pull/18181
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18181:
URL: https://github.com/apache/hudi/pull/18181#issuecomment-4031334386
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11952",
"triggerID" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"triggerType" : "PUSH"
}, {
"hash" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11959",
"triggerID" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"triggerType" : "PUSH"
}, {
"hash" : "1fa54c5ec3a53bd670538522aad69264a4e2aaa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12239",
"triggerID" : "1fa54c5ec3a53bd670538522aad69264a4e2aaa2",
"triggerType" : "PUSH"
}, {
"hash" : "8aec291b99a5bbe180a7919ce5ac9805df7f1bf1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12536",
"triggerID" : "8aec291b99a5bbe180a7919ce5ac9805df7f1bf1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8aec291b99a5bbe180a7919ce5ac9805df7f1bf1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12536)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18181:
URL: https://github.com/apache/hudi/pull/18181#issuecomment-3947987540
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18181?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `57.14286%` with `6 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 45.42%. Comparing base ([`ec04479`](https://app.codecov.io/gh/apache/hudi/commit/ec044790fe71f04710472ad63e7859f1b29e3985?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`1fa54c5`](https://app.codecov.io/gh/apache/hudi/commit/1fa54c5ec3a53bd670538522aad69264a4e2aaa2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 988 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18181?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...c/main/java/org/apache/hudi/table/HoodieTable.java](https://app.codecov.io/gh/apache/hudi/pull/18181?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Ftable%2FHoodieTable.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL0hvb2RpZVRhYmxlLmphdmE=) | 0.00% | [2 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pache/hudi/common/config/HoodieMetadataConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18181?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fconfig%2FHoodieMetadataConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2NvbmZpZy9Ib29kaWVNZXRhZGF0YUNvbmZpZy5qYXZh) | 77.77% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (ec04479) and HEAD (1fa54c5). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (ec04479) | HEAD (1fa54c5) |
>|------|------|------|
>||1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18181 +/- ##
=============================================
- Coverage 61.43% 45.42% -16.01%
+ Complexity 23082 8374 -14708
=============================================
Files 2108 1177 -931
Lines 127636 60253 -67383
Branches 14534 6415 -8119
=============================================
- Hits 78409 27371 -51038
+ Misses 42873 29907 -12966
+ Partials 6354 2975 -3379
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18181/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18181/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.42% <57.14%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18181?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18181?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fconfig%2FHoodieWriteConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `74.43% <100.00%> (ø)` | |
| [...pache/hudi/common/config/HoodieMetadataConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18181?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fconfig%2FHoodieMetadataConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2NvbmZpZy9Ib29kaWVNZXRhZGF0YUNvbmZpZy5qYXZh) | `77.88% <77.77%> (ø)` | |
| [...c/main/java/org/apache/hudi/table/HoodieTable.java](https://app.codecov.io/gh/apache/hudi/pull/18181?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Ftable%2FHoodieTable.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL0hvb2RpZVRhYmxlLmphdmE=) | `57.96% <0.00%> (ø)` | |
... and [3240 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18181/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #18181:
URL: https://github.com/apache/hudi/pull/18181#issuecomment-4041705158
hey @yihua : reminder to review the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18181:
URL: https://github.com/apache/hudi/pull/18181#discussion_r2910863621
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java:
##########
@@ -114,14 +110,9 @@ protected Option<HoodieTableMetadataWriter> getMetadataWriter(
HoodieTableMetadataWriter metadataWriter = streamingWrites
? SparkMetadataWriterFactory.createWithStreamingWrites(getContext().getStorageConf(), config, failedWritesCleaningPolicy, getContext(), Option.of(triggeringInstantTimestamp))
: SparkMetadataWriterFactory.create(getContext().getStorageConf(), config, failedWritesCleaningPolicy, getContext(), Option.of(triggeringInstantTimestamp), metaClient.getTableConfig());
- try {
- if (isMetadataTableExists || metaClient.getStorage().exists(
- HoodieTableMetadata.getMetadataTableBasePath(metaClient.getBasePath()))) {
- isMetadataTableExists = true;
- return Option.of(metadataWriter);
- }
- } catch (IOException e) {
- throw new HoodieMetadataException("Checking existence of metadata table failed", e);
+ if (isMetadataTableExists || metadataWriter.isInitialized()) {
Review Comment:
Good question. The MDT partitions will NOT go stale in this scenario. Here's why:
The metadata writer's `getMetadataPartitionsToUpdate()` method (in `HoodieBackedTableMetadataWriter.java:1171`) uses `dataMetaClient.getTableConfig().getMetadataPartitions()` as the source of truth for which partitions to update — not the write config's enabled partition settings. So even if the write config doesn't explicitly enable certain MDT partitions, as long as the table config marks them as available, the writer will continue to update them.
With this PR's change to add the `isMetadataTableAvailable()` check, the writer is always created when MDT exists on disk, ensuring the partitions stay in sync regardless of the write config's `metadata.enable` setting.
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -509,6 +509,18 @@ public final class HoodieMetadataConfig extends HoodieConfig {
+ "The index name either starts with or matches exactly can be one of the following: "
+ StringUtils.join(Arrays.stream(MetadataPartitionType.values()).map(MetadataPartitionType::getPartitionPath).collect(Collectors.toList()), ", "));
+ // Config to enable/disable automatic deletion of MDT partitions
Review Comment:
Removed the redundant comment in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18181:
URL: https://github.com/apache/hudi/pull/18181#issuecomment-4030541223
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11952",
"triggerID" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"triggerType" : "PUSH"
}, {
"hash" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11959",
"triggerID" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"triggerType" : "PUSH"
}, {
"hash" : "1fa54c5ec3a53bd670538522aad69264a4e2aaa2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12239",
"triggerID" : "1fa54c5ec3a53bd670538522aad69264a4e2aaa2",
"triggerType" : "PUSH"
}, {
"hash" : "8aec291b99a5bbe180a7919ce5ac9805df7f1bf1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8aec291b99a5bbe180a7919ce5ac9805df7f1bf1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fa54c5ec3a53bd670538522aad69264a4e2aaa2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12239)
* 8aec291b99a5bbe180a7919ce5ac9805df7f1bf1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18181:
URL: https://github.com/apache/hudi/pull/18181#issuecomment-3947830883
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11952",
"triggerID" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"triggerType" : "PUSH"
}, {
"hash" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11959",
"triggerID" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"triggerType" : "PUSH"
}, {
"hash" : "1fa54c5ec3a53bd670538522aad69264a4e2aaa2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12239",
"triggerID" : "1fa54c5ec3a53bd670538522aad69264a4e2aaa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0e6352e9e9b06ea35696d23dc382070568d8ae90 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11959)
* 1fa54c5ec3a53bd670538522aad69264a4e2aaa2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12239)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18181:
URL: https://github.com/apache/hudi/pull/18181#issuecomment-4030559237
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11952",
"triggerID" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"triggerType" : "PUSH"
}, {
"hash" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11959",
"triggerID" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"triggerType" : "PUSH"
}, {
"hash" : "1fa54c5ec3a53bd670538522aad69264a4e2aaa2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12239",
"triggerID" : "1fa54c5ec3a53bd670538522aad69264a4e2aaa2",
"triggerType" : "PUSH"
}, {
"hash" : "8aec291b99a5bbe180a7919ce5ac9805df7f1bf1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12536",
"triggerID" : "8aec291b99a5bbe180a7919ce5ac9805df7f1bf1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fa54c5ec3a53bd670538522aad69264a4e2aaa2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12239)
* 8aec291b99a5bbe180a7919ce5ac9805df7f1bf1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12536)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18181:
URL: https://github.com/apache/hudi/pull/18181#issuecomment-3887585836
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11952",
"triggerID" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"triggerType" : "PUSH"
}, {
"hash" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab6556646cae6540bb0057c0d7d5bb8a3a917f69 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11952)
* 0e6352e9e9b06ea35696d23dc382070568d8ae90 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18181:
URL: https://github.com/apache/hudi/pull/18181#issuecomment-3887575866
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab6556646cae6540bb0057c0d7d5bb8a3a917f69 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18181:
URL: https://github.com/apache/hudi/pull/18181#issuecomment-3887817310
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11952",
"triggerID" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"triggerType" : "PUSH"
}, {
"hash" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11959",
"triggerID" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab6556646cae6540bb0057c0d7d5bb8a3a917f69 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11952)
* 0e6352e9e9b06ea35696d23dc382070568d8ae90 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11959)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18181:
URL: https://github.com/apache/hudi/pull/18181#discussion_r2910857538
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java:
##########
@@ -101,7 +97,7 @@ protected Option<HoodieTableMetadataWriter> getMetadataWriter(
if (isMetadataTable()) {
return Option.empty();
}
- if (config.isMetadataTableEnabled()) {
+ if (getMetaClient().getTableConfig().isMetadataTableAvailable() || config.isMetadataTableEnabled()) {
Review Comment:
Good point. The intent is specifically to keep the writer alive when `auto-delete` is disabled. When `isMetadataTableAvailable()` is true (MDT exists on disk per table config) but `isMetadataTableEnabled()` is false in write config, we now create the writer so that the MDT stays in sync. The `isAutoDeleteMdtPartitionsEnabled` guard in `maybeDeleteMetadataTable()` ensures we don't accidentally delete the MDT when a misconfigured writer disables metadata.
Added a clarifying comment in the latest commit explaining this intent.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -1121,6 +1121,9 @@ protected Option<HoodieTableMetadataWriter> getMetadataWriter(
* Deletes the metadata table if the writer disables metadata table with hoodie.metadata.enable=false
*/
public void maybeDeleteMetadataTable() {
Review Comment:
Updated the Javadoc to mention the `auto.delete.partitions` config in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18181:
URL: https://github.com/apache/hudi/pull/18181#discussion_r2913380702
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java:
##########
@@ -114,14 +110,9 @@ protected Option<HoodieTableMetadataWriter> getMetadataWriter(
HoodieTableMetadataWriter metadataWriter = streamingWrites
? SparkMetadataWriterFactory.createWithStreamingWrites(getContext().getStorageConf(), config, failedWritesCleaningPolicy, getContext(), Option.of(triggeringInstantTimestamp))
: SparkMetadataWriterFactory.create(getContext().getStorageConf(), config, failedWritesCleaningPolicy, getContext(), Option.of(triggeringInstantTimestamp), metaClient.getTableConfig());
- try {
- if (isMetadataTableExists || metaClient.getStorage().exists(
- HoodieTableMetadata.getMetadataTableBasePath(metaClient.getBasePath()))) {
- isMetadataTableExists = true;
- return Option.of(metadataWriter);
- }
- } catch (IOException e) {
- throw new HoodieMetadataException("Checking existence of metadata table failed", e);
+ if (isMetadataTableExists || metadataWriter.isInitialized()) {
Review Comment:
Updated the logic in the latest commit. The writer is now only created for a disabled MDT when `auto.delete.partitions` is also disabled. When auto-delete is enabled (default), the existing turn-off behavior is preserved — the writer is not created and `maybeDeleteMetadataTable()` cleans up the MDT. This fixes the CI failures in `testTurnOffMetadataTableAfterEnable` and `testTurnOffMetadataIndexAfterEnable`.
The condition is now:
```java
if (config.isMetadataTableEnabled()
|| (getMetaClient().getTableConfig().isMetadataTableAvailable() && !config.isAutoDeleteMdtPartitionsEnabled()))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata-table): add config to disable automatic deletion of MDT partitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18181:
URL: https://github.com/apache/hudi/pull/18181#issuecomment-3947821681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11952",
"triggerID" : "ab6556646cae6540bb0057c0d7d5bb8a3a917f69",
"triggerType" : "PUSH"
}, {
"hash" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11959",
"triggerID" : "0e6352e9e9b06ea35696d23dc382070568d8ae90",
"triggerType" : "PUSH"
}, {
"hash" : "1fa54c5ec3a53bd670538522aad69264a4e2aaa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1fa54c5ec3a53bd670538522aad69264a4e2aaa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0e6352e9e9b06ea35696d23dc382070568d8ae90 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11959)
* 1fa54c5ec3a53bd670538522aad69264a4e2aaa2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org